Analytics Azure

OLake on Ubuntu 24.04 on Azure User Guide

| Product: OLake on Ubuntu 24.04 LTS on Azure

Overview

This guide covers the deployment and configuration of OLake on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. OLake is an open source ETL and change data capture platform that continuously moves data out of operational databases and lands it in an analytics ready lakehouse. It reads from PostgreSQL, MySQL, MongoDB, Oracle, SQL Server, Db2, Kafka and S3, and writes Apache Parquet to object storage or tables to Apache Iceberg.

Two replication modes are supported. A full refresh copies a table in its entirety, which is the simplest way to get started and to reload a table on demand. Change data capture reads the source database write ahead log so that inserts, updates and deletes stream into the lakehouse continuously, without repeatedly rescanning the source. Everything is driven from a web console: register a source, register a destination, discover the tables and columns available, choose which streams to replicate and in which mode, then run the job and watch its history.

What is included:

  • OLake web console and API (olakego/ui v0.4.9)
  • OLake sync worker (olakego/ui-worker v0.3.10), which runs each sync in an isolated connector container
  • OLake PostgreSQL source connector (olakego/source-postgres v0.8.2), pre pulled so the first sync does not wait on a download
  • Temporal 1.28.4 workflow engine with a PostgreSQL 16 metadata and visibility store
  • A bundled demo source: PostgreSQL 16 seeded on first boot with a synthetic dataset generated on the VM itself
  • A bundled demo destination: SeaweedFS 4.39, an Apache 2.0 S3 compatible object store
  • nginx as the single front door on port 80
  • DuckDB 1.5.4, used by the bundled verification script to read the Parquet that OLake writes
  • Ubuntu 24.04 LTS, fully patched, with unattended security upgrades enabled
  • 24/7 cloudimg support

The docker, olake firstboot, olake, olake postboot and nginx services all active, and the six containers of the stack running and healthy

Key facts:

Item Value
Platform Azure Virtual Machine, Ubuntu 24.04 LTS
Default SSH user azureuser
Web console http://<vm-ip>/
Credentials file /root/olake-credentials.txt (mode 0600, root only)
Compose file /etc/olake/compose.yaml
Per VM environment /etc/olake/olake.env (mode 0600, root only)

Prerequisites

  1. An active Azure subscription
  2. Permission to create virtual machines, virtual networks and network security groups
  3. An SSH key pair for the azureuser account
  4. A network security group that allows inbound TCP 22 for SSH and TCP 80 for the web console, both restricted to your own address range

Recommended VM size: Standard_B2s (2 vCPU, 4 GB RAM) is sufficient for evaluation and for the bundled demo dataset, and is the size this image is tested on end to end. For production replication of large tables, size up in line with the volume you intend to move: the OLake project documents 8 vCPU and 16 GB as its own guidance for production workloads, since each concurrent sync runs in its own connector container.

Step 1: Deploy from the Azure Portal

  1. In the Azure Portal choose Create a resource and search the Marketplace for OLake on Ubuntu 24.04 LTS by cloudimg
  2. Select the offer and choose Create
  3. Pick your subscription, resource group and region
  4. Choose the size Standard_B2s or larger
  5. Set the authentication type to SSH public key and the username to azureuser
  6. On the Networking tab, allow inbound SSH (22) and HTTP (80) from your own address range only
  7. Choose Review and create, then Create

Step 2: Deploy from the Azure CLI

az group create --name olake-rg --location eastus

az vm create \
  --resource-group olake-rg \
  --name olake-vm \
  --image cloudimg:olake-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

az vm open-port --resource-group olake-rg --name olake-vm --port 80 --priority 1010

Restrict port 80 to your own address range rather than leaving it open to the internet. See Step 15 for putting the console behind HTTPS.

Step 3: Connect to your VM

ssh azureuser@<vm-ip>

To find the address of the VM you just created:

az vm show --resource-group olake-rg --name olake-vm --show-details --query publicIps --output tsv

Step 4: Confirm the services are running

The image runs five systemd units. olake-firstboot generates this VM's unique secrets before anything starts, olake brings the container stack up, and olake-postboot creates the administrator account and seeds the demo data.

systemctl is-active docker.service olake-firstboot.service olake.service olake-postboot.service nginx.service

All five report active. The container stack itself is six containers:

sudo docker ps --format 'table {{.Names}}\t{{.Status}}'

You should see olake-ui, olake-worker, olake-temporal, olake-postgres, olake-sample-db and olake-storage, all Up and the health checked ones reporting healthy.

Step 5: Read the per instance credentials

Nothing in this image ships with a known password. On the first boot of every VM a unique console administrator password, metadata database password, credential encryption key, demo database password and object storage key pair are generated and written to a root only file.

sudo cat /root/olake-credentials.txt

The file is mode 0600 and owned by root, and it lists the administrator login, the web console URL, and the connection details for the bundled demo source and destination.

The olake credentials file at mode 0600 root root, listing only the names of the generated values, with a note that every one of them is unique to this VM and that values are never shown in documentation

Step 6: Verify the security model

The console API refuses anonymous access and refuses a forged session cookie, and accepts only this VM's administrator credential.

curl -s -o /dev/null -w 'unauthenticated -> HTTP %{http_code}\n' http://127.0.0.1/api/v1/project/123/sources
curl -s -o /dev/null -w 'forged cookie   -> HTTP %{http_code}\n' -H 'Cookie: olake-session=not-a-real-session' http://127.0.0.1/api/v1/project/123/sources

Both return 401. Now authenticate with this VM's administrator credential and call exactly the same URL:

curl -s -c /tmp/olake-cj -o /dev/null -X POST -H 'Content-Type: application/json' \
  -d '{"username":"<ADMIN_USERNAME>","password":"<ADMIN_PASSWORD>"}' http://127.0.0.1/login
curl -s -b /tmp/olake-cj -o /dev/null -w 'authenticated   -> HTTP %{http_code}\n' http://127.0.0.1/api/v1/project/123/sources
rm -f /tmp/olake-cj

This returns 200. The pairing matters: the same URL that was refused anonymously is accepted with the per instance credential, which shows the refusals were real authorisation decisions rather than a service that was simply not running.

curl showing the console API returning HTTP 401 for an unauthenticated request and for a forged session cookie, and HTTP 200 for the same URL once the per instance administrator credential has authenticated

Step 7: Run the bundled replication check

The image ships a verification script that proves a real replication end to end. It writes a table of known rows into the demo source, runs a sync through OLake, then reads back the Parquet that OLake actually wrote and compares it to the source row for row.

sudo /usr/local/sbin/olake-roundtrip.sh

It prints a line beginning OK round-trip: that reports the row count replicated and confirms the content digest matches the source exactly. The check asserts on the data itself, not on the job status, so a sync that reported success while moving zero rows would fail it.

The bundled round trip script reporting OK: the console refuses unauthenticated and forged requests, accepts the per instance credential, and 250 of 250 rows landed in the object store as Parquet with a content digest matching the source exactly

Step 8: Sign in to the web console

Open <WEB_URL> in a browser and sign in with the ADMIN_USERNAME and ADMIN_PASSWORD from the credentials file.

The OLake console sign in page with the per instance administrator username entered and the password masked

Step 9: Run your first replication

The image arrives with a demo job already registered, called Demo: customers to lakehouse. It replicates the demo.customers table from the bundled PostgreSQL source into the bundled object store as Parquet. It has never been run, so the first sync is yours.

Open Jobs, find the demo job and choose Sync from its actions menu. The run completes in a few seconds for the demo dataset, and the Last Run status column turns to Completed.

The OLake jobs list showing the demo job, its source Sample Postgres, its destination Local Lakehouse, and a Last Run status of Completed

Step 10: Inspect the job history and logs

Selecting the job opens Job Logs and History, which lists every run with its start time, runtime and status, and gives you the connector logs for each run through View logs.

The job history for the demo job, listing completed sync runs each with a start time, a runtime of six seconds, a status of Completed and a View logs action

Step 11: Review the bundled source and destination

Sources lists the bundled demo database, and shows which jobs are associated with it. Destinations lists the bundled object store. Both are ordinary OLake entries: edit them, or add your own alongside them.

The OLake sources list showing the bundled Sample Postgres source and the demo job associated with it

Step 12: Inspect the demo source directly

The bundled source is a normal PostgreSQL 16 database. It is published on the container gateway address rather than on a public interface, so it is reachable from the VM and from the connector containers but not from the internet.

sudo docker exec -e PGPASSWORD='<SAMPLE_DB_PASSWORD>' olake-sample-db \
  psql -U olake_demo -d olake_demo -c 'SELECT count(*) AS customers FROM demo.customers'

sudo docker exec -e PGPASSWORD='<SAMPLE_DB_PASSWORD>' olake-sample-db \
  psql -U olake_demo -d olake_demo -c 'SELECT count(*) AS orders FROM demo.orders'

The dataset is 500 customers and 2000 orders, generated on this VM at first boot. It contains no third party data.

Step 13: Read the replicated Parquet

The destination is an S3 compatible object store, so the objects OLake writes can be listed and read with ordinary S3 tooling. The image includes DuckDB, which reads Parquet directly.

sudo python3 - <<'PY'
import boto3, os
from botocore.config import Config
env = dict(l.strip().split("=", 1) for l in open("/etc/olake/olake.env")
           if "=" in l and not l.startswith("#"))
s3 = boto3.client("s3", endpoint_url=env["S3_ENDPOINT"],
                  aws_access_key_id=env["S3_ACCESS_KEY"],
                  aws_secret_access_key=env["S3_SECRET_KEY"],
                  region_name="us-east-1",
                  config=Config(s3={"addressing_style": "path"}))
for o in (s3.list_objects_v2(Bucket=env["S3_BUCKET"]).get("Contents") or [])[:10]:
    print(o["Key"], o["Size"], "bytes")
PY

Each sync writes Parquet under a prefix named for the destination database and table. To query a downloaded object directly:

sudo duckdb -c "SELECT 1 AS duckdb_ready"

Step 14: Connect your own source and destination

To replicate your own data, add a source and a destination in the console and create a job across them.

  • Sources accepts PostgreSQL, MySQL, MongoDB, Oracle, SQL Server, Db2, Kafka and S3. For change data capture from PostgreSQL, the source database needs wal_level=logical and a replication slot; the bundled demo database is already configured that way so you can try the mode before pointing OLake at your own systems.
  • Destinations accepts Amazon S3 (written as Parquet) and Apache Iceberg. For Iceberg, supply your catalog: AWS Glue, a REST catalog, Hive or JDBC, together with the S3 location of the warehouse. The bundled object store is a normal S3 endpoint, so you can point a destination at your own bucket by replacing the endpoint, keys and bucket name.

The connector image for a source type is pulled the first time it is used. The PostgreSQL connector is already present in this image; other source types download on first use, so allow outbound access to your container registry.

Step 15: Server components

Component Version Role
OLake console and API v0.4.9 Web UI and REST API on 127.0.0.1:8000 behind nginx
OLake worker v0.3.10 Runs each sync in its own connector container
OLake PostgreSQL connector v0.8.2 Reads the source database
Temporal 1.28.4 Workflow engine, PostgreSQL backed visibility
PostgreSQL (metadata) 16 Temporal and console state
PostgreSQL (demo source) 16 Bundled synthetic dataset, wal_level=logical
SeaweedFS 4.39 Bundled S3 compatible destination
nginx Ubuntu 24.04 Front door on port 80
DuckDB 1.5.4 Reads Parquet for the bundled verification script

Every container image is pinned by digest, so the versions above are exactly what runs.

Step 16: Filesystem layout

Path Description
/ Root filesystem
/etc/olake/compose.yaml The container stack definition
/etc/olake/olake.env Per VM secrets, mode 0600 root only
/etc/olake/seaweed-s3.json Object store identities for this VM
/var/lib/olake/config OLake job configuration and connector working files
/var/lib/olake/storage Object store data for the bundled destination
/root/olake-credentials.txt Per VM credentials, mode 0600 root only
/mnt Azure temporary resource disk

Step 17: Managing the service

sudo systemctl status olake.service --no-pager

To restart the whole stack, or to stop it when the VM is idle:

sudo systemctl restart olake.service

sudo systemctl stop olake.service

Container logs for any single component:

sudo docker logs --tail 50 olake-ui

Step 18: Use your own domain and HTTPS

The console is served over plain HTTP on port 80, which is appropriate for a private network or an evaluation. For anything beyond that, put it behind TLS before exposing it. Point a DNS record at the VM, then install a certificate with Certbot and the nginx plugin, which rewrites /etc/nginx/sites-available/olake to serve HTTPS and redirect port 80. Restrict port 80 and 443 in the network security group to the address ranges that need them.

Step 19: Security recommendations

  1. Restrict inbound 22 and 80 in the network security group to your own address ranges. Nothing else in this image listens on a public interface: the console binds to 127.0.0.1 behind nginx, and the demo database and object store bind to the container gateway address, which is not routable from outside the VM.
  2. Change the administrator password from the console after your first sign in, and keep /root/olake-credentials.txt readable only by root.
  3. Put the console behind HTTPS before exposing it beyond a trusted network, as described in Step 18.
  4. Treat the bundled demo source and destination as a demonstration. Remove the demo job, source and destination once you have connected your own systems.
  5. The worker runs syncs by launching containers through the Docker socket, which is root equivalent on the host. Restrict SSH access to the VM accordingly.
  6. Keep the operating system patched. Unattended security upgrades are enabled by default.

Step 20: Troubleshooting

The console does not answer on port 80. Check that the stack is up with sudo systemctl status olake.service --no-pager and sudo docker ps. On the very first boot the Temporal schema is created before the console starts, which takes up to a minute longer than later boots.

A job stays in a running state. Look at the run in Job Logs and History and open View logs for the connector output. The worker launches a connector container per sync, so sudo docker ps -a shows the connector container and its exit state.

A sync fails immediately for a source you added. The connector image for that source type is downloaded on first use. Confirm the VM has outbound access to the container registry, then retry.

The demo job is missing. It is created on first boot only. Check sudo journalctl -u olake-postboot.service --no-pager for the seeding log.

Step 21: Support and licensing

OLake is licensed under the Apache License 2.0. Temporal, SeaweedFS, PostgreSQL and DuckDB are shipped under their own open source licences. This image is not affiliated with, endorsed by, or sponsored by OLake, Datazip, Temporal, SeaweedFS or the PostgreSQL Global Development Group; those names are used only to identify the software included here.

For assistance with this image, contact cloudimg support:

Deploy on Azure

Find this image on the Azure Marketplace as OLake on Ubuntu 24.04 LTS by cloudimg.