OLake Database to Lakehouse Replication on AWS User Guide

Overview

This image runs OLake, the open source ETL and change data capture platform for moving data out of operational databases and into an analytics ready lakehouse. OLake replicates tables from databases such as PostgreSQL, MySQL, MongoDB, Oracle and SQL Server into Apache Iceberg or as Parquet in object storage, where analytics engines can query it. It supports both full refresh loads, which copy a table in its entirety, and change data capture, which reads the database write ahead log so that inserts, updates and deletes stream into the lakehouse continuously without repeatedly rescanning the source.

Everything is driven from a web console: register a source, register a destination, discover the tables and columns available, choose which streams to replicate and in which mode, then run the job and watch its history. Each sync runs as an isolated connector process, so a large table replication cannot destabilise the console, and job runs, schedules and logs are all visible in one place.

OLake is normally a multi container deployment. This image runs the official upstream containers, pinned by digest, under a single systemd service behind an nginx reverse proxy, so the whole stack starts and stops as one unit. To make the platform useful the moment it boots, the image also bundles both ends of a pipeline: a demo PostgreSQL source, seeded on first boot with a synthetic dataset generated on the instance itself, and an S3 compatible object store destination, both already registered in the console. You can run a real replication before connecting any of your own systems.

nginx publishes the console on port 80 and is the only external surface. The OLake API, the Temporal workflow engine, the metadata database, the demo source database and the object store all bind to the loopback interface or the internal Docker gateway and are not reachable from outside the instance.

The administrator password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/olake-credentials.txt with mode 0600, so only the root user can read it. First boot also generates the metadata database password, the credential encryption key, the demo database password and the object store access key pair, so none of the upstream default credentials exist anywhere in the image.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the OLake web console
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for OLake. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. The instance runs a six container stack (the OLake API and worker, a Temporal workflow engine, a metadata database, the demo source database and the object store), so smaller burstable types are not recommended. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the console. Leave the root volume at the default size or larger; the container images and volumes live on a separate data volume that the image provisions automatically.

Select Launch instance. First boot initialisation takes a few minutes after the instance state becomes Running and the status checks pass, while the image generates its per instance credentials, starts the container stack, creates the administrator account and seeds the demonstration dataset.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg OLake Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=olake}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

Step 4: Retrieve the Administrator Password and Demo Details

The administrator login is unique to your instance and was generated on first boot. The credentials file records the console URL, the administrator user name, the generated password, and the connection details for the bundled demo source and destination.

Confirm the file is present and read the non secret fields:

sudo grep -E '^(WEB_URL|ADMIN_USERNAME|S3_ENDPOINT|S3_BUCKET|SAMPLE_DB_HOST|SAMPLE_DB_PORT|SAMPLE_DB_NAME|SAMPLE_DB_USER)=' /root/olake-credentials.txt

On a freshly launched instance this reports the console URL, the administrator user name (olakeadmin), and the demo source and object store details:

WEB_URL=http://203.0.113.10/
ADMIN_USERNAME=olakeadmin
SAMPLE_DB_HOST=172.17.0.1
SAMPLE_DB_PORT=55432
SAMPLE_DB_NAME=olake_demo
SAMPLE_DB_USER=olake_demo
S3_ENDPOINT=http://172.17.0.1:8333
S3_BUCKET=olake-lake

To read the generated administrator password itself, run sudo cat /root/olake-credentials.txt and copy the value on the ADMIN_PASSWORD line. Keep it somewhere safe. The file is mode 0600 and owned by root, so it is readable only through sudo. The same file records the demo database password and the object store access and secret keys, all unique to this instance.

Step 5: Confirm the Services Are Running

Over SSH, confirm the host services are active:

sudo systemctl is-active docker olake nginx

All three report active. The olake service brings up the container stack; you can see the individual containers and their health with Docker:

sudo docker ps --format 'table {{.Names}}\t{{.Status}}'

The six containers report Up and, where they define a health check, (healthy):

NAMES             STATUS
olake-worker      Up 7 minutes
olake-ui          Up 7 minutes (healthy)
olake-temporal    Up 7 minutes (healthy)
olake-storage     Up 8 minutes (healthy)
olake-sample-db   Up 8 minutes (healthy)
olake-postgres    Up 8 minutes (healthy)

Step 6: Confirm the Login Wall Is In Place

Upstream OLake ships with a published default administrator account. This image does not: it removes that account and creates a per instance administrator on first boot, and the console refuses any unauthenticated API request. Confirm that on your own instance. The console front door answers on port 80:

curl -s -o /dev/null -w 'front door: HTTP %{http_code}\n' http://127.0.0.1/

An unauthenticated request to the sources API is refused:

curl -s -o /dev/null -w 'unauthenticated API: HTTP %{http_code}\n' http://127.0.0.1/api/v1/project/123/sources

The front door returns 200 and the unauthenticated API request returns 401, so nothing behind the console is served without a session.

Step 7: Sign In to the Console

Browse to http://<public-ip>/ in your browser. OLake redirects to its sign in page. Enter olakeadmin as the user name and the password you retrieved in Step 4.

Step 8: Explore the Pre-Configured Demo Pipeline

After signing in, the console opens the Jobs view. A demonstration job named Demo: customers to lakehouse is already registered, replicating the seeded customers table from the bundled PostgreSQL source (Sample Postgres) to the bundled object store destination (Local Lakehouse).

The Sources view under Ingestion lists the registered sources. The bundled Sample Postgres source is a real PostgreSQL 16 database, seeded on first boot with a synthetic dataset of 500 customers and 2000 orders generated on the instance itself.

The Destinations view lists the registered destinations. The bundled Local Lakehouse destination writes Parquet to the instance's own S3 compatible object store, so a full replication can be demonstrated with no external cloud account.

Step 9: Run Your First Replication

Open the Demo: customers to lakehouse job from the Jobs view and choose Sync to run it. OLake spawns an isolated connector container, reads the customers table from the demo source and writes it as Parquet to the object store. When the run completes, the job's Last Run status shows Completed, and the run history records the row counts and timing.

Because the demo source runs with wal_level=logical, you can also switch the streams from full refresh to change data capture mode and re run the job to see incremental replication, reading the database write ahead log instead of rescanning the table.

Step 10: Register Your Own Source and Destination

To replicate your own data, register a new source with + Create Source and a new destination with + Create Destination, then create a job that connects them.

• Source - choose the database type (PostgreSQL, MySQL, MongoDB, Oracle and more) and enter its host, port, database, user and password. OLake discovers the available tables and columns so you can select which streams to replicate and whether to use full refresh or change data capture.
• Destination - choose Apache Iceberg or Parquet and provide the object store bucket, region, path and credentials. The bundled object store is a convenience for the demo; point new destinations at your own Amazon S3 bucket or Iceberg catalogue for production data.

The bundled demo source and object store can be left in place or deleted once you have your own pipelines running.

Step 11: Configure System Settings

The Settings view under System configures global behaviour such as logging, data retention, job defaults and outgoing webhook alerts for notifying Slack, Teams or a custom endpoint when a job succeeds or fails.

Step 12: Review the Dedicated Data Volume

The container image layers, the containerd snapshots and the OLake container volumes sit on their own EBS data volume, mounted at /var/lib/docker by filesystem UUID so the layout survives reboots and instance replacement. This keeps several gigabytes of image and volume data off the operating system disk.

findmnt -no SOURCE,TARGET,FSTYPE,SIZE /var/lib/docker

The mount reports the dedicated volume:

/dev/nvme1n1 /var/lib/docker ext4 48.9G

The volume can be resized independently in the EC2 console without touching the operating system disk. If you replicate large tables into the bundled object store, grow this volume so the Parquet output and container storage have room.

Step 13: Understand the Bundled Demo Source and Object Store

The bundled demo components exist so the console is useful immediately, and both are addressed on the internal Docker gateway (172.17.0.1), reachable from the connector containers but not from outside the instance:

• Sample Postgres - a PostgreSQL 16 database on the gateway at port 55432, database olake_demo, seeded with the synthetic demo.customers and demo.orders tables. Its password is recorded in the credentials file.
• Local Lakehouse - an S3 compatible object store on the gateway at port 8333, bucket olake-lake. Its access and secret keys are recorded in the credentials file and anonymous access is disabled.

These are ordinary components: delete them from the console whenever you are ready to work only with your own systems.

Enabling HTTPS

The console is published over plain HTTP on port 80 so it works immediately on launch. For any access beyond a trusted network, terminate TLS in front of it. The nginx site is at /etc/nginx/sites-available/olake. Two common approaches:

• Put the instance behind an AWS Application Load Balancer with an ACM certificate and forward port 443 to the instance's port 80, restricting the instance security group to the load balancer.
• Obtain a certificate for your own domain and add a TLS server block to the nginx site, then reload nginx.

Restrict the security group so port 80 is reachable only from your load balancer or management network once TLS is in place.

Backup and Maintenance

• Operating system updates - the host runs Ubuntu 24.04 with unattended security upgrades enabled. Apply the latest packages periodically with sudo apt-get update && sudo apt-get upgrade.
• Job and metadata state - OLake's job definitions, schedules and run history live in the metadata database inside the container stack, on the /var/lib/docker data volume. Snapshot that EBS volume to back the state up.
• Replicated output - for production, point destinations at your own Amazon S3 bucket or Iceberg catalogue, which you back up with your existing S3 or lakehouse tooling, rather than relying on the bundled demo object store.
• Restart the stack - the whole container stack is one systemd unit. Restart it with sudo systemctl restart olake if you ever need to cycle it.

Support

This image is published and supported by cloudimg. For assistance with deployment or operation, contact support@cloudimg.co.uk. Include the instance ID, the AWS Region, and any relevant output from sudo docker ps and sudo journalctl -u olake when you report an issue.

OLake is open source software licensed under the Apache License 2.0. cloudimg is not affiliated with, endorsed by, or sponsored by OLake, Datazip, Temporal, SeaweedFS or the PostgreSQL Global Development Group. The OLake, Temporal, SeaweedFS and PostgreSQL names are used nominatively to identify the open source software in this image.