Conduit on Ubuntu 24.04 on Azure User Guide

Overview

Conduit is an open source data streaming engine that moves data in real time between data stores and message systems. It connects sources and destinations, Apache Kafka, PostgreSQL, Amazon S3, files and many more, into declarative pipelines that stream change data continuously, with built in connectors and processors and no JVM required. Conduit exposes an HTTP REST API and an interactive OpenAPI/Swagger web UI, so you can create, start, inspect and monitor pipelines and connectors directly from the browser or from your own automation. It is a single Go binary managed by systemd, so it stays lightweight and starts instantly.

The cloudimg image installs Conduit 0.17.0 from the pinned upstream release, with its checksum verified, and puts the HTTP API and web UI behind an nginx reverse proxy. Because the Conduit API and UI have no built in authentication, the appliance is secure by default: Conduit's HTTP and gRPC APIs are bound to loopback only, and nginx fronts them on port 80 behind an HTTP Basic Auth gate whose password is generated uniquely and at random on the first boot of every VM, so no shared or default credential is ever baked into the image. A fully self contained demo pipeline generates sample records and logs them locally, so the engine is running and processing records on first boot, with nothing leaving the VM. You replace the demo pipeline with your own whenever you are ready.

What is included:

• Conduit 0.17.0 single Go binary (/usr/local/bin/conduit), installed from the pinned upstream release with its SHA256 verified
• conduit.service running as the dedicated conduit user, with the HTTP API and Swagger UI bound to loopback 127.0.0.1:8080 and the gRPC API to 127.0.0.1:8084
• nginx.service reverse proxy on TCP 80 with a per VM HTTP Basic Auth gate (user admin), TLS ready
• conduit-data-streaming-firstboot.service generating the per VM password and resolving this VM's URL into the MOTD and a root only info note on first boot
• An unauthenticated static /healthz endpoint for load balancer and probe checks
• A self contained demo pipeline in /etc/conduit/pipelines/demo.yaml so the engine is running and producing records out of the box
• Pipeline state held in a dedicated data directory /var/lib/conduit
• Ubuntu 24.04 LTS base, latest patches, unattended security upgrades enabled

Prerequisites

An active Azure subscription, an SSH key, and a VNet with a subnet. Recommended VM size: Standard_B2s (Conduit is light for the demo pipeline; scale up for heavier streaming workloads).

Step 1: Deploy from the Azure Portal

Search the Azure Marketplace for Conduit, choose the plan, and create the VM. In the networking step attach an NSG that allows inbound TCP 22 (SSH) and TCP 80 (the API and UI, behind HTTP Basic Auth) from your client networks only. Put a TLS reverse proxy or certificate in front of port 80 for production.

Step 2: Deploy from the Azure CLI

RG="conduit-prod"; LOCATION="eastus"; VM_NAME="conduit-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/conduit-data-streaming-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name cdt-vnet --address-prefix 10.109.0.0/16 --subnet-name cdt-subnet --subnet-prefix 10.109.1.0/24
az network nsg create -g "$RG" --name cdt-nsg
az network nsg rule create -g "$RG" --nsg-name cdt-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name cdt-nsg --name allow-web --priority 110 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 80 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_B2s --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name cdt-vnet --subnet cdt-subnet --nsg cdt-nsg --public-ip-sku Standard

Step 3: Connect via SSH

ssh azureuser@<vm-ip>

conduit.service, nginx.service and conduit-data-streaming-firstboot.service all start automatically on first boot.

Step 4: Retrieve the Per-VM Password

On first boot the appliance generates a random HTTP Basic Auth password for user admin, unique to this VM, and writes it to a root only file. There is no shared or default credential in the image. Read it over SSH:

sudo cat /root/conduit-data-streaming-credentials.txt

The file holds this VM's URL, username and password:

CONDUIT_URL=http://<vm-ip>/
CONDUIT_USERNAME=admin
CONDUIT_PASSWORD=<unique-per-vm-password>

Keep this password safe — it is the only credential for the API and web UI, and every VM gets a different one.

Step 5: Verify the Services

sudo systemctl is-active conduit nginx conduit-data-streaming-firstboot
sudo test -f /var/lib/cloudimg/conduit-data-streaming-firstboot.done && echo FIRSTBOOT_DONE
sudo ss -tlnp | grep -E ':80 |:8080 |:8084 '

Expected output — all three services are active, nginx is bound to the public port 80 and the Conduit APIs are bound to loopback 127.0.0.1:8080 and 127.0.0.1:8084 only:

active
active
active
FIRSTBOOT_DONE
LISTEN 0 4096 127.0.0.1:8080 0.0.0.0:*  users:(("conduit",...))
LISTEN 0 4096 127.0.0.1:8084 0.0.0.0:*  users:(("conduit",...))
LISTEN 0 511  0.0.0.0:80      0.0.0.0:*  users:(("nginx",...))

Step 6: Secure by Default — the Auth Gate

The Conduit API and UI have no built in login, so nginx fronts them with an HTTP Basic Auth gate. An unauthenticated /healthz is available for load balancer probes, but the API and UI are gated: unauthenticated and wrong password requests are rejected with 401, and only the per VM password is accepted. Read your password from the file and prove the gate:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'GET /healthz (unauth)            -> HTTP %{http_code}\n' http://127.0.0.1/healthz
curl -s -o /dev/null -w 'GET /v1/pipelines (no cred)      -> HTTP %{http_code}\n' http://127.0.0.1/v1/pipelines
curl -s -o /dev/null -w 'GET /v1/pipelines (wrong pw)     -> HTTP %{http_code}\n' -u admin:wrong http://127.0.0.1/v1/pipelines
curl -s -o /dev/null -w 'GET /v1/pipelines (per-VM pw)    -> HTTP %{http_code}\n' -u "admin:$PW" http://127.0.0.1/v1/pipelines

The health probe is open, the API is gated, a wrong password is rejected, and the per VM password authenticates:

GET /healthz (unauth)            -> HTTP 200
GET /v1/pipelines (no cred)      -> HTTP 401
GET /v1/pipelines (wrong pw)     -> HTTP 401
GET /v1/pipelines (per-VM pw)    -> HTTP 200

Step 7: Open the Swagger Web UI

Open http://<vm-ip>/openapi/ in a browser and sign in as admin with the password from Step 4. Conduit serves an interactive OpenAPI/Swagger UI that documents every endpoint of the HTTP API and lets you call it directly from the browser — list pipelines and connectors, create and start pipelines, and inspect the running engine.

Step 8: Inspect the Pipelines Endpoint

Expand GET /v1/pipelines in the Pipelines service. Swagger UI shows the operation's parameters and response schema. Click Try it out, then Execute, to call the live API from the browser.

Step 9: See the Live Demo Pipeline

The response from Execute shows the running demo pipeline, demo-generator-to-log, with its state reported as STATUS_RUNNING and its two connectors listed. This is the self contained generator to log pipeline that ships in the image, proving the engine is streaming records out of the box.

Step 10: Inspect the Connectors

Expand GET /v1/connectors and Execute it the same way to see the demo pipeline's connectors: a builtin:generator source that produces sample records and a builtin:log destination that logs them locally. Each connector shows its plugin, type and settings.

Step 11: Check the Pipeline via the API

Everything the UI shows is available as JSON from the HTTP API, so you can wire Conduit's state into your own tooling. Read your password from the file and query the pipelines:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/v1/pipelines \
  | jq '[.[] | {id: .id, status: .state.status, connectors: .connectorIds}]'

The demo pipeline reports as running with its two connectors:

[
  {
    "id": "demo-generator-to-log",
    "status": "STATUS_RUNNING",
    "connectors": [
      "demo-generator-to-log:source-generator",
      "demo-generator-to-log:destination-log"
    ]
  }
]

Step 12: The Demo Pipeline Config

The demo pipeline is defined in one declarative file, /etc/conduit/pipelines/demo.yaml. Check the Conduit version and view the config:

conduit version
cat /etc/conduit/pipelines/demo.yaml

The config wires a builtin:generator source to a builtin:log destination:

v0.17.0 linux/amd64
version: 2.2
pipelines:
  - id: demo-generator-to-log
    status: running
    ...
    connectors:
      - id: source-generator
        type: source
        plugin: builtin:generator
        ...
      - id: destination-log
        type: destination
        plugin: builtin:log

Step 13: Customize the Pipeline

Replace the demo pipeline with your own. Add or edit a pipeline file under /etc/conduit/pipelines/, for example to stream from a Postgres source to a Kafka destination:

version: 2.2
pipelines:
  - id: pg-to-kafka
    status: running
    connectors:
      - id: source-pg
        type: source
        plugin: builtin:postgres
        settings:
          url: "postgres://user:pass@db-host:5432/mydb?sslmode=require"
          tables: "public.orders"
      - id: dest-kafka
        type: destination
        plugin: builtin:kafka
        settings:
          servers: "broker:9092"
          topic: "orders"

Apply your changes with sudo systemctl restart conduit. Conduit provisions the pipelines from the files on start. The full connector and processor reference is at conduit.io/docs.

Step 14: Managing the Service

sudo systemctl show conduit --property=ActiveState,SubState,MainPID,NRestarts

Conduit is managed by systemd, so it restarts automatically on failure and on boot:

ActiveState=active
SubState=running
MainPID=...
NRestarts=0

Restart it after editing your pipelines with sudo systemctl restart conduit. Logs, including the demo pipeline's record output, are available with sudo journalctl -u conduit -f.

Step 15: Add TLS (Optional)

For production, terminate TLS at nginx. Point a DNS record at the VM, then use the packaged nginx with a certificate from Let's Encrypt:

sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d <your-domain>

certbot installs the certificate, rewrites the nginx server block to listen on 443 and sets up automatic renewal. The HTTP Basic Auth gate stays in front of the API and UI over TLS.

Step 16: Security Recommendations

• Restrict the NSG so ports 80 and 22 only reach trusted networks.
• Terminate TLS at nginx (Step 15) so the API, UI and its Basic Auth credentials travel encrypted.
• Rotate the Basic Auth password if needed by regenerating /etc/nginx/.conduit.htpasswd with sudo htpasswd -B /etc/nginx/.conduit.htpasswd admin and reloading nginx.
• Keep the API private — the Conduit API can create, start and stop pipelines, so treat access as privileged even behind Basic Auth.
• Review your connectors so data only flows to the sources and destinations you intend, and keep connector credentials in the pipeline settings secured.
• Patch the OS monthly with sudo apt-get update && sudo apt-get upgrade; unattended security upgrades are already enabled.

Step 17: Support and Licensing

Conduit is licensed under the Apache License 2.0 — no per CPU or per user fee. cloudimg provides commercial support separately.

• Email: support@cloudimg.co.uk
• Website: www.cloudimg.co.uk
• Support hours: 24/7, 24h response SLA

Deploy on Azure

Launch Conduit on Ubuntu 24.04 with 24/7 support from cloudimg.

View on Marketplace

Need Help?

Our support team is available 24/7. support@cloudimg.co.uk