Conduit Data Streaming on AWS User Guide

Overview

This image runs Conduit, an open source data streaming engine that moves data in real time between data stores and message systems. Conduit connects sources and destinations - Apache Kafka, PostgreSQL, Amazon S3, files and many more - into declarative pipelines that stream change data continuously, with built-in connectors and processors and no JVM required. It exposes an HTTP REST API and an interactive OpenAPI/Swagger web UI so you can create, start, inspect and monitor pipelines and connectors from the browser or from automation.

Conduit is installed as a single Go binary, pinned to version 0.17.0 (github.com/ConduitIO/conduit, Apache-2.0 licensed), downloaded from the official GitHub release and verified against its published SHA256 checksum before installation - no source build and no runtime interpreter. It runs as a dedicated conduit system user under systemd (conduit.service), with its badger state database and connectors living under /var/lib/conduit.

The image is secure by default. Conduit's HTTP API and Swagger UI have no built-in authentication, so exposing them directly would be unsafe. This image binds both the HTTP API (127.0.0.1:8080) and the gRPC API (127.0.0.1:8084) to loopback only, and fronts the HTTP API and UI with nginx on port 80 behind an HTTP Basic Auth gate for user admin. On the first boot of every deployed instance, a one-shot service generates a fresh random 24-character Basic Auth password, bcrypt-hashes it into the nginx password file, writes the plaintext to /root/conduit-data-streaming-credentials.txt (mode 0600, root only), and then proves the whole stack end to end before handing over. Two instances launched from the same AMI never share a password, and no credential of any kind is baked into the image. An unauthenticated /healthz endpoint is provided for load-balancer probes.

The image also ships a fully self-contained demo pipeline. A builtin:generator source produces sample records once a second and forwards them to a builtin:log destination that logs them locally. Nothing leaves the instance, the pipeline is running and processing records on first boot, and the Swagger UI shows a live working pipeline out of the box - so you can explore a real pipeline immediately and then replace it with your own source and destination.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and inbound port 80 (and 443 if you add TLS) from the networks your operators will reach the Conduit UI on
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Conduit. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that allows inbound port 22 from your management network and inbound port 80 from the networks your operators use. Leave the root volume at the default size or larger.

Select Launch instance. First-boot initialisation takes well under a minute after the instance state becomes Running and the status checks pass.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Conduit Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --metadata-options 'HttpTokens=required,HttpEndpoint=enabled' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=conduit}]'

Connecting to Your Instance

Connect over SSH using the default login user for the operating system variant you launched. Replace <instance-public-ip> with the public IP address or DNS name shown in the EC2 console, and <key.pem> with the path to your private key.

Step 3: Retrieve Your Per-Instance Credentials

The Basic Auth password is generated uniquely on the first boot of your instance and written to a root-only file. Read it over SSH:

sudo cat /root/conduit-data-streaming-credentials.txt

The file contains the web URL, the username (admin) and the per-instance password. Keep it secret. Confirm the installed version and that both services are running:

conduit version; systemctl is-active conduit nginx

You should see v0.17.0 and both services reported as active.

Step 4: Verify the Secure-by-Default Binding

Confirm that Conduit is bound to loopback only and that nginx is the sole network-facing listener on port 80:

sudo ss -tlnp | grep -E ':(80|8080|8084)\b'

conduit listens on 127.0.0.1:8080 (HTTP API) and 127.0.0.1:8084 (gRPC) - loopback only - while nginx listens on 0.0.0.0:80. The Conduit API is never reachable off the instance except through the nginx Basic Auth gate.

Now confirm the unauthenticated health endpoint answers, and that the API itself is password-protected. An unauthenticated request to the API returns 401:

curl -s -o /dev/null -w 'healthz=%{http_code}\n' http://127.0.0.1/healthz
curl -s -o /dev/null -w 'api-unauth=%{http_code}\n' http://127.0.0.1/v1/pipelines

The health endpoint returns healthz=200 and the unauthenticated API request returns api-unauth=401 - every API call must present the per-instance password.

Step 5: Confirm the Demo Pipeline is Running Through the API

Read the per-instance password from the credentials file into a shell variable, then list the pipelines through the authenticated API:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/v1/pipelines | jq '.[] | {id, status: .state.status}'

You will see the demo-generator-to-log pipeline with status STATUS_RUNNING. Now confirm records are actually flowing by reading Conduit's Prometheus metrics - the pipeline execution counter is greater than zero:

PW=$(sudo grep '^CONDUIT_PASSWORD=' /root/conduit-data-streaming-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/metrics | awk '/^conduit_pipeline_execution_duration_seconds_count/{s+=$NF} END{print "records processed:", s+0}'

The count is a positive, growing number - the engine is processing the demo records out of the box.

Step 6: Sign in to the Swagger Web Interface

Browse to http://<instance-public-ip>/ and sign in with the username admin and the password from /root/conduit-data-streaming-credentials.txt. You reach the Conduit OpenAPI/Swagger UI, a browser front for the full HTTP API where you can list, create, start, inspect and delete pipelines and connectors.

Expand GET /v1/pipelines, choose Try it out and Execute to see the live demo pipeline. The 200 response shows demo-generator-to-log in STATUS_RUNNING with its source and destination connectors - a real pipeline processing records with nothing leaving the instance.

Expand GET /v1/connectors/plugins and execute it to browse the built-in connector catalog - Kafka, PostgreSQL, Amazon S3, files and more - the sources and destinations you can wire into your own pipelines.

Step 7: Replace the Demo Pipeline with Your Own

Pipelines are defined as YAML files under /etc/conduit/pipelines. The shipped demo.yaml is a self-contained example; replace it with your own source and destination. For instance, to stream from Apache Kafka into PostgreSQL you would write a pipeline using the builtin:kafka source and a builtin:postgres destination. Edit the pipeline file with your preferred editor - for example sudo nano /etc/conduit/pipelines/demo.yaml - and then apply it by restarting Conduit:

sudo systemctl restart conduit

Conduit reloads every pipeline under /etc/conduit/pipelines on start. You can also create and start pipelines dynamically through the HTTP API (POST /v1/pipelines, POST /v1/connectors, POST /v1/pipelines/{id}/start) directly from the Swagger UI, without editing files.

Step 8: Add Your Own Domain and TLS

For production, put Conduit behind a real domain and a certificate from a trusted authority so the Basic Auth password and API traffic are encrypted in transit. Point a DNS record at your instance, open port 443 in your security group, then obtain a certificate - for example with certbot for your-domain.example.com - and add a TLS server block to the nginx site at /etc/nginx/sites-available/cloudimg-conduit that proxies to the same 127.0.0.1:8080 upstream. Reload nginx after any change:

sudo nginx -t && sudo systemctl reload nginx

Service Management

Both services are managed by systemd:

systemctl status conduit nginx

Conduit's logs are available through the journal:

sudo journalctl -u conduit -n 50 --no-pager

Restart the stack after any configuration change:

sudo systemctl restart conduit nginx

Support

Every subscription includes 24/7 technical support from cloudimg engineers via email and live chat, covering deployment, first-boot credential retrieval, upgrades to new Conduit releases, pipeline and connector configuration for Kafka, PostgreSQL, Amazon S3 and other sources and destinations, reverse proxy and TLS termination, bringing your own domain, and API and Basic Auth administration. Email support@cloudimg.co.uk. Critical issues receive a one-hour average response time.