Streaming & Messaging AWS

Conduit 0.21.2 on AWS User Guide

Last updated: 2026-10-29 | Product: Conduit 0.21.2 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Conduit 0.21.2 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Conduit is a data streaming engine that moves data between systems in real time. A pipeline joins one or more sources (PostgreSQL change data capture, Kafka topics, Amazon S3, files, a test data generator) to one or more destinations, with processors in between that filter, reshape or enrich each record. Pipelines are declared in YAML or created through Conduit's REST and gRPC APIs, and Conduit ships with a web UI that shows every pipeline and its live record flow.

What the cloudimg image adds:

Conduit 0.21.2, the release that closes three upstream security advisories. Conduit 0.20.5 and earlier served an unauthenticated API, let pipeline settings read the engine's environment and local files and make server-side requests, and passed the engine's environment to plugins. 0.21 fixes all three; this image runs 0.21.2 (0.21.0 refused pipelines without a dead-letter queue and 0.21.1 was never built). See "What changed since Conduit 0.17" below.
API token authentication switched on. Every HTTP request and every gRPC call to Conduit needs this instance's API token, also from processes on the instance itself. The token is generated on your instance at first boot and read from a root-only file; Conduit refuses to start without it.
Nothing reachable off the instance except SSH and HTTPS. Conduit's HTTP API (8080) and gRPC API (8084) listen on 127.0.0.1 only, with Conduit's own DNS-rebinding, cross-origin and content-type checks on. nginx is the only service on a network address.
The UI, Swagger UI and REST API over HTTPS only. nginx serves them on port 443 (port 80 only redirects) in two ways: sign in with the admin login (HTTP Basic, the password generated at first boot and stored only as a bcrypt hash) and nginx adds the API token for you, or send the API token yourself as a bearer token. Failed logins and wrong tokens are rate limited, requests for host names the instance does not serve are dropped, other sites' Origin headers are refused, and the live record streams (WebSockets) are served only to pages on this instance.
A sandboxed engine. Whoever holds the login or the token creates pipelines, and pipelines open connections and read files. So ${VAR} references in pipeline settings expand nothing, the file connector is confined to /var/lib/conduit/files, the built-in HTTP processors refuse private, loopback and metadata addresses, cloud credentials of the instance are never used, and Conduit runs as the unprivileged conduit user with systemd sandboxing: the credentials file, the TLS key and the login hash are unreadable, and the engine (with any plugin it starts) cannot reach the instance metadata service.
No default secret. The login, the API token and the TLS private key are generated on your instance at first boot; the image ships none of them, and Conduit starts with an empty pipeline database.
HTTPS from the first boot. The certificate is generated on your instance for its own names and addresses. It is an end-entity certificate: it can only vouch for this instance and can never sign other certificates. Install your own CA-issued certificate with one command (Step 13); the boot-time renewal never replaces it.
Hardened defaults. nginx hands Conduit the real client address and never the host or forwarding headers a client sends, Conduit logs without colour codes so its journal is plain text, SSH accepts keys only, and Ubuntu 24.04 LTS is fully patched at build time with unattended security upgrades switched on.

What is on the image: Conduit 0.21.2 (the official linux-x86_64 release, verified against the release's signed checksums file), with its built-in connectors (generator, file, log, Kafka, PostgreSQL, Amazon S3) and processors; nginx 1.24 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent. A demo pipeline (a test data generator feeding the log destination) runs from the first boot. No compiler toolchain is installed.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

The Conduit UI over HTTPS: the pipelines list with the demo pipeline and an orders pipeline, both running

Prerequisites

An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
A security group that allows TCP 22 (SSH) and TCP 443 (the UI and API) from your admin network, plus TCP 80 if you want plain HTTP requests redirected to HTTPS. There is nothing else to open: Conduit's own ports listen on the loopback address only and refuse connections from other machines even if a security group allows them. Pipelines open outbound connections to your sources and destinations (databases, Kafka brokers, S3), so allow that outbound traffic.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Conduit 0.21.2 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger for heavy pipelines), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=conduit}]'

First boot takes about a minute after the instance is running: the boot service generates the login, the API token and the certificate, then Conduit and nginx start.

Step 2: Connect to your instance

Version	SSH user
Conduit 0.21.2 on Ubuntu 24.04	`ubuntu`

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services and the listening set

systemctl is-active conduit-metadata-names conduit-firstboot conduit nginx
sudo test -f /var/lib/cloudimg/conduit-firstboot.done && echo FIRSTBOOT_DONE

All four units are active and the first boot marker is present:

active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network addresses; Conduit's HTTP API (8080), its gRPC API (8084) and its internal connector utilities server (a random port) listen on 127.0.0.1:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:38483 users:(("conduit"))
127.0.0.1:8080 users:(("conduit"))
127.0.0.1:8084 users:(("conduit"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

Step 4: Get the URL, login, API token and certificate fingerprint

The credentials file is readable by root only. It holds the URL, the admin user and password, the Conduit API token and the SHA-256 fingerprint of this instance's certificate:

sudo cat /root/conduit-credentials.txt

# Conduit 0.21.2 on Ubuntu 24.04 - generated on first boot by conduit-firstboot.service.
# The UI login and the Conduit API token are unique to this instance; none was baked into the image. Keep this file
# secret. Whoever has either of them can create pipelines, which read from and write to the systems you connect and
# make outbound connections from this instance: share them only with people you would trust with those systems.
# nginx stores the password only as a bcrypt hash (/etc/nginx/conduit-htpasswd). The UI, Swagger UI (/openapi/) and
# REST API (/v1/) are served over HTTPS only (port 443; port 80 redirects). The certificate is self signed (minted on
# this instance): before you sign in, check that the SHA-256 fingerprint your browser shows matches
# CONDUIT_TLS_CERT_SHA256 below - or install a CA-issued certificate with
#   sudo conduit-install-cert <chain.pem> <key.pem>
# Change the served name: sudo conduit-set-hostname <dns-name>      Re-verify: sudo conduit-selfcheck
CONDUIT_URL=https://<public-ip>/
CONDUIT_HOST=<public-ip>
CONDUIT_UI_USER=admin
CONDUIT_UI_PASSWORD=<CONDUIT_UI_PASSWORD>
CONDUIT_API_TOKEN=<api-token>
CONDUIT_API_TOKEN_FILE=/etc/conduit/api-token
CONDUIT_TLS_CERT_SHA256=<sha256-fingerprint>
CONDUIT_TLS_CERT_FILE=/etc/cloudimg/conduit-ui.pem
# REST API over HTTPS with the token (curl reads it from a file, never the command line):
#   sudo curl -K /root/conduit-api.curl --cacert /etc/cloudimg/conduit-ui.pem --resolve <public-ip>:443:127.0.0.1 https://<public-ip>/v1/pipelines
# CLI (gRPC on 127.0.0.1:8084):  sudo conduit pipelines ls --api.token-file /etc/conduit/api-token
# User guide: https://www.cloudimg.co.uk/guides/conduit-0-21-2-on-ubuntu-24-04-aws/

The URL, the user, the token file and the fingerprint (without the password or the token):

sudo grep -E '^CONDUIT_(URL|UI_USER|API_TOKEN_FILE|TLS_CERT_SHA256)=' /root/conduit-credentials.txt

CONDUIT_URL=https://<public-ip>/
CONDUIT_UI_USER=admin
CONDUIT_API_TOKEN_FILE=/etc/conduit/api-token
CONDUIT_TLS_CERT_SHA256=<sha256-fingerprint>

The host is the instance's public IP address from the EC2 instance metadata (the private address when the instance has no public one). The certificate also names the public DNS name, the private IP address and the private DNS name. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it (unless you installed your own, Step 13).

The certificate's names and constraints, read from the instance:

openssl x509 -in /etc/cloudimg/conduit-ui.pem -noout -ext subjectAltName,basicConstraints,keyUsage,extendedKeyUsage

X509v3 Subject Alternative Name:
    IP Address:<public-ip>, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<private-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication

Step 5: Open the UI and the Swagger UI

Browse to the CONDUIT_URL from Step 4. The certificate is self signed and unique to this instance, so the browser warns about it the first time. Check the certificate before you continue: open the browser's certificate viewer for the page and compare its SHA-256 fingerprint with CONDUIT_TLS_CERT_SHA256 from the credentials file. Continue only if they match, and only for this site. Never add this certificate to an operating-system or browser trust store; for a certificate browsers trust without a warning, install a CA-issued one (Step 13).

The browser then asks for a user name and password: sign in as admin with the password from the credentials file. To print just the password on the instance:

sudo awk -F= '$1=="CONDUIT_UI_PASSWORD" {print $2}' /root/conduit-credentials.txt

The Pipelines page lists every pipeline with its state (screenshot at the top of this guide). Conduit 0.21.2's UI always shows a yellow banner saying No auth configured: upstream renders it on every page whatever the server's settings, so it does not describe this instance, where every request needs the login or the API token (Step 6 shows the refusals). Dismiss it with ×.

Select a pipeline to see its sources, processors and destinations with their record rates, and its Live record flow, which streams the records passing through each stage:

A pipeline's page: the generator source, the field.set processor and the log destination at 2 records per second, with the live record flow below

View opens a record with its key, payload, metadata and how each stage changed it:

A record opened from the live record flow: position, operation, key, metadata and the per-stage diff

The Swagger UI at /openapi/ (behind the same login) describes the whole REST API and lets you call it from the browser with Try it out:

The Swagger UI over HTTPS: GET /v1/pipelines executed with Try it out, answering 200 with the pipeline list

HTTP Basic authentication has no sign-out button: close the browser to end the session on a shared computer.

Step 6: The REST API, the CLI and Conduit's own protections

Every API call needs this instance's API token. There are three ways to call Conduit:

The REST API over HTTPS, from anywhere that can reach port 443, with the token as a bearer token. /root/conduit-api.curl holds it as a curl header line, so the token never appears on a command line.
The REST API on the instance, directly on http://127.0.0.1:8080, with the same token.
The conduit command-line tool on the instance, which talks to the gRPC API on 127.0.0.1:8084 and reads the token from /etc/conduit/api-token (--api.token-file, so the token is never in the process list).

On the instance the examples pin the instance's public name to 127.0.0.1 with --resolve, so they work whatever the security group allows, and check the server against this instance's certificate:

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem https://$H/v1/pipelines \
  | python3 -c 'import json,sys; [print(p["id"], p["state"]["status"]) for p in json.load(sys.stdin)]'
curl -s -o /dev/null -w 'over HTTPS without the login or the token: %{http_code}\n' --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem https://$H/v1/pipelines
sudo conduit pipelines ls --api.token-file /etc/conduit/api-token

demo-generator-to-log STATUS_RUNNING
over HTTPS without the login or the token: 401
+-----------------------+---------+----------------------+----------------------+
|          ID           |  STATE  |       CREATED        |     LAST_UPDATED     |
+-----------------------+---------+----------------------+----------------------+
| demo-generator-to-log | running | 2026-10-29T16:52:19Z | 2026-10-29T16:52:19Z |
+-----------------------+---------+----------------------+----------------------+

Conduit 0.21 refuses requests itself, before any of its own code runs, when the token is missing or wrong, when the Host header is not a loopback name (a DNS-rebinding attack from a web page), when the request comes from another site's page (Origin), or when a change is not sent as JSON (a cross-site form post). Each refusal carries its error code:

why() { python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["message"], "/", d["details"][0]["reason"])'; }
printf 'no token:           '; curl -s http://127.0.0.1:8080/v1/pipelines | why
printf 'foreign Host:       '; sudo curl -s -K /root/conduit-api.curl -H 'Host: rebind.example' http://127.0.0.1:8080/v1/pipelines | why
printf 'foreign Origin:     '; sudo curl -s -K /root/conduit-api.curl -H 'Origin: https://evil.example' http://127.0.0.1:8080/v1/pipelines | why
printf 'form-style POST:    '; sudo curl -s -K /root/conduit-api.curl -X POST -H 'Content-Type: text/plain' --data '{}' http://127.0.0.1:8080/v1/pipelines | why
printf 'gRPC without token: '; { conduit pipelines ls 2>&1 || true; } | tail -1 | sed -E 's/.*code = ([A-Za-z]+) desc = (.*)/\2 \/ gRPC \1/'

no token:           missing or invalid API token / api.unauthenticated
foreign Host:       the Host header is not allowed for this API / api.host_forbidden
foreign Origin:     the request Origin is not allowed / api.origin_forbidden
form-style POST:    mutating requests must use Content-Type: application/json / api.content_type_unsupported
gRPC without token: missing or invalid API token / gRPC Unauthenticated

From your workstation, copy the certificate (Step 4) and keep the token in a file only you can read, then call the API the same way:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/cloudimg/conduit-ui.pem conduit.pem
openssl x509 -in conduit.pem -noout -fingerprint -sha256      # must match CONDUIT_TLS_CERT_SHA256
( umask 077; printf 'header = "Authorization: Bearer %s"\n' "<your-token>" > conduit-api.curl )
curl -K conduit-api.curl --cacert conduit.pem https://<public-ip>/v1/pipelines

The live record streams (/v1/connectors/<id>/inspect and /v1/processors/<id>/inspect-in or inspect-out) are WebSockets that the UI opens; a script calling them must send Sec-Fetch-Site: same-origin, because nginx serves them only to pages of this instance.

Step 7: Create a pipeline from YAML

Pipelines are declared in YAML. This one generates two order records per second, tags each with a region in its metadata and writes them to the log destination. Every pipeline should name a dead-letter queue, where records that fail go: this one stops the pipeline on the first failed record (window-nack-threshold: 0) instead of acknowledging it, so nothing is lost. Conduit 0.21 refuses a dead-letter queue that would acknowledge failed records while keeping only hashes of them.

mkdir -p ~/conduit-demo && cd ~/conduit-demo
cat > orders.yaml <<'YAML'
version: "2.2"
pipelines:
  - id: orders-demo
    status: running
    name: orders-demo
    description: Generated orders, enriched with a field, logged at debug level
    connectors:
      - id: orders-source
        type: source
        plugin: builtin:generator
        settings:
          format.type: structured
          format.options.order_id: int
          format.options.customer: string
          rate: "2"
          sdk.schema.extract.key.enabled: "false"
      - id: orders-log
        type: destination
        plugin: builtin:log
        settings:
          level: debug
          sdk.schema.extract.key.enabled: "false"
    processors:
      - id: tag-region
        plugin: field.set
        settings:
          field: .Metadata.region
          value: eu-west-1
    dead-letter-queue:
      plugin: builtin:log
      settings:
        level: warn
      window-size: 1
      window-nack-threshold: 0
YAML
sudo conduit pipelines apply orders.yaml --yes --api.token-file /etc/conduit/api-token

Plan for pipeline "orders-demo" (hash 35e6495d):
  [+] create pipeline  orders-demo          in place
  [+] create connector orders-demo:orders-demo:orders-source in place
  [+] create connector orders-demo:orders-demo:orders-log in place
  [+] create processor orders-demo:orders-demo:tag-region in place

4 to create, 0 to update, 0 to delete.

Start it, then read its dead-letter queue setting back and its record counters (Conduit shows setting values as *** in API answers):

sudo conduit pipelines start orders-demo --api.token-file /etc/conduit/api-token
sleep 5
H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem https://$H/v1/pipelines/orders-demo/dead-letter-queue; echo
sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem https://$H/metrics \
  | awk '/^conduit_connector_bytes_count\{.*orders-demo/ {sub(/.*component_id="/, ""); sub(/",.* /, " records="); print}'

Pipeline orders-demo  [running]
{"plugin":"builtin:log","settings":{"level":"***"},"windowSize":"1","windowNackThreshold":"0"}
orders-demo:orders-demo:orders-log records=11
orders-demo:orders-demo:orders-source records=11

Open the pipeline in the UI to watch the records (Step 5). conduit pipelines apply keeps a pipeline in step with its file: edit the file and apply it again to change the pipeline (stop it first if Conduit says the change needs a restart). The file is the pipeline's source of truth, so the API and the UI cannot delete it. To keep it across restarts, install the file where Conduit loads pipelines at every start; Conduit drops applied pipelines whose file is not there when it restarts:

sudo install -m 0644 orders.yaml /etc/conduit/pipelines/orders.yaml
sudo systemctl restart conduit

/etc/conduit/pipelines/demo.yaml holds the demo pipeline the same way: remove it and restart Conduit to drop the demo. Pipelines you create through the REST API or the UI are kept in Conduit's database instead and can be changed and deleted through the API.

Step 8: Connect real sources and destinations over TLS

Point the built-in connectors at your systems in the same YAML (or through the API). Keep their transport encrypted and pin the server's certificate for that client only:

Kafka (builtin:kafka): set tls.enabled: "true" and give the broker's CA certificate (or, for a self-signed broker, the broker's own certificate) in the connector's caCert setting as PEM text. Check its fingerprint with the broker's administrator first, and never set insecureSkipVerify. Conduit 0.21 refuses SASL without TLS.
PostgreSQL (builtin:postgres): use sslmode=verify-full in the connection URL and name the CA certificate with sslrootcert. Certificate files must be in /var/lib/conduit/secrets; Conduit refuses paths anywhere else.

Copy a CA certificate into the secrets directory as the conduit user, and pin a self-signed server certificate only in that connector, never in the instance's system trust store:

openssl x509 -in db-ca.pem -noout -fingerprint -sha256        # compare with the database administrator's value
sudo -u conduit tee /var/lib/conduit/secrets/db-ca.pem < db-ca.pem > /dev/null

A pipeline that streams changes from PostgreSQL into a Kafka topic (host names and credentials in angle brackets):

version: "2.2"
pipelines:
  - id: orders-cdc
    status: running
    connectors:
      - id: pg-orders
        type: source
        plugin: builtin:postgres
        settings:
          url: "postgres://<user>:${PG_PASSWORD}@<db-host>:5432/<database>?sslmode=verify-full&sslrootcert=/var/lib/conduit/secrets/db-ca.pem"
          tables: "public.orders"
      - id: kafka-orders
        type: destination
        plugin: builtin:kafka
        settings:
          servers: "<broker-1>:9093,<broker-2>:9093"
          topic: "orders"
          tls.enabled: "true"
          caCert: |
            -----BEGIN CERTIFICATE-----
            <the broker CA certificate, PEM>
            -----END CERTIFICATE-----
    dead-letter-queue:
      plugin: builtin:file
      settings:
        path: /var/lib/conduit/files/orders-cdc-dlq.jsonl
      window-size: 1
      window-nack-threshold: 0

Keep passwords out of pipeline files with ${NAME} references. Conduit expands only variables you list in env.allow, so add the name there and give Conduit the value from a root-only file:

sudo nano /etc/conduit/conduit.yaml                        # env: allow: [PG_PASSWORD]
sudo install -d -m 0755 /etc/systemd/system/conduit.service.d
sudo install -m 0600 /dev/null /etc/conduit/secrets.env    # then: sudo nano /etc/conduit/secrets.env -> PG_PASSWORD=...
printf '[Service]\nEnvironmentFile=/etc/conduit/secrets.env\n' | sudo tee /etc/systemd/system/conduit.service.d/secrets.conf
sudo systemctl daemon-reload && sudo systemctl restart conduit

Anyone who can create pipelines can use a listed variable's value in a setting, so list only variables meant for pipelines. By default built-in connectors may connect to any address except the cloud metadata endpoints; to restrict them, list the allowed targets under connectors.egress.allow in /etc/conduit/conduit.yaml (for example db.internal:5432 or 10.0.0.0/16:9093) and restart Conduit.

Step 9: The pipeline sandbox

Creating a pipeline means telling Conduit to open files and connections, so the image limits what a pipeline can reach. The file connector reads and writes only in /var/lib/conduit/files; it refuses everything else, including links and ../ paths that lead out of that directory. Inside the engine's sandbox /root and the TLS key cannot even be looked up:

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
api() { sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem -H 'Content-Type: application/json' "$@"; }
P=$(api --data '{"config":{"name":"sandbox-files"}}' https://$H/v1/pipelines | python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])')
for f in /root/conduit-credentials.txt /etc/conduit/api-token /etc/nginx/ssl/conduit.key /var/lib/conduit/files/../conduit.db; do
  api --data "{\"type\":\"TYPE_SOURCE\",\"plugin\":\"builtin:file\",\"pipelineId\":\"$P\",\"config\":{\"name\":\"f\",\"settings\":{\"path\":\"$f\"}}}" https://$H/v1/connectors \
    | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["details"][0]["reason"], "-", d["message"])'
done
api -X DELETE https://$H/v1/pipelines/$P >/dev/null && echo "pipeline sandbox-files deleted"

config.path_not_allowed - path "/root/conduit-credentials.txt" is not allowed: the path cannot be resolved
config.path_not_allowed - path "/etc/conduit/api-token" is not allowed: the path is outside every allowed directory
config.path_not_allowed - path "/etc/nginx/ssl/conduit.key" is not allowed: the path cannot be resolved
config.path_not_allowed - path "/var/lib/conduit/files/../conduit.db" is not allowed: the path is outside every allowed directory
pipeline sandbox-files deleted

The built-in HTTP processors (webhook.http, the Ollama, OpenAI and Cohere processors) refuse private, loopback, link-local and cloud metadata addresses at the moment they connect, whatever the URL looks like. Here a pipeline sends its record to the instance metadata service and is stopped:

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
api() { sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem -H 'Content-Type: application/json' "$@"; }
id() { python3 -c 'import json,sys; print(json.load(sys.stdin)["id"])'; }
P=$(api --data '{"config":{"name":"sandbox-metadata"}}' https://$H/v1/pipelines | id)
api --data "{\"type\":\"TYPE_SOURCE\",\"plugin\":\"builtin:generator\",\"pipelineId\":\"$P\",\"config\":{\"name\":\"g\",\"settings\":{\"format.type\":\"structured\",\"format.options.id\":\"int\",\"recordCount\":\"1\"}}}" https://$H/v1/connectors | id > /dev/null
api --data "{\"type\":\"TYPE_DESTINATION\",\"plugin\":\"builtin:log\",\"pipelineId\":\"$P\",\"config\":{\"name\":\"l\",\"settings\":{\"level\":\"debug\"}}}" https://$H/v1/connectors | id > /dev/null
api --data "{\"plugin\":\"builtin:webhook.http\",\"parent\":{\"type\":\"TYPE_PIPELINE\",\"id\":\"$P\"},\"config\":{\"settings\":{\"request.url\":\"http://169.254.169.254/latest/meta-data/\"}}}" https://$H/v1/processors | id > /dev/null
api -X POST --data '{}' https://$H/v1/pipelines/$P/start > /dev/null
for i in $(seq 1 30); do
  E=$(api https://$H/v1/pipelines/$P | python3 -c 'import json,sys; print(json.load(sys.stdin)["state"]["error"])')
  case "$E" in *"egress guard"*) break ;; esac; sleep 1
done
printf '%s\n' "$E" | grep -o 'outbound request refused by the egress guard.*'

outbound request refused by the egress guard (dial to 169.254.169.254:80): v4_link_local_or_metadata

Behind those checks, systemd blocks the metadata service for the whole engine (Step 11), and Conduit runs as the unprivileged conduit user that can write only its own state directory:

systemctl show -p User,NoNewPrivileges,ProtectSystem,ProtectHome,PrivateTmp,ReadWritePaths,MemoryDenyWriteExecute conduit.service

User=conduit
ReadWritePaths=/var/lib/conduit
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
NoNewPrivileges=yes
MemoryDenyWriteExecute=yes

Standalone connectors. The image runs only Conduit's built-in connectors and processors. A standalone connector is a separate program you install in /usr/local/lib/conduit/connectors (owned by root, so only an administrator can add one) and Conduit starts it as the conduit user: it gets a minimal environment, not Conduit's, and the sandbox and metadata block apply to it too, but it is native code that can read whatever the conduit user can read, including Conduit's configuration and API token. Install only connectors you trust, and keep secrets out of Conduit's environment unless a pipeline needs them.

Clean up the pipelines of Steps 7 and 9 (the applied orders-demo is stopped here; it disappears at the next restart of Conduit unless you installed its file in Step 7):

H=$(sudo sed -n 's/^CONDUIT_HOST=//p' /root/conduit-credentials.txt)
api() { sudo curl -s -K /root/conduit-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/conduit-ui.pem -H 'Content-Type: application/json' "$@"; }
ID=$(api https://$H/v1/pipelines | python3 -c 'import json,sys; print(next((p["id"] for p in json.load(sys.stdin) if p["config"]["name"] == "sandbox-metadata"), ""))')
if [ -n "$ID" ]; then
  api -X POST --data '{"force":true}' https://$H/v1/pipelines/$ID/stop > /dev/null; sleep 2
  for x in $(api https://$H/v1/pipelines/$ID | python3 -c 'import json,sys; d=json.load(sys.stdin); print(" ".join(["processors/" + i for i in d.get("processorIds") or []] + ["connectors/" + i for i in d.get("connectorIds") or []]))'); do
    api -X DELETE "https://$H/v1/$x" > /dev/null
  done
  api -X DELETE https://$H/v1/pipelines/$ID > /dev/null && echo "deleted sandbox-metadata"
fi
sudo conduit pipelines stop orders-demo --api.token-file /etc/conduit/api-token

deleted sandbox-metadata
Pipeline orders-demo  [stopped]

Step 10: Security checks

conduit-selfcheck re-proves the whole appliance with this instance's credentials. It checks the units and the per-instance files, the served certificate, the HTTPS front door (the login and the token, unknown host names, foreign origins, WebSocket rules, form posts, ; path parameters and encoded traversal), Conduit's own refusals on 127.0.0.1:8080 and on gRPC, and the exact listening set. It then creates, runs and stops a real pipeline over HTTPS with the token, and proves the sandbox from inside pipelines: the file connector refuses the credentials file, the token and the TLS key, ${VAR} references are refused, the HTTP processor cannot reach the metadata service in any of its spellings or Conduit's own API, and from inside Conduit's systemd unit the metadata service is unreachable while loopback answers. It cleans up after itself and takes about a minute:

sudo conduit-selfcheck | tail -n 1

SELFCHECK OK

Run it without | tail to see every check.

Step 11: The instance metadata block

Conduit (and so every plugin process it starts), nginx and the boot service are denied the instance metadata service (169.254.169.254) by systemd IPAddressDeny rules, so neither a pipeline nor a request to the front door can read the instance's IAM credentials. The only service allowed to reach the metadata service is conduit-metadata-names, which runs once at boot, reads the instance's host names and addresses (no credentials) for the certificate, and may reach nothing else:

for u in conduit nginx conduit-firstboot conduit-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u.service)" "$(systemctl show -p IPAddressAllow --value $u.service)"; done

conduit: deny=::ffff:168.63.129.16/128 169.254.169.254/32 ::ffff:169.254.169.254/128 fd00:ec2::254/128 168.63.129.16/32 allow=
nginx: deny=169.254.169.254/32 ::ffff:168.63.129.16/128 168.63.129.16/32 fd00:ec2::254/128 ::ffff:169.254.169.254/128 allow=
conduit-firstboot: deny=168.63.129.16/32 ::ffff:168.63.129.16/128 fd00:ec2::254/128 169.254.169.254/32 ::ffff:169.254.169.254/128 allow=
conduit-metadata-names: deny=::/0 0.0.0.0/0 allow=fd00:ec2::254/128 169.254.169.254/32

Pipelines can still reach your sources and destinations on the network and the internet; only the metadata address is blocked. Conduit's option to let built-in connectors use the instance's IAM role (connectors.ambient-credentials.enabled) is off and could not reach the role anyway, so give the S3 connector static keys of its own, scoped to its bucket.

Step 12: What changed since Conduit 0.17

Many Conduit installations still run 0.17 to 0.20.5. Three upstream security advisories were fixed in 0.21 (and 0.20.6), and they change how you use Conduit:

The API now needs a token (GHSA-v79j-j88p-m4ff). Before, anyone who could reach Conduit's API could create and run pipelines, and a web page you visited could do the same through your browser. Now every call carries the API token, the API listens on loopback by default, and requests from other sites or with a rebound host name are refused. On this image you send the token yourself or sign in at nginx, which adds it.
Pipeline settings can no longer read the engine's secrets or files (GHSA-m3mw-fpw8-w765). Before, a setting could pull any environment variable of the engine into a connector (${AWS_SECRET_ACCESS_KEY}), point the file connector at any file, or make an HTTP processor call internal addresses. Now ${NAME} works only for names you list in env.allow, file paths are confined to allowed directories, and built-in connectors and processors refuse metadata and (for processors) private addresses.
Plugins no longer inherit the engine's environment (GHSA-gwvc-9h8p-x5r7). Standalone connectors now get a minimal environment; add a variable they need under connectors.standalone.env.allow.

Two more 0.21 changes affect existing pipeline files: every pipeline needs a dead-letter queue that cannot lose a record (the default one stops the pipeline at the first failure, as in Step 7), and the log connector writes only hashes of records unless you set payload: "true". Move pipelines from an older Conduit by re-creating them from their YAML on this instance after checking those two points and any ${NAME} references.

Step 13: Your own DNS name and a CA-issued certificate

To serve Conduit under your own DNS name, point the name at the instance (an Elastic IP is recommended) and set it. This writes the name into nginx and issues a new self-signed certificate that covers it:

sudo conduit-set-hostname conduit.example.com

To install a certificate from your CA (or Let's Encrypt), copy the full chain (server certificate first) and the unencrypted private key to the instance, then:

sudo conduit-install-cert /path/to/fullchain.pem /path/to/privkey.pem conduit.example.com

The boot-time renewal never replaces a certificate installed this way. sudo conduit-set-hostname <name> goes back to a self-signed certificate. Both commands update the URL and fingerprint in /root/conduit-credentials.txt.

Step 14: Change the UI password or the API token

The password is stored as a bcrypt hash in /etc/nginx/conduit-htpasswd. To set a new one without putting it on a command line, let htpasswd prompt for it, then update the curl config and the credentials file:

sudo htpasswd -B -C 10 /etc/nginx/conduit-htpasswd admin
sudo nano /root/conduit-curl.conf /root/conduit-credentials.txt
sudo systemctl reload nginx

To rotate the API token, write a new 64-character hex token (for example from openssl rand -hex 32) into /etc/conduit/api-token, into the default line of /etc/nginx/conduit-upstream-auth.conf, into /root/conduit-api.curl and the credentials file, then sudo systemctl restart conduit and sudo systemctl reload nginx. Edit the files with an editor rather than sed, so the token never appears on a command line. Conduit reads the token once at start.

Step 15: Logs, versions and file locations

conduit --version
journalctl -u conduit -b --no-pager -o cat | grep -E 'grpc API started|API token authentication|http API started|pipeline configs provisioned' | sed -E 's/^[0-9T:+-]+ //' | awk '!seen[$0]++' | awk 'n < 4 {print; n++}'

v0.21.2 linux/amd64
INF pipeline configs provisioned component=provisioning.Service created=["demo-generator-to-log"] deleted=[] pipelines_path=/etc/conduit/pipelines
INF grpc API started address=127.0.0.1:8084
INF API token authentication is enabled for the HTTP and gRPC APIs
INF http API started address=127.0.0.1:8080

Path	Contents
`/usr/local/bin/conduit`	the Conduit 0.21.2 binary (server and CLI)
`/etc/conduit/conduit.yaml`	Conduit configuration (no secrets; `sudo conduit config --config.path /etc/conduit/conduit.yaml` prints the effective values)
`/etc/conduit/api-token`	this instance's API token (root:conduit 0640)
`/etc/conduit/pipelines/`	pipeline files Conduit loads at every start (the demo)
`/var/lib/conduit`	the pipeline database, plus `files/` (the file connector's directory) and `secrets/` (certificate files for connectors)
`/etc/nginx/sites-available/cloudimg-conduit`	the nginx front door
`/root/conduit-credentials.txt`, `/root/conduit-curl.conf`, `/root/conduit-api.curl`	login, API token, curl configs (root only)
`/usr/share/doc/conduit`	Conduit's licence and the release provenance

Conduit logs to the journal (journalctl -u conduit), including pipeline errors; nginx logs to /var/log/nginx/. A pipeline's last error is also in GET /v1/pipelines/<id> (state.error) and on its page in the UI.

Step 16: Backup and maintenance

Back up /var/lib/conduit with Conduit stopped (sudo systemctl stop conduit) to keep API-created pipelines and connector positions, plus /etc/conduit (your pipeline files and settings). Keep your pipeline YAML in version control as well.
Ubuntu security updates install automatically (unattended-upgrades). Conduit is a single binary in /usr/local/bin; to move to a newer Conduit release, launch the newer version from the listing and apply your pipeline files there.
Size the instance for your pipelines: m5.large suits development and modest streams; use larger instances for high record rates or many pipelines.

Step 17: Security recommendations

Limit TCP 443 and 22 in the security group to your admin network. The login and the API token create pipelines that reach your systems: share them only with people you would trust with those systems.
Serve Conduit under your own DNS name with a certificate from a trusted CA (Step 13). Until then, check the self signed certificate's fingerprint (Step 4) before you sign in.
Keep the API on the loopback address with the token file: Conduit refuses to start if either API is bound to another address, the token file is missing or readable by others, or a wildcard browser origin is allowed.
List in env.allow and connectors.egress.allow only what your pipelines need, give connectors credentials of their own scoped to their data, and install standalone connectors only from sources you trust.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk. Conduit is licensed under the Apache License 2.0 (the licence text is on the instance in /usr/share/doc/conduit); nginx is licensed under the 2-clause BSD licence. Conduit is a trademark of its respective owner; cloudimg is not affiliated with or endorsed by the Conduit project or its owners.