Streaming & Messaging AWS

Debezium 3.7.1 on AWS User Guide

Last updated: 2026-11-07 | Product: Debezium 3.7.1 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Debezium 3.7.1 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Debezium is the open source change data capture (CDC) platform. Its connectors read a database's own change log (PostgreSQL logical replication, the MongoDB change stream) and turn every inserted, updated and deleted row or document into an event on an Apache Kafka topic, so other systems can react to changes without polling the database. This image runs Debezium 3.7.1.Final inside Kafka Connect on a single-node Apache Kafka 4.3.2 cluster in KRaft mode, ready to point at your own databases.

What the cloudimg image adds:

The Kafka Connect REST API is never exposed. Upstream Kafka Connect answers its REST API without a login. Here it listens on the loopback address only and is reached through nginx over HTTPS on port 443 with a per-instance login (failed logins are rate limited). The REST API itself also requires a login that only nginx holds, so even a local user on the instance cannot create connectors without it.
Change events over TLS with a password. Kafka clients connect on TCP 9094 over SASL_SSL (SCRAM-SHA-512, TLS 1.3 or 1.2), the only Kafka listener reachable from the network. Kafka's local listener (9092), its KRaft controller (9093) and the Kafka Connect REST API (8083) listen on the loopback address only and require a login as well. Kafka's authorizer is on: a least-privilege reader user may read the change event topics and nothing else.
No default login, no shared secret. Every password (the Connect REST login, the Kafka admin, reader, broker and Connect users, the controller login), the KRaft cluster id and the TLS key are generated on your instance at first boot. None is baked into the image and none ever appears on a command line.
Database passwords stay out of connector configs. Connectors reference a root-owned secrets file (${file:...}), so the REST API and Kafka's config topic show the reference, never the password.
A hardened Kafka Connect. Only the shipped Debezium connectors can be loaded (FileStream and MirrorMaker 2 are removed), connectors cannot override the worker's Kafka logins, the JNDI and LDAP login modules are refused (the CVE-2023-25194 class), and Kafka Connect runs under systemd sandboxing as an unprivileged user with no access to the instance metadata service.
A certificate that is yours alone. nginx (443) and Kafka (9094) use one end-entity certificate generated on your instance. Your clients pin it after checking its fingerprint (Step 7), or you replace it with a certificate from a certificate authority (Step 10).
Fail closed. Kafka, Kafka Connect and nginx start only after the boot service has written your secrets, and configuration checks refuse to start them with a missing secret, a plaintext network listener, an unauthenticated REST API or a connector plugin outside the shipped set.
Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Debezium 3.7.1.Final, the PostgreSQL connector and the MongoDB source and sink connectors (from Maven Central, each archive checked against its published SHA-512 digest and the Debezium release signing key), Apache Kafka 4.3.2 with Kafka Connect (the upstream release, checked against its published SHA-512 digest), the Eclipse Temurin 21.0.13+9 Java runtime, nginx 1.24 from Ubuntu 24.04, cloud-init and the AWS Systems Manager agent. The MySQL connector is not included.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
A security group that allows TCP 22 (SSH) from your admin network, TCP 443 (the Kafka Connect REST API) from the networks you manage connectors from, and TCP 9094 (Kafka SASL_SSL) from the networks your consumers run in, plus TCP 80 if you want plain HTTP requests redirected to HTTPS. The listing's default security group opens these to everyone: narrow them to your own networks. Never open 8083, 9092 or 9093: they listen on the loopback address only.
A database for Debezium to capture from, reachable from the instance: PostgreSQL 10 or later with logical replication (wal_level=logical, a user with the REPLICATION attribute), including Amazon RDS and Aurora PostgreSQL with rds.logical_replication=1, or a MongoDB replica set or sharded cluster.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Debezium 3.7.1 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=debezium}]'

First boot takes about three minutes after the instance is running: the boot service generates the passwords and the certificate and formats the KRaft storage, then Kafka starts, then Kafka Connect creates its internal topics and opens its REST API.

Step 2: Connect to your instance

Version	SSH user
Debezium 3.7.1 on Ubuntu 24.04	`ubuntu`

ssh -i <your-key.pem> ubuntu@<public-dns>

Step 3: Check the services

systemctl is-active debezium-metadata-names debezium-firstboot kafka kafka-connect nginx
sudo test -f /var/lib/cloudimg/debezium-firstboot.done && echo FIRSTBOOT_DONE

All five units are active and the first boot marker is present:

active
active
active
active
active
FIRSTBOOT_DONE

Only SSH, nginx and the Kafka SASL_SSL listener (9094) listen on network ports; Kafka's local listener (9092), its controller (9093) and the Kafka Connect REST API (8083) listen on the loopback address:

sudo ss -Hlnt | awk '{print $4}' | grep -v '127.0.0.5' | sort -u

*:9094
0.0.0.0:22
0.0.0.0:443
0.0.0.0:80
[::]:22
[::]:443
[::]:80
[::ffff:127.0.0.1]:8083
[::ffff:127.0.0.1]:9092
[::ffff:127.0.0.1]:9093

sudo debezium-selfcheck re-runs the appliance's own end to end check at any time (about four minutes): logins and refusals on every Kafka listener and on the REST API, a produce and consume round trip over SASL_SSL with the pinned certificate, the connector plugin set, the Kafka Connect hardening and the metadata block. It ends with SELFCHECK OK.

Step 4: Get your logins, bootstrap address and certificate fingerprint

Everything you need is in /root/debezium-credentials.txt, readable by root only. Show it with sudo cat /root/debezium-credentials.txt; here it is with the passwords hidden:

sudo sed -E 's/(PASSWORD=).*/\1<hidden>/' /root/debezium-credentials.txt

The file lists:

Line	What it is
`DEBEZIUM_CONNECT_URL`	The Kafka Connect REST API, `https://<public-dns>/`
`DEBEZIUM_CONNECT_USER`, `DEBEZIUM_CONNECT_PASSWORD`	The REST API login (`admin`)
`KAFKA_BOOTSTRAP_TLS`	The bootstrap address for clients on other hosts, `<public-dns>:9094`
`KAFKA_ADMIN_USER`, `KAFKA_ADMIN_PASSWORD`	The Kafka admin user (`admin`)
`KAFKA_READER_USER`, `KAFKA_READER_PASSWORD`	The least-privilege `reader`: reads topics whose names start with `cdc.` in consumer groups whose names start with `cdc-`, nothing else
`KAFKA_TLS_CERT_SHA256`	The SHA-256 fingerprint of this instance's certificate

Ready-made files sit next to it, all root 0600:

/root/debezium-connect.curl: the REST API login as a curl config file, for curl -K.
/root/kafka-client.properties: the admin login for Kafka's loopback listener (used by kafka-local).
/root/kafka-client-tls.properties and /root/kafka-reader-tls.properties: SASL_SSL client configurations for the admin and reader users.

Keep the logins out of command lines and shell history: curl reads the REST login from a file with -K, and Kafka tools read theirs from a properties file with --command-config.

Step 5: Use the Kafka Connect REST API

connect-api calls this instance's REST API through nginx with the login from /root/debezium-connect.curl, so no password goes on a command line. List the connector plugins:

sudo connect-api GET /connector-plugins | python3 -c 'import json,sys; [print(p["class"], p["type"], p["version"]) for p in json.load(sys.stdin)]'

The answer lists exactly the shipped connectors:

io.debezium.connector.mongodb.MongoDbSinkConnector sink 3.7.1.Final
io.debezium.connector.mongodb.MongoDbConnector source 3.7.1.Final
io.debezium.connector.postgresql.PostgresConnector source 3.7.1.Final

From your own workstation, copy the curl login file and the certificate (check its fingerprint as in Step 7), then call the REST API with -K:

scp -i <your-key.pem> ubuntu@<public-dns>:/etc/cloudimg/kafka-broker.pem ./kafka-broker.pem
ssh -i <your-key.pem> ubuntu@<public-dns> 'sudo cat /root/debezium-connect.curl' > debezium-connect.curl
chmod 600 debezium-connect.curl
curl -K debezium-connect.curl --cacert kafka-broker.pem https://<public-dns>/connectors

--cacert trusts this instance's certificate for that one curl call only. Without the login the REST API answers 401, and after too many failed logins 429. https://<public-dns>/healthz answers without a login, for load balancer health checks.

Step 6: Capture changes from PostgreSQL

Prepare the database. On the PostgreSQL server set wal_level = logical (on Amazon RDS: the rds.logical_replication parameter), then create a user for Debezium and a publication:

CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD '<a-strong-password>';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium;
CREATE PUBLICATION dbz_publication FOR TABLE public.customers;

Put the database password in a secrets file, not in the connector. Kafka Connect resolves ${file:...} references to files in /etc/kafka-connect/secrets only. On the instance:

sudo install -o root -g kafka-connect -m 0640 /dev/null /etc/kafka-connect/secrets/inventory-db.properties
sudoedit /etc/kafka-connect/secrets/inventory-db.properties

Write one line, password=<the debezium user's password>, and save.

Create the connector. Write its configuration to a file. The topic prefix starts with cdc. so that the reader user may consume the change events, and the two topic.creation lines let Kafka Connect create the change event topics (the broker never creates topics automatically):

cat > inventory-postgres.json <<'EOF'
{
  "name": "inventory-postgres",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "<your-postgres-host>",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${file:/etc/kafka-connect/secrets/inventory-db.properties:password}",
    "database.dbname": "<your-database>",
    "database.sslmode": "require",
    "plugin.name": "pgoutput",
    "publication.name": "dbz_publication",
    "slot.name": "debezium_inventory",
    "topic.prefix": "cdc.inventory",
    "table.include.list": "public.customers",
    "topic.creation.default.replication.factor": "1",
    "topic.creation.default.partitions": "1"
  }
}
EOF
sudo connect-api POST /connectors inventory-postgres.json
sudo connect-api GET /connectors/inventory-postgres/status

The connector and its task report RUNNING. Use "database.sslmode": "verify-full" with "database.sslrootcert" pointing at your database's CA certificate (stored under /etc/kafka-connect/secrets) to check the server's identity; Amazon RDS publishes its CA bundle.

GET /connectors/inventory-postgres/config shows the ${file:...} reference, never the password. Changes to public.customers now arrive on the topic cdc.inventory.public.customers:

sudo kafka-local topics --list
sudo kafka-local console-consumer --topic cdc.inventory.public.customers --from-beginning --max-messages 1 --timeout-ms 60000

Each event is a JSON envelope with before, after, source and op (c create, u update, d delete, r initial snapshot read).

Pause, resume or delete a connector with sudo connect-api PUT /connectors/inventory-postgres/pause, .../resume or sudo connect-api DELETE /connectors/inventory-postgres. Deleting a connector does not drop its replication slot: drop it on the database (SELECT pg_drop_replication_slot('debezium_inventory');) so the server does not keep write-ahead log for it.

MongoDB works the same way with "connector.class": "io.debezium.connector.mongodb.MongoDbConnector" and "mongodb.connection.string": "${file:/etc/kafka-connect/secrets/mongo.properties:uri}" (a replica set or sharded cluster; put tls=true in the connection string).

Connectors cannot override the worker's Kafka client settings (the producer.override.* and consumer.override.* properties are refused), and only the plugins listed in Step 5 can be loaded.

Step 7: Read change events from another host

Consumers on other hosts connect to KAFKA_BOOTSTRAP_TLS (port 9094) with SASL_SSL and SCRAM-SHA-512. Two things are needed: a password, and a way for the client to trust the broker's certificate.

Trust the broker by pinning its certificate. The certificate is self-signed and unique to this instance. Copy its public half to the client host, check its SHA-256 fingerprint against KAFKA_TLS_CERT_SHA256 in /root/debezium-credentials.txt (Step 4), and then pin it in a truststore file that only this Kafka client uses. Never add it to the operating system's, a browser's or the Java runtime's own trust store. If you prefer certificates your clients already trust, install a certificate from a certificate authority with debezium-install-cert (Step 10) and leave the truststore lines out.

On the client host, copy the certificate and check the fingerprint (it must print the same value as KAFKA_TLS_CERT_SHA256):

scp -i <your-key.pem> ubuntu@<public-dns>:/etc/cloudimg/kafka-broker.pem ./kafka-broker.pem
openssl x509 -in kafka-broker.pem -noout -fingerprint -sha256

Then write a client configuration file on the client host, readable by you only, with the reader's password from Step 4 (in the file, never on a command line):

# on <client-host>
umask 077
cat > reader.properties <<'EOF'
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="reader" password="<KAFKA_READER_PASSWORD>";
ssl.truststore.type=PEM
ssl.truststore.location=/path/to/kafka-broker.pem
EOF

Read the change events with the Kafka command line tools on the client host (the consumer group name must start with cdc-):

# on <client-host>
kafka-console-consumer.sh --bootstrap-server <public-dns>:9094 --command-config reader.properties \
  --topic cdc.inventory.public.customers --group cdc-quickstart --from-beginning

Client libraries use the same settings: in librdkafka-based clients (Python confluent-kafka, Go, .NET, Node.js) set security.protocol=SASL_SSL, sasl.mechanism=SCRAM-SHA-512, sasl.username, sasl.password and ssl.ca.location=/path/to/kafka-broker.pem. The broker checks the client's host name against the certificate, so connect with one of the names it carries (openssl x509 -in kafka-broker.pem -noout -ext subjectAltName). The template /root/kafka-reader-tls.properties holds the same settings.

Step 8: Manage Kafka users and permissions

kafka-scram-user adds, rotates and deletes SCRAM-SHA-512 users. It generates each password on the instance and writes a ready-to-use client configuration, /root/kafka-users/<name>-client-tls.properties (root only), never printing the password or putting it on a command line. The users the image manages itself, broker and connect, cannot be changed with it.

sudo kafka-scram-user list
sudo kafka-scram-user add orders-service
sudo kafka-scram-user rotate reader
sudo kafka-scram-user delete orders-service

Kafka's StandardAuthorizer is on and nothing is allowed without an ACL, so a new user can do nothing until you grant it, for example read access to one connector's topics:

sudo kafka-local acls --add --allow-principal User:orders-service --operation Read --operation Describe \
  --topic cdc.inventory. --resource-pattern-type prefixed
sudo kafka-local acls --add --allow-principal User:orders-service --operation Read --group orders- --resource-pattern-type prefixed

rotate admin and rotate reader also update /root/debezium-credentials.txt and the matching /root/kafka-*.properties.

Step 9: Security checks

These checks show the appliance's defences in place. Each refusal is the expected answer.

The REST API needs its login, even on the instance. Port 8083 answers 401 without the login that only nginx holds:

sudo -u nobody curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8083/connectors

Kafka's loopback listeners need a login too. A client without SASL gets no answer from the controller (9093) or the local listener (9092):

printf 'security.protocol=PLAINTEXT\n' > /tmp/noauth.properties
timeout 30 /opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/noauth.properties --list >/dev/null 2>&1; echo "rc=$?"
rm -f /tmp/noauth.properties

The broker drops the request before a login, so the tool never lists a topic and is stopped by the timeout:

rc=124

The services cannot reach the instance metadata service. Kafka, Kafka Connect, nginx and the boot service are blocked from 169.254.169.254; Kafka Connect, which reaches your databases, is blocked from the whole link-local range. The only unit that reads instance metadata is debezium-metadata-names, which writes the instance's four EC2 names and nothing else. The AMI requires IMDSv2.

Step 10: Your own DNS name and certificate

If you give the instance a DNS name of your own (an Elastic IP plus a Route 53 record, for example), make the appliance advertise and serve it:

sudo debezium-set-hostname debezium.example.com

This re-issues the self-signed certificate for the new name (check the new fingerprint in /root/debezium-credentials.txt) and updates Kafka's advertised listener and the URLs in the credentials file.

To use a certificate from a certificate authority instead, copy the full chain and its private key to the instance and run:

sudo debezium-install-cert /path/to/fullchain.pem /path/to/privkey.pem debezium.example.com

nginx and Kafka then serve that certificate, later boots keep it, and your clients no longer need the truststore lines.

Step 11: Versions, provenance and file locations

Item	Where
Debezium connectors	`/opt/debezium/connectors` (read-only; sha256 manifest in `/usr/local/lib/cloudimg/debezium-plugin-jars.sha256`)
Kafka and Kafka Connect	`/opt/kafka` (tools in `/opt/kafka/bin`)
Kafka configuration	`/etc/kafka/server.properties`
Kafka Connect worker configuration	`/etc/kafka-connect/connect-distributed.properties`
Connector secrets	`/etc/kafka-connect/secrets` (root:kafka-connect, files `0640`)
Kafka and Kafka Connect logs	`journalctl -u kafka -u kafka-connect`, `/var/log/kafka-connect`
Provenance of every download	`/usr/share/doc/debezium/PROVENANCE.txt`

Kafka data is on the root volume. Size the root volume for your change event retention when you launch (Kafka's default retention is 7 days).

Step 12: Backup and maintenance

Ubuntu security updates install automatically. Kafka, Kafka Connect and Debezium are updated by cloudimg in new versions of this listing.
Connector offsets, configurations and status live in Kafka topics (connect-offsets, connect-configs, connect-status) on the root volume: snapshot the volume (or use AWS Backup) to keep them together with the change event topics.
Kafka Connect resumes each connector from its stored offset after a restart: sudo systemctl restart kafka-connect.

Step 13: Security recommendations

Narrow the security group to your own networks: 443 from your admin networks, 9094 from your consumers, 22 from your admin network.
Give each consuming application its own Kafka user with ACLs for its topics (Step 8) instead of sharing admin.
Use TLS to your databases and keep database passwords in /etc/kafka-connect/secrets, never in a connector configuration.
Rotate passwords with kafka-scram-user rotate, and replace the self-signed certificate with a CA-issued one for production (Step 10).

Support and licensing

Debezium and Apache Kafka are licensed under the Apache License 2.0; the Eclipse Temurin runtime is licensed under the GNU General Public License version 2 with the Classpath Exception. cloudimg provides the image, its hardening and support; the software is open source and unmodified. For help with this image, contact cloudimg support through the listing's support page.