Streaming & Messaging AWS

Apache Kafka 4.4 on AWS User Guide

Last updated: 2026-10-29 | Product: Apache Kafka 4.4 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Apache Kafka 4.4 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Apache Kafka is the open source distributed event streaming platform: producers write records to topics, Kafka stores them durably in order, and any number of consumers read them at their own pace. This image runs a single-node Kafka 4.4.0 cluster in KRaft mode (broker and controller in one process, no ZooKeeper), ready for applications on other hosts to connect securely.

What the cloudimg image adds:

Encrypted and authenticated from the first boot. The only Kafka listener reachable from the network is SASL_SSL on port 9094: TLS 1.3 (or 1.2) plus SCRAM-SHA-512 passwords, the same mechanism Amazon MSK uses for SASL/SCRAM. There is no plaintext listener on the network, and the broker refuses to start if one is ever configured.
No default login, no shared secret. The Kafka admin password, the broker's own inter-broker password, AKHQ's Kafka password, the AKHQ admin password and session secret, the KRaft cluster id and the TLS key are all generated on your instance at first boot. None of them is baked into the image, and none ever appears on a command line (the boot service hands the SCRAM passwords to Kafka through a root-only file).
A certificate that is yours alone. Kafka (9094) and the AKHQ web UI (443) use one certificate generated on your instance. It is an end-entity certificate: it names only this instance and can never sign other certificates. Your clients pin it after checking its fingerprint (Step 6), or you replace it with a certificate from a certificate authority (Step 9).
The AKHQ web UI with its own login. AKHQ 0.28.0 lets you browse topics and messages, watch consumer groups and lag, and manage the cluster from a browser. It is served by nginx over HTTPS, needs a login for everything, rate limits the login form, refuses cross-origin requests and refuses state-changing links that do not come from AKHQ itself.
Fail closed. Kafka, AKHQ and nginx start only after the boot service has written your secrets, and configuration checks refuse to start them with a missing or default secret, a plaintext network listener or a public AKHQ address.
Hardened defaults. Kafka, AKHQ, nginx and the boot service are blocked from the instance metadata service, the AMI requires IMDSv2, the Kafka controller and the local SASL listener are on the loopback address only, JMX is off, topics are never created automatically, and every Java service runs under systemd sandboxing as an unprivileged user.
Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Apache Kafka 4.4.0 (the upstream release, verified against its published SHA-512 checksum) on OpenJDK 21 from Ubuntu 24.04, AKHQ 0.28.0 (the upstream release jar, verified against its sha256 digest) on Eclipse Temurin 25.0.5 JRE, nginx 1.24 from Ubuntu 24.04, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

The AKHQ Topics page listing the cloudimg topic with its messages and the quickstart consumer group

Prerequisites

An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
A security group that allows TCP 22 (SSH) from your admin network, TCP 9094 (Kafka SASL_SSL) from the networks your Kafka clients run in, and TCP 443 (AKHQ) from the networks you browse from, plus TCP 80 if you want plain HTTP requests redirected to HTTPS. The listing's default security group opens these to everyone: narrow them to your own networks. Never open 9092, 9093, 8080 or 28081: they listen on the loopback address only.
For clients on other hosts: the Kafka command line tools (or any Kafka client library) and network access to port 9094.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Apache Kafka 4.4 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=kafka}]'

First boot takes about two minutes after the instance is running: the boot service generates the passwords and the certificate and formats the KRaft storage, Kafka starts and writes its first metadata snapshot, then AKHQ and nginx start.

Step 2: Connect to your instance

Version	SSH user
Apache Kafka 4.4 on Ubuntu 24.04	`ubuntu`

ssh -i <your-key.pem> ubuntu@<public-dns>

Step 3: Check the services

systemctl is-active kafka-metadata-names kafka-firstboot kafka akhq nginx
sudo test -f /var/lib/cloudimg/kafka-firstboot.done && echo FIRSTBOOT_DONE

All five units are active and the first boot marker is present:

active
active
active
active
active
FIRSTBOOT_DONE

Only SSH, nginx and the Kafka SASL_SSL listener (9094) listen on network ports; Kafka's local SASL listener (9092), its controller (9093) and AKHQ (8080, 28081) listen on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

*:9094 users:(("java"))
0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))
[::ffff:127.0.0.1]:28081 users:(("java"))
[::ffff:127.0.0.1]:8080 users:(("java"))
[::ffff:127.0.0.1]:9092 users:(("java"))
[::ffff:127.0.0.1]:9093 users:(("java"))

Step 4: Get your bootstrap address, passwords and certificate fingerprint

Everything you need is in /root/kafka-credentials.txt, readable by root only. Show it with sudo cat /root/kafka-credentials.txt; here it is with the two passwords hidden:

sudo sed -E 's/(PASSWORD=).*/\1<hidden>/' /root/kafka-credentials.txt

# Apache Kafka 4.4 on Ubuntu 24.04 - generated on first boot by kafka-firstboot.service.
# Every password below is unique to this instance; none was baked into the image. Keep this file secret.
#
# Kafka from other hosts: SASL_SSL with SCRAM-SHA-512 on KAFKA_BOOTSTRAP_TLS (open port 9094 in your security group
# to your client networks only). Client config template: /root/kafka-client-tls.properties. The broker certificate is
# self-signed (minted on this instance, never a CA): pin it in a truststore file used only by your Kafka client and
# check it is this one -
#   openssl x509 -in kafka-broker.pem -noout -fingerprint -sha256   must print KAFKA_TLS_CERT_SHA256
# or install a CA-issued certificate: sudo kafka-install-cert <chain.pem> <key.pem>
# Kafka on this instance: 127.0.0.1:9092 (sudo kafka-local topics --list). More users: sudo kafka-scram-user add <name>
KAFKA_BOOTSTRAP_TLS=<public-dns>:9094
KAFKA_BOOTSTRAP_LOCAL=127.0.0.1:9092
KAFKA_SASL_MECHANISM=SCRAM-SHA-512
KAFKA_ADMIN_USER=admin
KAFKA_ADMIN_PASSWORD=<hidden>
KAFKA_TLS_CERT_SHA256=<sha256-fingerprint>
#
# AKHQ web UI: open AKHQ_URL and log in as AKHQ_ADMIN_USER (same certificate; compare the fingerprint your browser shows).
AKHQ_URL=https://<public-dns>/
AKHQ_ADMIN_USER=admin
AKHQ_ADMIN_PASSWORD=<hidden>
# Change the advertised/served name: sudo kafka-set-hostname <dns-name>    Re-verify the appliance: sudo kafka-selfcheck
# User guide: https://www.cloudimg.co.uk/guides/apache-kafka-4-4-on-ubuntu-24-04-aws/

The bootstrap address is the instance's public DNS name from the EC2 instance metadata (inside your VPC it resolves to the private address, outside to the public one; without a public name the public IP or private address is used). The certificate names the public and private DNS names and IP addresses too. If you stop and start the instance without an Elastic IP, the boot service advertises the new name and issues a new certificate for it; to keep one name, use your own DNS name (Step 9).

The broker certificate's SHA-256 fingerprint, printed from the public copy of the certificate:

openssl x509 -in /etc/cloudimg/kafka-broker.pem -noout -fingerprint -sha256

sha256 Fingerprint=<sha256-fingerprint>

Step 5: Use Kafka on the instance

kafka-local runs any Kafka command line tool against the broker's loopback SASL listener with your admin client configuration (/root/kafka-client.properties), so no password ever goes on a command line. The image creates one topic, cloudimg, to get you started (topics are never created automatically):

sudo kafka-local topics --list

cloudimg

Produce three keyed messages and read them back as a consumer group:

printf '%s\n' 'order-1001:{"item":"SKU-0042","qty":2}' 'order-1002:{"item":"SKU-0007","qty":1}' 'order-1003:{"item":"SKU-0042","qty":5}' \
  | sudo kafka-local console-producer --topic cloudimg --reader-property parse.key=true --reader-property key.separator=:
sudo kafka-local console-consumer --topic cloudimg --from-beginning --max-messages 3 --formatter-property print.key=true --group quickstart

order-1001  {"item":"SKU-0042","qty":2}
order-1002  {"item":"SKU-0007","qty":1}
order-1003  {"item":"SKU-0042","qty":5}
Processed a total of 3 messages

Create your own topics the same way, for example sudo kafka-local topics --create --topic orders --partitions 3 --replication-factor 1. On a single node the replication factor is always 1. The tools themselves are in /opt/kafka/bin; run them with --bootstrap-server 127.0.0.1:9092 --command-config /root/kafka-client.properties.

Step 6: Connect a Kafka client from another host

Clients on other hosts connect to KAFKA_BOOTSTRAP_TLS (port 9094) with SASL_SSL and SCRAM-SHA-512. Two things are needed: a password, and a way for the client to trust the broker's certificate.

Trust the broker by pinning its certificate. The certificate is self-signed and unique to this instance. Copy its public half to the client host, check its SHA-256 fingerprint against KAFKA_TLS_CERT_SHA256 in /root/kafka-credentials.txt (Step 4), and then pin it in a truststore file that only this Kafka client uses. Never add it to the operating system's, a browser's or the Java runtime's own trust store. If you prefer certificates your clients already trust, install a certificate from a certificate authority with kafka-install-cert (Step 9) and leave the truststore lines out.

On the client host, copy the certificate and check the fingerprint (it must print the same value as KAFKA_TLS_CERT_SHA256):

scp -i <your-key.pem> ubuntu@<public-dns>:/etc/cloudimg/kafka-broker.pem ./kafka-broker.pem
openssl x509 -in kafka-broker.pem -noout -fingerprint -sha256

Then write a client configuration file on the client host, readable by you only. Put the password in the file, never on a command line (use KAFKA_ADMIN_PASSWORD, or better a user of its own from Step 7):

# on <client-host>
umask 077
cat > client.properties <<'EOF'
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="<password>";
ssl.truststore.type=PEM
ssl.truststore.location=/path/to/kafka-broker.pem
EOF

Produce and consume over the network with the Kafka 4.4 command line tools on the client host:

# on <client-host>
echo 'hello from another host' | kafka-console-producer.sh --bootstrap-server <public-dns>:9094 --command-config client.properties --topic cloudimg
kafka-console-consumer.sh --bootstrap-server <public-dns>:9094 --command-config client.properties --topic cloudimg --from-beginning --max-messages 1

Client libraries use the same settings: in librdkafka-based clients (Python confluent-kafka, Go, .NET, Node.js) set security.protocol=SASL_SSL, sasl.mechanism=SCRAM-SHA-512, sasl.username, sasl.password and ssl.ca.location=/path/to/kafka-broker.pem. The broker checks the client's host name against the certificate, so connect with one of the names it carries (openssl x509 -in kafka-broker.pem -noout -ext subjectAltName). A ready-made template, with the admin password and the on-instance path of the certificate, is at /root/kafka-client-tls.properties.

Amazon MSK's IAM authentication does not apply here: the broker has no route to the instance metadata service and clients authenticate with SCRAM over TLS.

Step 7: Manage Kafka users

kafka-scram-user adds, rotates and deletes SCRAM-SHA-512 users. It generates each password on the instance and writes it into a ready-to-use client configuration, /root/kafka-users/<name>-client-tls.properties (readable by root only): it never prints the password and never puts it on a command line (it reaches Kafka through a root-only file). The users the image itself manages, broker and akhq, cannot be changed with it.

sudo kafka-scram-user list

admin  SCRAM-SHA-512=iterations=4096
broker  SCRAM-SHA-512=iterations=4096
akhq  SCRAM-SHA-512=iterations=4096

Give each application its own user, then copy its client file and the broker certificate to the application host (Step 6):

sudo kafka-scram-user add orders-service
sudo kafka-scram-user rotate orders-service
sudo kafka-scram-user delete orders-service

rotate admin also updates /root/kafka-credentials.txt and /root/kafka-client*.properties. Do not create or delete SCRAM users with kafka-configs.sh --add-config-file: it mangles the SCRAM mechanism name and prints the password in its error message.

Authorization: like Amazon MSK's default, this broker has no authorizer, so every authenticated user can create and delete topics and read every topic. Before you give a login to a less trusted application, enable Kafka's StandardAuthorizer with ACLs (authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer and super.users=User:admin;User:broker;User:akhq;User:ANONYMOUS in /etc/kafka/server.properties, where ANONYMOUS is the loopback-only controller listener), then grant each user its own topics with sudo kafka-local acls --add .... See the Kafka authorization documentation.

Step 8: The AKHQ web UI

Open AKHQ_URL (https://<public-dns>/) in your browser. It warns that the certificate is self signed: open the certificate details, check that its SHA-256 fingerprint matches the one from Step 4, and only then continue to the site. Log in as admin with AKHQ_ADMIN_PASSWORD.

The AKHQ login page

The Topics page lists your topics, then a topic's Data tab shows its messages with keys, partitions and offsets, live tail included:

The cloudimg topic in AKHQ with the three order messages, their keys, partitions and offsets

Consumer Groups shows every group and its lag:

The Consumer Groups page showing the quickstart group with a lag of 0 on the cloudimg topic

Nodes shows the broker as AKHQ sees it over the loopback listener:

The Nodes page showing broker 1 at 127.0.0.1:9092 as the controller

AKHQ logs in to Kafka as its own SCRAM user, akhq, whose password is in /etc/akhq/secrets.yml (root and the akhq group only). To add AKHQ users or manage other clusters, edit /etc/akhq/secrets.yml (logins) or /etc/akhq/application.yml (connections) and run sudo systemctl restart akhq; AKHQ refuses to start with HTTP Basic or anonymous access switched on, on a public address, or without its own session secret.

Step 9: Your own DNS name and certificate

To advertise Kafka and serve AKHQ under your own DNS name (point it at the instance's Elastic IP first), run:

sudo kafka-set-hostname kafka.example.com

This sets Kafka's advertised listener to kafka.example.com:9094, issues a new self-signed certificate for the name, restarts Kafka and reloads nginx. Clients pin the new certificate (Step 6).

To use a certificate from a certificate authority instead (for example from your company CA or Let's Encrypt), put the certificate chain (server certificate first) and its private key on the instance and run:

sudo kafka-install-cert /path/to/fullchain.pem /path/to/privkey.pem kafka.example.com

Kafka (9094) and AKHQ (443) then use your certificate, clients that already trust your CA connect without a pinned truststore, and the boot service never replaces it. sudo kafka-set-hostname <name> returns to a self-signed certificate.

Step 10: Security checks

kafka-selfcheck re-verifies the whole appliance with your instance's own credentials. It runs a produce and consume round trip over the loopback listener and over SASL_SSL with the pinned certificate, and proves wrong passwords are refused. It also checks that port 9094 speaks only TLS, and that the AKHQ login, session secret, origin guard and cross-site guard work. It confirms the internals refuse network connections and the metadata service is blocked. Finally it creates, uses and deletes a temporary Kafka user while checking that no password reaches a process command line, the journal or a log file:

sudo kafka-selfcheck

Apache Kafka appliance self-check (SASL_SSL <public-dns>:9094, AKHQ https://<public-dns>/)
  [ok] loopback SASL (SCRAM-SHA-512, admin) on 127.0.0.1:9092: topics listed (cloudimg present), produce + consume round trip
  [ok] wrong SCRAM password on 127.0.0.1:9092 -> refused by SCRAM (invalid credentials)
  [ok] SASL_SSL (TLS + SCRAM-SHA-512, admin) on <public-dns>:9094: produce + consume round trip with the PINNED certificate and hostname verification
  [ok] wrong SCRAM password on SASL_SSL -> refused by SCRAM (invalid credentials)
  [ok] KRaft metadata snapshot present (<snapshot>); snapshots every 60 s keep SCRAM users safe across restarts (Kafka 4.4.0 ScramDelta bug)
  [ok] wire: 9092 answers ApiVersions but drops Metadata before SASL; 9094 speaks TLS only (TLS 1.3 default, TLS 1.2 accepted, nothing older enabled; plaintext Kafka request: no response), drops Metadata before SASL; a system-CA-only client refuses the self-signed certificate
  [ok] HTTPS AKHQ login page -> 200
  [ok] topic API without a session -> 401; HTTP Basic (even with the right password) -> 401
  [ok] wrong password -> 303 back to the login page, no session cookie
  [ok] admin login with the per-instance password -> session; AKHQ (its own SCRAM login) lists the topics and the self-check consumer group
  [ok] session token is HS256-signed with this instance's own secret (re-signed copy -> 200)
  [ok] tokens signed with AKHQ's built-in default secret or a random secret -> 401
  [ok] port 80 -> 301 https://<public-dns>/; unknown Host -> 444; CONNECT refused; cross-origin POST -> 403; WebSocket upgrade -> 403
  [ok] AKHQ (8080/28081) and Kafka LOCAL/CONTROLLER (9092/9093) refuse the private address; SASL_SSL 9094 answers on it; AKHQ speaks no TLS itself
  [ok] instance metadata refused inside the kafka, akhq and nginx cgroups (loopback control 200)
  [ok] TLS certificate: end-entity (CA:FALSE, serverAuth, no certificate signing), names <public-dns>; the same on 443 and 9094; SHA-256 <sha256-fingerprint>
  [ok] cross-site GET guard: 13/13 (/logout); Kafka Connect restart/pause/resume/task-restart: 24/24 foreign, none or missing signals -> nginx 403
  [ok] SCRAM user cloudimg-selfcheck-<random>: created by kafka-scram-user (ConfigCommand via a 0600 java @argfile), logged in over SASL_SSL, deleted -> refused; argv sampler snapshots=662 distinct=48 values=3 hits=0; journal + /var/log: 0 hits for the new user's, admin and AKHQ passwords
SELFCHECK OK - Kafka SASL_SSL (pinned end-entity certificate, SCRAM-SHA-512) and loopback SASL round trips, wrong passwords refused, TLS-only off-host listener, AKHQ login and session secret, origin and cross-site guards, loopback-only internals, metadata block verified.

The metadata block is a systemd IPAddressDeny on every service. The one exception is kafka-metadata-names, which runs at boot, only reads the instance's DNS names and addresses for the advertised listener, and may reach nothing but the metadata service:

for u in kafka akhq nginx kafka-firstboot kafka-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u)" "$(systemctl show -p IPAddressAllow --value $u)"; done

kafka: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow=
akhq: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow=
nginx: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow=
kafka-firstboot: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 allow=
kafka-metadata-names: deny=::/0 0.0.0.0/0 allow=fd00:ec2::254/128 169.254.169.254/32

Step 11: Versions, provenance and file locations

/opt/kafka/bin/kafka-topics.sh --version 2>/dev/null | tail -1
java -version 2>&1 | head -1
sudo kafka-local metadata-quorum describe --status 2>/dev/null | grep -E 'LeaderId|CurrentVoters'

4.4.0
openjdk version "21.0.12.1" 2026-08-18
LeaderId:               1
CurrentVoters:          [{"id": 1, "endpoints": ["CONTROLLER://127.0.0.1:9093"]}]

What	Where
Kafka configuration	`/etc/kafka/server.properties` (listeners, advertised listener, retention)
Kafka data (topics) and KRaft metadata	`/var/lib/kafka/data/logs`, `/var/lib/kafka/data/metadata`
Kafka logs	`/var/log/kafka` and `journalctl -u kafka`
Broker TLS keystore (key + certificate, Kafka group only)	`/etc/kafka/tls/kafka-keystore.pem`
Public broker certificate (to pin)	`/etc/cloudimg/kafka-broker.pem`
Your passwords and addresses	`/root/kafka-credentials.txt`, `/root/kafka-client.properties`, `/root/kafka-client-tls.properties`, `/root/kafka-users/`
AKHQ configuration and secrets	`/etc/akhq/application.yml`, `/etc/akhq/secrets.yml`
Build provenance and licences	`/usr/share/doc/apache-kafka/`, `/usr/share/doc/akhq/`

Two settings in server.properties work around a bug in Kafka 4.4.0: metadata.log.max.snapshot.interval.ms=60000 and metadata.max.retention.ms=3600000. Without frequent metadata snapshots, deleting a SCRAM user could lock every user out at the next restart. Keep both: Kafka refuses to start without the first.

Step 12: Backup and maintenance

Back up /var/lib/kafka/data (with Kafka stopped, or with EBS snapshots of the volume) together with /etc/kafka, /etc/akhq and /root/kafka-*: the SCRAM users and the cluster id live in the KRaft metadata, so a data copy without them cannot be opened with new passwords.
Storage: topics live on the root volume. Grow the EBS volume (and the file system with sudo growpart and sudo resize2fs) as your retention needs, or change log.retention.hours (7 days by default).
Updates: Ubuntu security updates install automatically. Kafka and AKHQ are pinned releases; cloudimg publishes new AMI versions with newer releases.
Logs: journalctl -u kafka -u akhq -u nginx and /var/log/kafka/server.log.

Step 13: Security recommendations

Restrict ports 9094, 443 and 22 in your security group to the networks that need them.
Give each application its own Kafka user (Step 7) and keep the admin password for administration; rotate passwords with kafka-scram-user rotate.
Enable an authorizer and ACLs before sharing the cluster with less trusted clients (Step 7).
Pin the broker certificate in each client's own truststore after checking the fingerprint, or install a CA-issued certificate (Steps 6 and 9).
Keep /root/kafka-credentials.txt and the client files private; they are root-only on the instance.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat at support@cloudimg.co.uk. Apache Kafka and AKHQ are licensed under the Apache License 2.0; the licence texts ship in /usr/share/doc/apache-kafka/LICENSE and /usr/share/doc/akhq/LICENSE. Apache Kafka, Kafka and the Apache feather logo are trademarks of the Apache Software Foundation. cloudimg is not affiliated with or endorsed by the Apache Software Foundation or the AKHQ project.