Streaming & Messaging AWS

Apache Kafka + Karapace 4.4 on AWS User Guide

| Product: Apache Kafka + Karapace 4.4 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Apache Kafka + Karapace 4.4 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Apache Kafka is the distributed event streaming platform. Karapace (from Aiven, Apache License 2.0) adds two Confluent-compatible HTTP services on top of it: a Schema Registry (Avro, JSON Schema and Protobuf schemas with compatibility checks) and a Kafka REST proxy (produce and consume over HTTP without a Kafka client library). The image runs all three on one instance, ready for development, testing, single-tenant streaming and proof-of-concept work.

What the cloudimg image adds:

  • No default credentials, no shared secret. Two passwords are generated on your instance at first boot: the Kafka SASL/SCRAM-SHA-256 password of the cloudimg user, and a separate HTTP password for the Schema Registry and REST proxy. The image ships neither, and Kafka's storage (with a fresh cluster id) is created on your instance at first boot.
  • HTTPS from the first boot. nginx serves the Schema Registry on port 8081 and the REST proxy on port 8082 over HTTPS with a certificate generated on your instance. It is an end-entity certificate: it can only vouch for this instance's own names and can never sign other certificates. A plain HTTP request on those ports is redirected to HTTPS, and requests for host names the instance does not serve are dropped.
  • Authentication on every request. HTTP Basic authentication (user cloudimg) is checked by nginx on both ports and only ever over TLS. Failed logins are rate limited per client address; correct logins are never slowed down.
  • Browser attacks refused. A request carrying another site's Origin is refused (including CORS preflights), Karapace's own permissive CORS headers are removed, and WebSocket upgrades are refused. The REST proxy's consumer API (/consumers/...) is additionally guarded against cross-site requests: reading records also commits the consumer's offsets, so a link or hidden image on another site must never be able to trigger it with a password your browser remembers. Clients send one header, Sec-Fetch-Site: same-origin, on consumer calls (Step 6).
  • Kafka and Karapace on localhost only. Kafka's only client listener is SASL/SCRAM on 127.0.0.1:9092 (the controller is on 127.0.0.1:9093), and Karapace listens on 127.0.0.1:18081 and 127.0.0.1:18082. Each service refuses to stay up if it finds itself on another address.
  • Fail closed. Kafka, Karapace and nginx start only after the boot service has written your secrets, and configuration checks refuse to start Kafka or Karapace with a missing or placeholder password or a public address.
  • Hardened defaults. Every service is blocked from the instance metadata service, the AMI requires IMDSv2, nginx hands Karapace the real client address and the configured host name (never values a client sends) and never forwards your password to it, and all services run under systemd sandboxing as unprivileged users.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Apache Kafka 4.4.0 (Scala 2.13, verified against its published SHA-512) on OpenJDK 21 (from Ubuntu 24.04), Karapace 6.2.4 (built from the release tarball of the 6.2.4 tag, verified against its SHA-256, with upstream's pinned Python dependencies) on Python 3.12, protoc 3.21 (Karapace uses it for Protobuf schemas), nginx 1.24, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows TCP 22 (SSH) from your admin network and TCP 8081 (Schema Registry) and TCP 8082 (REST proxy) from the networks your applications run in. Never open 9092, 9093, 18081 or 18082: they listen on the loopback address only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Apache Kafka + Karapace 4.4 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=kafka-karapace}]'

First boot takes about a minute after the instance is running: the boot service generates the passwords and the certificate and formats Kafka's storage, then Kafka, the Schema Registry, the REST proxy and nginx start.

Step 2: Connect to your instance

Version SSH user
Apache Kafka + Karapace 4.4 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

systemctl is-active kafka-karapace-firstboot kafka karapace-registry karapace-rest nginx
sudo test -f /var/lib/cloudimg/kafka-karapace-firstboot.done && echo FIRSTBOOT_DONE

All five units are active and the first boot marker is present:

active
active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network ports; Kafka and Karapace listen on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:8081 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:8082 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:18081 users:(("python"))
127.0.0.1:18082 users:(("karapace_rest_p"))
[::]:22 users:(("sshd"),("systemd"))
[::]:8081 users:(("nginx"),("nginx"),("nginx"))
[::]:8082 users:(("nginx"),("nginx"),("nginx"))
[::ffff:127.0.0.1]:9092 users:(("java"))
[::ffff:127.0.0.1]:9093 users:(("java"))

Step 4: Get your URLs and passwords

sudo cat /root/kafka-karapace-credentials.txt

The file is readable by root only:

# Apache Kafka + Karapace 4.4 on Ubuntu 24.04 - generated on first boot by kafka-karapace-firstboot.service.
# Both passwords are unique to this instance; none was baked into the image. Keep this file secret.
# Schema Registry and REST proxy: HTTPS with HTTP Basic auth (HTTP_USER / HTTP_PASSWORD). The certificate is self
# signed (minted on this instance): compare its SHA-256 fingerprint with
#   sudo openssl x509 -in /etc/nginx/ssl/karapace.crt -noout -fingerprint -sha256
# or install a CA-issued certificate in /etc/nginx/ssl. Change the served name: sudo kafka-karapace-set-hostname <dns-name>
SCHEMA_REGISTRY_URL=https://<public-ip>:8081/
REST_PROXY_URL=https://<public-ip>:8082/
HTTP_USER=cloudimg
HTTP_PASSWORD=<HTTP_PASSWORD>
# Kafka (loopback only: use an SSH tunnel, or run clients on this instance). Client config: /root/kafka-client.properties
KAFKA_BOOTSTRAP=127.0.0.1:9092
KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
KAFKA_SASL_MECHANISM=SCRAM-SHA-256
KAFKA_SASL_USER=cloudimg
KAFKA_SASL_PASSWORD=<KAFKA_SASL_PASSWORD>
# The REST proxy's consumer API (/consumers/...) needs the header 'Sec-Fetch-Site: same-origin' (cross-site guard).
# Re-verify the appliance: sudo kafka-karapace-selftest     User guide: https://www.cloudimg.co.uk/guides/apache-kafka-karapace-4-4-on-ubuntu-24-04-aws/

The URLs use the instance's public IP address from the EC2 instance metadata (the private address when the instance has no public one). The certificate also names the public DNS name, the private IP address, 127.0.0.1 and localhost. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it. To serve the endpoints under your own DNS name, see Step 9.

The first boot also writes two root-only helper files so that no command in this guide ever puts a password on a command line: /root/karapace-curl.conf (the HTTP login for curl -K) and /root/kafka-client.properties (the SCRAM login for Kafka's command line tools and clients).

Print the certificate's SHA-256 fingerprint over SSH:

sudo openssl x509 -in /etc/nginx/ssl/karapace.crt -noout -fingerprint -sha256

sha256 Fingerprint=<sha256-fingerprint>

The certificate is self signed and unique to this instance. Do not add it to your operating system's or browser's trust store. On the instance and for a quick test from your computer, pass the certificate file to the one command that needs it (curl --cacert, after comparing its fingerprint); for your applications, install a certificate from a certificate authority (Step 9).

Step 5: The Schema Registry (HTTPS, port 8081)

On the instance the commands use https://127.0.0.1:8081/ (the instance's own names are all served, and the certificate names 127.0.0.1); from elsewhere use your SCHEMA_REGISTRY_URL. sudo curl -K /root/karapace-curl.conf reads the login from the root-only file. Without it the registry answers 401:

curl -s -o /dev/null -w 'anonymous: HTTP %{http_code}\n' --cacert /etc/nginx/ssl/karapace.crt https://127.0.0.1:8081/subjects
sudo curl -s --cacert /etc/nginx/ssl/karapace.crt -K /root/karapace-curl.conf https://127.0.0.1:8081/subjects; echo

anonymous: HTTP 401
[]

Register an Avro schema for the subject orders-value, read it back, and check whether a change is compatible before you deploy it (the registry's default compatibility mode is BACKWARD):

SR=https://127.0.0.1:8081
C=(-s --cacert /etc/nginx/ssl/karapace.crt -K /root/karapace-curl.conf -H 'Content-Type: application/vnd.schemaregistry.v1+json')
sudo curl "${C[@]}" --data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"shop\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}' "$SR/subjects/orders-value/versions"; echo
sudo curl "${C[@]}" "$SR/subjects/orders-value/versions/latest"; echo
sudo curl "${C[@]}" --data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"shop\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"EUR\"}]}"}' "$SR/compatibility/subjects/orders-value/versions/latest"; echo
sudo curl "${C[@]}" --data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"shop\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"}]}"}' "$SR/compatibility/subjects/orders-value/versions/latest"; echo

{"id":1}
{"subject":"orders-value","version":1,"id":1,"schema":"{\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}],\"name\":\"Order\",\"namespace\":\"shop\",\"type\":\"record\"}"}
{"is_compatible":true}
{"is_compatible":false,"messages":["reader type: int not compatible with writer type: string"]}

Adding a field with a default is a compatible change; changing a field's type is not, and registering it would be refused. Karapace stores the schemas in the Kafka topic _schemas, so they live as long as the broker's data.

Step 6: The Kafka REST proxy (HTTPS, port 8082)

The REST proxy produces to and consumes from Kafka topics over HTTP. It does not create topics; the image pre-creates the topic cloudimg (create your own with Step 7). Produce a JSON record:

RP=https://127.0.0.1:8082
C=(-s --cacert /etc/nginx/ssl/karapace.crt -K /root/karapace-curl.conf)
sudo curl "${C[@]}" -H 'Accept: application/vnd.kafka.v2+json' "$RP/topics"; echo
sudo curl "${C[@]}" -H 'Content-Type: application/vnd.kafka.json.v2+json' -H 'Accept: application/vnd.kafka.v2+json' \
  --data '{"records":[{"key":"order-1","value":{"id":"order-1","amount":42.5}}]}' "$RP/topics/cloudimg"; echo

["cloudimg","_schemas","__consumer_offsets"]
{"key_schema_id":null,"offsets":[{"offset":0,"partition":0}],"value_schema_id":null}

To consume, create a consumer instance, subscribe it and fetch records. Every request under /consumers/ must carry the header Sec-Fetch-Site: same-origin (the cross-site guard; nginx answers 403 without it, before the request reaches Karapace):

RP=https://127.0.0.1:8082
C=(-s --cacert /etc/nginx/ssl/karapace.crt -K /root/karapace-curl.conf -H 'Sec-Fetch-Site: same-origin')
sudo curl "${C[@]}" -H 'Content-Type: application/vnd.kafka.v2+json' --data '{"name":"guide-1","format":"json","auto.offset.reset":"earliest"}' "$RP/consumers/guide-group"; echo
sudo curl "${C[@]}" -o /dev/null -w 'subscribe: HTTP %{http_code}\n' -H 'Content-Type: application/vnd.kafka.v2+json' --data '{"topics":["cloudimg"]}' "$RP/consumers/guide-group/instances/guide-1/subscription"
for i in 1 2 3 4 5; do R=$(sudo curl "${C[@]}" -H 'Accept: application/vnd.kafka.json.v2+json' "$RP/consumers/guide-group/instances/guide-1/records?timeout=3000"); [ "$R" != "[]" ] && break; done; echo "$R"
sudo curl -s --cacert /etc/nginx/ssl/karapace.crt -K /root/karapace-curl.conf -o /dev/null -w 'without the header: HTTP %{http_code}\n' -H 'Accept: application/vnd.kafka.json.v2+json' "$RP/consumers/guide-group/instances/guide-1/records"
sudo curl "${C[@]}" -o /dev/null -w 'delete consumer: HTTP %{http_code}\n' -X DELETE -H 'Content-Type: application/vnd.kafka.v2+json' "$RP/consumers/guide-group/instances/guide-1"

{"base_uri":"consumers/guide-group/instances/guide-1","instance_id":"guide-1"}
subscribe: HTTP 204
[{"key":"order-1","offset":0,"partition":0,"timestamp":1793222378799,"topic":"cloudimg","value":{"amount":42.5,"id":"order-1"}}]
without the header: HTTP 403
delete consumer: HTTP 204

From your own application, set the header once on the HTTP session so every consumer call carries it. A minimal Python REST proxy client (pip install requests) that reads the HTTP login from ~/.netrc (machine <instance-host> login cloudimg password <HTTP_PASSWORD>, mode 0600), so the password is never in the code or on a command line:

import requests

RP = "https://<instance-host>:8082"
s = requests.Session()                      # the login comes from ~/.netrc
s.headers["Sec-Fetch-Site"] = "same-origin"  # required on every /consumers/ call
s.verify = "karapace.crt"                    # this instance's certificate (fingerprint checked, Step 4),
                                             # or True once a CA-issued certificate is installed (Step 9)
V2 = "application/vnd.kafka.v2+json"

r = s.post(f"{RP}/consumers/app-group", headers={"Content-Type": V2},
           json={"name": "app-1", "format": "json", "auto.offset.reset": "earliest"})
r.raise_for_status()
base = f"{RP}/{r.json()['base_uri'].lstrip('/')}"  # Karapace returns a relative base_uri
s.post(f"{base}/subscription", headers={"Content-Type": V2}, json={"topics": ["cloudimg"]}).raise_for_status()
try:
    for rec in s.get(f"{base}/records", params={"timeout": 3000},
                     headers={"Accept": "application/vnd.kafka.json.v2+json"}).json():
        print(rec["topic"], rec["offset"], rec["value"])
finally:
    s.delete(base, headers={"Content-Type": V2})

karapace.crt here is a copy of /etc/nginx/ssl/karapace.crt that you fetched over SSH and whose SHA-256 fingerprint you compared with the one the instance prints (Step 4). The client uses it only for this connection; do not add it to your operating system's, browser's or Java runtime's trust store. Without the Sec-Fetch-Site header every /consumers/ call gets HTTP 403.

Consumers commit their offsets automatically as they fetch (Karapace's default), so a new consumer in the same group continues where the last one stopped. The REST proxy also produces Avro, JSON Schema and Protobuf records (application/vnd.kafka.avro.v2+json and friends, with value_schema or value_schema_id), registering and looking up schemas in the Schema Registry for you.

Notes:

  • nginx refuses (400) any REST proxy URL whose path carries an encoded /, ., \ or NUL, or a .. segment. Kafka topic names cannot contain those characters; give consumer groups and instances names without them.
  • A browser page on another site cannot read any response (no CORS headers are sent) and cannot call the consumer API with a password your browser remembers. Applications that run in a browser on your own domain should go through your own backend instead.

Step 7: Kafka itself (SASL/SCRAM on 127.0.0.1:9092)

Kafka's command line tools are in /opt/kafka/bin. On the instance, --command-config /root/kafka-client.properties supplies the SCRAM login (the tools run under sudo because the file is root only):

K="sudo env LOG_DIR=/tmp/kafka-cli KAFKA_JMX_OPTS=-Dcli=1 KAFKA_OPTS= /opt/kafka/bin"
$K/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config /root/kafka-client.properties --create --if-not-exists --topic orders --partitions 3 --replication-factor 1
printf 'order-1:{"id":"order-1","amount":42.5}\norder-2:{"id":"order-2","amount":17.0}\n' | $K/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 \
  --command-config /root/kafka-client.properties --topic orders --reader-property parse.key=true --reader-property key.separator=:
$K/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --command-config /root/kafka-client.properties --topic orders \
  --from-beginning --max-messages 2 --timeout-ms 20000 --formatter-property print.key=true 2>/dev/null
$K/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config /root/kafka-client.properties --list

Created topic orders.
order-2 {"id":"order-2","amount":17.0}
order-1 {"id":"order-1","amount":42.5}
__consumer_offsets
_schemas
cloudimg
orders

A client without the SCRAM login, or with a wrong password, is refused (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256).

Clients on your own computer. Kafka listens on the loopback address only, so reach it through an SSH tunnel. Copy the client config once (it holds the password: keep it private), open the tunnel and point any Kafka client at 127.0.0.1:9092:

ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /root/kafka-client.properties' > kafka-client.properties && chmod 600 kafka-client.properties
ssh -i <your-key.pem> -N -L 9092:127.0.0.1:9092 ubuntu@<public-ip>
# then, in another terminal on your computer (with Kafka's command line tools installed):
#   kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --command-config kafka-client.properties --list

The same three settings work for any Kafka client library (security.protocol=SASL_PLAINTEXT, sasl.mechanism=SCRAM-SHA-256 and the JAAS line or the library's username/password settings). The tunnel encrypts the traffic between your computer and the instance. For clients on other hosts without a tunnel, add a SASL_SSL listener with a certificate to /etc/kafka/server.properties (and the port to your security group); never add a SASL_PLAINTEXT listener on a public address.

The Schema Registry and REST proxy from your computer. Copy the certificate and the curl login, compare the fingerprint with the one from Step 4, then call the public URL:

ssh -i <your-key.pem> ubuntu@<public-ip> 'cat /etc/nginx/ssl/karapace.crt' > karapace.crt
openssl x509 -in karapace.crt -noout -fingerprint -sha256
ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /root/karapace-curl.conf' > karapace-curl.conf && chmod 600 karapace-curl.conf
curl -s --cacert karapace.crt -K karapace-curl.conf https://<public-ip>:8081/subjects

Kafka clients that use the Schema Registry (Confluent serializers and compatible libraries) take schema.registry.url=https://<your-dns-name>:8081, basic.auth.credentials.source=USER_INFO and basic.auth.user.info=cloudimg:<HTTP_PASSWORD>. Give production applications a certificate from a certificate authority (Step 9) so they verify it with their normal trust settings.

Amazon MSK IAM authentication and other instance-role features are not available to these services: the instance metadata service is blocked for Kafka, Karapace, nginx and the boot service on purpose (Step 11).

Step 8: Configuration and tuning

Path What it is
/etc/kafka/server.properties broker settings (listeners, retention, partitions)
/etc/default/kafka broker JVM heap (KAFKA_HEAP_OPTS, default 1 GB)
/etc/karapace/registry.env, /etc/karapace/rest.env Karapace settings (KARAPACE_* variables, written at first boot)
/var/lib/kafka/data Kafka's data and metadata (the schemas live in the _schemas topic)
/etc/nginx/sites-available/cloudimg-kafka-karapace the HTTPS front door; certificate and key in /etc/nginx/ssl/

After changing a file, restart the service it belongs to and watch it come up:

sudo systemctl restart kafka karapace-registry karapace-rest && sudo journalctl -u kafka -u karapace-registry -u karapace-rest -n 20 --no-pager

Keep listeners on 127.0.0.1 (or add a SASL_SSL listener) and KARAPACE_HOST=127.0.0.1: the configuration checks refuse to start Kafka with a listener on another address and Karapace on anything but 127.0.0.1. For example, KARAPACE_COMPATIBILITY in registry.env sets the default compatibility mode (BACKWARD, FORWARD, FULL, their _TRANSITIVE forms or NONE); per subject use PUT /config/<subject>.

Step 9: Your own DNS name and certificate

Point a DNS record at the instance, then tell the appliance to serve that name. This updates nginx, issues a new self signed end-entity certificate that names it, accepts https://<name>:8081 and https://<name>:8082 as allowed origins and updates the URLs in the credentials file:

sudo kafka-karapace-set-hostname kafka.<your-domain>

The name is saved in /etc/cloudimg/kafka-karapace-hostname and survives reboots. For a certificate clients trust without extra settings, use one from a certificate authority (your own CA or Let's Encrypt with a DNS challenge): replace /etc/nginx/ssl/karapace.crt and /etc/nginx/ssl/karapace.key (keep the key mode 0600) and run sudo systemctl reload nginx. Running kafka-karapace-set-hostname again later issues a new self signed certificate over it, so set the name first and install your certificate afterwards.

Step 10: Rotating the passwords

Both passwords are random 32-character hexadecimal values, and the configuration checks accept only that form, so rotation generates a new random value. Read it from /root/kafka-karapace-credentials.txt afterwards.

The HTTP password (Schema Registry and REST proxy). The new value reaches every file through standard input, never a command line:

P=$(openssl rand -hex 16)
printf 'cloudimg:%s\n' "$(printf '%s\n' "$P" | openssl passwd -6 -stdin)" | sudo tee /etc/nginx/karapace.htpasswd >/dev/null
B=$(printf 'cloudimg:%s' "$P" | base64 -w0)
printf '"~^[Bb][Aa][Ss][Ii][Cc] +%s$" "";\n' "${B//+/\\+}" | sudo tee /etc/nginx/kafka-karapace-authkey.conf >/dev/null
printf 'user = "cloudimg:%s"\n' "$P" | sudo tee /root/karapace-curl.conf >/dev/null
printf '%s' "$P" | sudo python3 -I -c '
import re, sys
p = sys.stdin.read(); f = "/root/kafka-karapace-credentials.txt"
s = open(f).read(); open(f, "w").write(re.sub(r"(?m)^HTTP_PASSWORD=.*$", lambda m: "HTTP_PASSWORD=" + p, s))
'
unset P B; sudo nginx -t && sudo systemctl reload nginx

The Kafka SCRAM password. The broker's SCRAM credential is changed first, then every file that holds it, then the services restart. Kafka's kafka-configs.sh would need the new password on its command line, so the block runs the same tool (org.apache.kafka.tools.ConfigCommand) through a root-only Java argument file instead: the command line only ever shows the file's name, and the file is shredded afterwards. The files are only rewritten once Kafka has confirmed the change:

P=$(openssl rand -hex 16); AF=$(sudo mktemp /run/kk-scram.XXXXXX); OUT=$(sudo mktemp /run/kk-scram-out.XXXXXX)
printf '%s\n' org.apache.kafka.tools.ConfigCommand --bootstrap-server 127.0.0.1:9092 --command-config /root/kafka-client.properties \
  --alter --entity-type users --entity-name cloudimg --add-config "SCRAM-SHA-256=[password=$P]" | sudo tee "$AF" >/dev/null
sudo sh -c 'java -Xmx256m -Dlog4j2.configurationFile=/opt/kafka/config/tools-log4j2.yaml -cp "/opt/kafka/libs/*" "@$1" > "$2" 2>&1' rotate "$AF" "$OUT"
sudo shred -u "$AF"
if sudo grep -q "^Completed updating config for user cloudimg" "$OUT"; then
printf '%s' "$P" | sudo python3 -I -c '
import re, sys
p = sys.stdin.read()
for f in ("/etc/kafka/kafka_server_jaas.conf", "/etc/kafka/cloudimg-admin.properties", "/root/kafka-client.properties"):
    s = open(f).read(); open(f, "w").write(re.sub(r"password=\"[0-9a-f]{32}\"", lambda m: "password=\"%s\"" % p, s))
for f in ("/etc/karapace/registry.env", "/etc/karapace/rest.env"):
    s = open(f).read(); open(f, "w").write(re.sub(r"(?m)^KARAPACE_SASL_PLAIN_PASSWORD=.*$", lambda m: "KARAPACE_SASL_PLAIN_PASSWORD=" + p, s))
f = "/root/kafka-karapace-credentials.txt"
s = open(f).read(); open(f, "w").write(re.sub(r"(?m)^KAFKA_SASL_PASSWORD=.*$", lambda m: "KAFKA_SASL_PASSWORD=" + p, s))
'
  sudo systemctl restart kafka karapace-registry karapace-rest && echo "SCRAM password rotated"
else
  echo "Kafka did not accept the change; nothing was rewritten"
fi
sudo shred -u "$OUT"; unset P

Then run sudo kafka-karapace-selftest (Step 11) to prove both passwords work. Additional Kafka users are created the same way (--entity-name <user> in the argument file); add an authorizer and ACLs in server.properties before you hand them out (by default every authenticated user may do everything).

Step 11: Security checks

kafka-karapace-selftest re-proves the whole appliance with this instance's credentials: the listening ports, a Kafka produce and consume round trip with the SCRAM password (and that a wrong password is refused), the Schema Registry (anonymous and wrong-password requests refused, register, fetch and a compatible and an incompatible change), the REST proxy (JSON and Avro produce, consume through the consumer API), the cross-origin and cross-site guards, the redirect and the refusal of other host names, the metadata block inside every service and that the served certificate is this instance's end-entity certificate:

sudo kafka-karapace-selftest

Apache Kafka + Karapace appliance self-test (https://<public-ip>:8081/ https://<public-ip>:8082/)
  [ok] public listeners 22, 8081, 8082 only; Kafka (9092/9093) and Karapace (18081/18082) refuse the private address
  [ok] Kafka cloudimg-selftest-<timestamp>: SCRAM-SHA-256 login, produce -> consume round trip; a wrong SCRAM password is refused
  [ok] Schema Registry: anonymous and wrong password -> 401 (one Basic challenge); register -> id 2 -> fetch; compatible change true, incompatible false; cross-origin POST -> 403
  [ok] REST proxy: anonymous and wrong password -> 401; JSON produce + Avro produce (schema id 3 in the registry); consumer create -> subscribe -> records returned the JSON record
  [ok] REST proxy consumer API: cross-site, same-site, none, no signal, foreign Origin and foreign Referer -> nginx 403 (the records were consumed only by the same-origin request)
  [ok] 8081/8082: plain HTTP -> 301 https://<public-ip>:<port>/; unknown Host -> 444; WebSocket upgrade -> 403; CONNECT refused
  [ok] instance metadata refused inside the kafka, karapace-registry, karapace-rest and nginx cgroups (loopback control 200)
  [ok] TLS certificate: end-entity (CA:FALSE, serverAuth, no certificate signing), names <public-ip>; SHA-256 fingerprint <sha256-fingerprint>
  [ok] cross-site GET guard (consumer records): 13/13 (/consumers/cgx-<timestamp>/instances/cix-<timestamp>/records)
SELFTEST OK - Kafka SCRAM round trip, Schema Registry register/fetch/compatibility and REST proxy produce/consume over HTTPS (end-entity certificate, per-instance passwords), origin and cross-site guards, loopback-only Kafka and Karapace, metadata block verified.

The metadata block is a systemd IPAddressDeny rule on each service; you can see it in effect:

for u in kafka karapace-registry karapace-rest nginx kafka-karapace-firstboot; do printf '%s: ' $u; systemctl show -p IPAddressDeny --value $u.service; done

kafka: fd00:ec2::254/128 168.63.129.16/32 169.254.169.254/32
karapace-registry: fd00:ec2::254/128 168.63.129.16/32 169.254.169.254/32
karapace-rest: fd00:ec2::254/128 168.63.129.16/32 169.254.169.254/32
nginx: fd00:ec2::254/128 168.63.129.16/32 169.254.169.254/32
kafka-karapace-firstboot: fd00:ec2::254/128 168.63.129.16/32 169.254.169.254/32

Step 12: Versions, provenance and file locations

sed -n '1,9p' /usr/share/doc/karapace/BUILD-PROVENANCE.txt
ls /opt/kafka/libs/kafka-clients-*.jar
java -version 2>&1 | sed -n 1p

Karapace 6.2.4 (Aiven-Open/karapace, Apache License 2.0) - built from the release tarball of the tag
source          https://github.com/Aiven-Open/karapace/archive/refs/tags/6.2.4.tar.gz
sha256          369da80c38f3de3449512e3c0a4fc47d15d58e9b53daa9c6fce0ab02c3da4224
tag commit      09cba53ffcc961038f608b34f7d00fc2df1d313c
licence         /usr/share/doc/karapace/LICENSE (sha256 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30)
dependencies    upstream requirements/requirements.txt (pinned, pip --no-deps): aiohttp==3.14.3 aiokafka==0.12.0 confluent-kafka==2.15.0 fastapi==0.141.1 protobuf==5.29.6
build only      Go 1.25.14 (a21ae5633a269bcd7e90cf767e48225633795e99d831742cbf3397064fee7712), gcc, python3.12-dev - not shipped
Apache Kafka    4.4.0 (Scala 2.13), https://archive.apache.org/dist/kafka/4.4.0/kafka_2.13-4.4.0.tgz
                sha512 f5ceba8a265ab6bb2d68362b1f6859a95e15d60ef92ba83b43fdf065b557dc18e98d6383c15b9977320e25b3a2c8b6f11decb17b0fa6c0f163423793a0eb95cb; OpenJDK 21 (Ubuntu main); 127.0.0.1 only
/opt/kafka/libs/kafka-clients-4.4.0.jar
openjdk version "21.0.12.1" 2026-08-18
Path What it is
/root/kafka-karapace-credentials.txt URLs and both passwords (root only)
/root/kafka-client.properties, /root/karapace-curl.conf the Kafka and HTTP logins for tools (root only)
/opt/kafka Apache Kafka 4.4.0 (its LICENSE and NOTICE are there)
/opt/karapace/venv Karapace 6.2.4 and its Python dependencies
/usr/share/doc/karapace/ Karapace's LICENSE and BUILD-PROVENANCE.txt
journalctl -u kafka Kafka's log (also karapace-registry, karapace-rest, nginx, kafka-karapace-firstboot)

Step 13: Backup and maintenance

  • Back up /var/lib/kafka/data (stop Kafka first, or use your own replication tooling) together with /etc/kafka, /etc/karapace, /etc/nginx/ssl, /etc/cloudimg and /root/kafka-karapace-credentials.txt. The SCRAM credentials live in Kafka's metadata, so a restored data directory needs the matching /etc/kafka files.
  • Ubuntu security updates install automatically (unattended-upgrades), including the OpenJDK runtime Kafka uses. Kafka in /opt/kafka and Karapace in /opt/karapace are updated by launching the newer version from the listing and moving your data across.
  • Kafka runs with a 1 GB heap and the two Karapace services need a few hundred MB each; an m5.large is comfortable. For heavier traffic give Kafka more heap in /etc/default/kafka and use a larger instance and a bigger data volume (or grow the root volume).
  • This is a single-node broker (replication factor 1). For production durability run a multi-broker cluster.

Step 14: Security recommendations

  • Limit TCP 8081 and 8082 in the security group to the networks your applications run in, and TCP 22 to your admin network. The REST proxy can read and write every topic; treat the HTTP password like a Kafka admin password.
  • Serve the endpoints under your own DNS name with a certificate from a trusted CA (Step 9). Until then, check the self signed certificate's fingerprint (Step 4) rather than adding it to a trust store.
  • Rotate both passwords if you share the credentials file (Step 10), and keep clients' copies of kafka-client.properties and karapace-curl.conf private.
  • Reach Kafka through SSH tunnels or a SASL_SSL listener only; keep everything else on 127.0.0.1 and publish nothing but nginx.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk. Apache Kafka is licensed under the Apache License 2.0 (its LICENSE file is in /opt/kafka), Karapace under the Apache License 2.0 (its LICENSE file is on the instance in /usr/share/doc/karapace), and OpenJDK under the GNU General Public License version 2 with the Classpath Exception. Apache Kafka and Kafka are trademarks of the Apache Software Foundation, and Karapace is an open source project of Aiven; cloudimg is not affiliated with or endorsed by the Apache Software Foundation or Aiven.