Apache Kafka + Karapace on Ubuntu 24.04 on Azure User Guide

Overview

This cloudimg image is a single-VM event-streaming appliance that pairs the Apache Kafka broker with Karapace — the open-source Confluent-compatible Schema Registry and Kafka REST proxy from Aiven. Kafka gives you durable publish/subscribe messaging; Karapace adds schema governance (Avro, JSON Schema, Protobuf) and an HTTP path for producing and consuming, so applications can talk to Kafka over plain REST or with strongly-typed schemas.

Kafka 4.2.0 (Scala 2.13) runs in single-VM KRaft mode (combined broker + controller, no ZooKeeper) with SASL/SCRAM-SHA-256 authentication. Karapace 6.2.0 runs as two services — the Schema Registry and the Kafka REST proxy — installed into a dedicated Python 3.12 virtual environment. A per-VM cloudimg password is generated at first boot and secures both the Kafka broker (SCRAM) and the Karapace endpoints (HTTP basic auth terminated by nginx).

What is included:

• Apache Kafka 4.2.0 (Scala 2.13) at /opt/kafka, single-VM KRaft broker+controller (kafka.service)
• Karapace 6.2.0 Schema Registry (karapace-registry.service) and Kafka REST proxy (karapace-rest.service)
• OpenJDK 17 JRE (Kafka runtime); Python 3.12 venv at /opt/karapace/venv (Karapace runtime)
• SASL/SCRAM-SHA-256 broker on 127.0.0.1:9092 (localhost-only — reachable off-host through the authenticated Karapace REST proxy)
• nginx auth wall exposing the Schema Registry on TCP 8081 and the Kafka REST proxy on TCP 8082
• Pre-created cloudimg topic (1 partition, replication 1)
• A single per-VM cloudimg password used for Kafka SCRAM and the Karapace basic-auth wall
• Credentials at /stage/scripts/kafka-karapace-credentials.log
• 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key, and a VNet + subnet. Standard_B2s (4 GB RAM) suits dev, test, single-tenant streaming, and PoC workloads. For production, raise to D4s/D8s and increase KAFKA_HEAP_OPTS in /etc/default/kafka.

NSG inbound rules: allow 22/tcp from your management CIDR, and 8081/tcp (Schema Registry) + 8082/tcp (Kafka REST proxy) from your client CIDR. The Kafka broker listens on 127.0.0.1:9092 (localhost only) and is not exposed directly — off-host clients use the authenticated REST proxy, or you add an external SASL_SSL listener yourself.

Step 1-3: Deploy + SSH

Deploy the image from the Azure Marketplace, then connect:

ssh azureuser@<vm-ip>

Step 4: Service status + versions

Confirm all four services are active and check the installed versions.

for s in kafka karapace-registry karapace-rest nginx; do
  printf '%-26s : ' "$s.service"; systemctl is-active "$s.service"
done
echo "Kafka  : $(ls /opt/kafka/libs/ | grep -oE '^kafka-clients-[0-9.]+' | head -1 | sed 's/kafka-clients-//')"
echo "Karapace: $(/opt/karapace/venv/bin/pip show karapace 2>/dev/null | awk '/^Version:/{print $2}')"

Step 5: Read the per-VM credentials

The first-boot service generates a unique password for this VM. Read it — the same password authenticates the Kafka broker and the Karapace endpoints.

sudo cat /stage/scripts/kafka-karapace-credentials.log

Note KAFKA_ADMIN_USER/KAFKA_ADMIN_PASSWORD, KARAPACE_USER/KARAPACE_PASSWORD, and the SCHEMA_REGISTRY_URL / REST_PROXY_URL values.

Step 6: Kafka produce and consume (SASL/SCRAM)

The broker rejects anonymous clients. Build an admin client config from the per-VM password, then produce a message to the cloudimg topic and consume it back.

PASS=$(sudo grep '^KAFKA_ADMIN_PASSWORD=' /stage/scripts/kafka-karapace-credentials.log | cut -d= -f2-)
cat > /tmp/cloudimg.properties <<EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="cloudimg" password="${PASS}";
EOF
echo -n 'anonymous client : '
timeout 12 /opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server 127.0.0.1:9092 >/dev/null 2>&1 && echo 'ACCEPTED' || echo 'REJECTED (auth required)'
MSG="cloudimg-demo-$(date +%s)"
echo "${MSG}" | /opt/kafka/bin/kafka-console-producer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/cloudimg.properties --topic cloudimg
echo "produced: ${MSG}"
echo -n 'consumed: '
timeout 20 /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --command-config /tmp/cloudimg.properties --topic cloudimg --from-beginning --timeout-ms 12000 2>/dev/null | grep -F "${MSG}" | head -1
rm -f /tmp/cloudimg.properties

Step 7: Karapace Schema Registry — register and read a schema

The Schema Registry is exposed on port 8081 behind an nginx basic-auth wall. Register an Avro schema for a subject, then read it back. Replace <vm-ip> with your VM's IP (or use 127.0.0.1 on the box).

PASS=$(sudo grep '^KARAPACE_PASSWORD=' /stage/scripts/kafka-karapace-credentials.log | cut -d= -f2-)
echo -n 'anonymous GET /subjects : HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 8 http://127.0.0.1:8081/subjects
echo 'register schema for subject demo-value:'
curl -s -u "cloudimg:${PASS}" -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  -d '{"schema":"{\"type\":\"record\",\"name\":\"demo\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"}]}"}' \
  http://127.0.0.1:8081/subjects/demo-value/versions
echo; echo 'read it back:'
curl -s -u "cloudimg:${PASS}" http://127.0.0.1:8081/subjects
echo
curl -s -u "cloudimg:${PASS}" http://127.0.0.1:8081/subjects/demo-value/versions/1
echo

Step 8: Karapace REST proxy — produce over HTTP

The Kafka REST proxy on port 8082 lets applications produce and consume without a Kafka client library. Produce a JSON record to the cloudimg topic over HTTP.

PASS=$(sudo grep '^KARAPACE_PASSWORD=' /stage/scripts/kafka-karapace-credentials.log | cut -d= -f2-)
echo -n 'anonymous GET /topics : HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 8 http://127.0.0.1:8082/topics
echo 'produce a JSON record to topic cloudimg:'
curl -s -u "cloudimg:${PASS}" -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  -d '{"records":[{"value":{"event":"order.created","id":42}}]}' \
  http://127.0.0.1:8082/topics/cloudimg
echo; echo 'topics visible to the REST proxy:'
curl -s -u "cloudimg:${PASS}" http://127.0.0.1:8082/topics
echo

Security notes

• No default credentials ship in the image. The cloudimg password is generated uniquely on each VM at first boot (openssl rand) and written to /stage/scripts/kafka-karapace-credentials.log (mode 0600, root only).
• One password, two walls. It secures the Kafka broker (SASL/SCRAM-SHA-256) and the Karapace Schema Registry + REST proxy (HTTP basic auth via nginx). Split them post-deploy if you prefer separate credentials.
• The broker is localhost-only. Kafka listens on 127.0.0.1:9092; the only off-host surfaces are the authenticated registry (8081) and REST proxy (8082). For direct broker access from other hosts, add an external SASL_SSL listener in /opt/kafka/config/server.properties and terminate TLS.
• Rotate the password by updating the Kafka SCRAM credential (kafka-configs.sh --alter --add-config 'SCRAM-SHA-256=...' --entity-type users --entity-name cloudimg) and the nginx htpasswd (htpasswd /etc/nginx/karapace.htpasswd cloudimg).

Next steps

• Add topics with kafka-topics.sh --create, and add brokers to scale the cluster horizontally.
• Set the registry's compatibility mode (BACKWARD by default) per subject via PUT /config/<subject>.
• Point your producers/consumers at the Schema Registry (http://<vm-ip>:8081) for schema validation, or at the REST proxy (http://<vm-ip>:8082) for HTTP-native streaming.

Backed by 24/7 cloudimg support.