Streaming & Messaging Azure

Karapace on Ubuntu 24.04 on Azure User Guide

| Product: Karapace on Ubuntu 24.04 LTS on Azure

Overview

Karapace is an open source implementation of the Schema Registry and Kafka REST Proxy APIs. The schema registry gives your event streams a central, versioned contract: producers and consumers store Avro, JSON Schema and Protobuf schemas under a subject, every schema keeps its own version history, and the registry checks each proposed change against a compatibility rule so a breaking change is caught before it reaches production rather than after a consumer has already failed on it. The REST proxy exposes the same cluster over plain HTTP, so any client that can make a web request can list topics, produce messages and consume them without a native Kafka driver.

Karapace is a registry and a proxy, not a store: its entire persistence layer is a compacted Kafka topic called _schemas. On its own it would start and then fail every real operation, so this cloudimg image bundles a broker with it: a single node Apache Kafka 4.3.1 broker running in KRaft mode, with no ZooKeeper, already wired up as Karapace's storage. The result is a complete, self contained schema registry appliance that is useful the moment the instance boots. If you already run a Kafka cluster, Step 9 shows how to point the registry at it instead.

The image ships Karapace 6.2.1 behind an nginx reverse proxy on a hardened, fully patched Ubuntu 24.04 LTS base. Karapace has no authentication of its own in this configuration, and an open schema registry lets anyone read, overwrite or delete every schema your producers and consumers depend on, so both APIs are protected with HTTP Basic authentication whose password is generated uniquely on the first boot of every VM. Both Karapace processes and the Kafka broker are bound to the loopback interface, so that authentication cannot be bypassed. Backed by 24/7 cloudimg support.

What is included:

  • Karapace 6.2.1 serving the Schema Registry API on port 80 and the Kafka REST Proxy API on port 8082, both through nginx
  • A bundled single node Apache Kafka 4.3.1 broker in KRaft mode, formatted fresh on each VM's first boot
  • HTTP Basic authentication on both APIs, with a per VM password generated on first boot and recorded in a root only file
  • No shipped default login: the password file is locked in the image, so no known or blank credential authenticates
  • The Kafka broker bound to 127.0.0.1 only (ports 9092 and 9093), never exposed to the network
  • The Karapace registry (8081) and REST proxy (8083) bound to 127.0.0.1 only, reachable solely through the authenticated nginx front
  • kafka.service, karapace-registry.service, karapace-rest.service and nginx.service as enabled systemd units
  • An unauthenticated /healthz endpoint for load balancer probes
  • No compiler or build toolchain in the shipped image

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is what the image is tuned for and is sufficient for the bundled broker plus both Karapace services; size up for heavier REST proxy workloads or larger schema volumes. NSG inbound: allow 22/tcp from your management network, 80/tcp for the schema registry API, and 8082/tcp if you intend to use the REST proxy. Karapace serves plain HTTP; for production, terminate TLS in front of it with your own domain. The bundled Kafka broker is never exposed: it listens on 127.0.0.1 only, so ports 9092 and 9093 stay off the network.

Step 1 - Deploy the image

Option A: Azure portal

  1. In the Azure portal, open Create a resource and search for Karapace on Ubuntu 24.04 LTS by cloudimg.
  2. Select the plan and click Create.
  3. On the Basics tab, choose your subscription, resource group and region, name the VM, and select Standard_B2s.
  4. Set Authentication type to SSH public key, with azureuser as the username, and provide your public key.
  5. On the Networking tab, allow inbound 22/tcp (from your IP), 80/tcp, and 8082/tcp if you need the REST proxy.
  6. Review and create. When deployment completes, note the VM's public IP address.

Option B: Azure CLI

az vm create \
  --resource-group my-karapace-rg \
  --name karapace \
  --image cloudimg:karapace-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard
az vm open-port --resource-group my-karapace-rg --name karapace --port 80 --priority 900
az vm open-port --resource-group my-karapace-rg --name karapace --port 8082 --priority 910

Step 2 - Confirm the services are running

SSH in as azureuser and confirm the four systemd units are active. kafka.service is the bundled broker, karapace-registry.service is the schema registry, karapace-rest.service is the Kafka REST proxy, and nginx.service is the authenticated reverse proxy.

systemctl is-active kafka.service karapace-registry.service karapace-rest.service nginx.service

Every service reports active.

The listener layout is the core of the security model: only nginx is on a public interface, and everything behind it is loopback only.

ss -lnt | grep -E ':(80|8082|8081|8083|9092|9093)\b'

You will see nginx listening on 0.0.0.0:80 and 0.0.0.0:8082, while the Karapace registry (127.0.0.1:8081), the Karapace REST proxy (127.0.0.1:8083) and the Kafka broker (127.0.0.1:9092 and 127.0.0.1:9093) are all bound to loopback. The broker's listener may be shown in the IPv4 mapped form [::ffff:127.0.0.1]:9092, which is the same loopback address.

An unauthenticated health endpoint is available for load balancer probes:

curl -s -o /dev/null -w 'GET /healthz -> HTTP %{http_code}\n' http://127.0.0.1/healthz

This returns HTTP 200.

The registry also exposes its own readiness detail. Note that this endpoint answers 200 with "healthy":true from the moment the process starts, while schema_registry_ready stays false until it has finished replaying the _schemas topic and elected itself primary. Writes only succeed once both inner flags are true, which normally takes a few seconds after boot:

curl -s http://127.0.0.1:8081/_health | python3 -m json.tool | head -12

The kafka, karapace-registry, karapace-rest and nginx systemd services all reporting active, with nginx listening on ports 80 and 8082 while the Karapace registry on 8081, the REST proxy on 8083 and the Kafka broker on 9092 and 9093 are bound to loopback only, and the unauthenticated health endpoint returning HTTP 200

Step 3 - Retrieve the per VM password

Both APIs require HTTP Basic authentication. The username is karapace and the password was generated on this VM during its first boot. It is stored in a root only file, and no other VM shares it.

sudo cat /root/karapace-info.txt

The file records the registry URL, the REST proxy URL, the username and the generated password. The file is mode 600 and owned by root:

stat -c '%n mode=%a owner=%U:%G' /root/karapace-info.txt

Confirm the secure by default posture. Without credentials, and with a wrong password, the registry refuses the request; with the per VM password it answers:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -o /dev/null -w 'no credentials  -> HTTP %{http_code}\n' http://127.0.0.1/subjects
curl -s -o /dev/null -w 'wrong password  -> HTTP %{http_code}\n' -u karapace:wrong-password http://127.0.0.1/subjects
curl -s -o /dev/null -w 'per VM password -> HTTP %{http_code}\n' -u "karapace:$KPASS" http://127.0.0.1/subjects

The first two return HTTP 401 and the third returns HTTP 200.

The per VM Karapace credentials file, owned root only with mode 600, showing the registry URL, the REST proxy URL, the username and the generated password masked, alongside the authentication round trip where an unauthenticated request and a wrong password are both rejected with 401 and the per VM password is accepted with 200

From your workstation, use the VM's public IP address in place of 127.0.0.1, for example curl -u karapace:<karapace-password> http://<vm-public-ip>/subjects.

Step 4 - Register your first schema

A subject is the namespace a schema lives under. The common convention is <topic>-value for the message value and <topic>-key for the key. Register an Avro schema for a payments topic:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"}]}"}' \
  http://127.0.0.1/subjects/payments-value/versions

The registry replies with the global identifier it assigned to the schema, for example {"id":1}. Registering the identical schema again is idempotent and returns the same id.

Step 5 - Read schemas back

List every subject the registry knows about:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/subjects

Read the latest version of a subject, which returns the subject, its version, the schema id and the schema itself:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/subjects/payments-value/versions/latest

List the versions of a subject, and fetch a schema directly by its global id:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/subjects/payments-value/versions
curl -s -u "karapace:$KPASS" http://127.0.0.1/schemas/ids/1

Step 6 - Check compatibility before you break a consumer

This is the reason a schema registry exists. Each subject is governed by a compatibility rule, and the registry evaluates a proposed schema against the existing version before you ever deploy it. The image default is BACKWARD, meaning a consumer using the new schema must still be able to read data written with the old one.

Check the current rule:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1/config

Adding a field with a default is backward compatible, because a consumer reading old data simply uses the default:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"GBP\"}]}"}' \
  http://127.0.0.1/compatibility/subjects/payments-value/versions/latest

This returns {"is_compatible":true}.

Adding a field without a default is not backward compatible, because a consumer using the new schema has no way to populate that field when it reads data written before the field existed. The registry catches it:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Payment\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"reference\",\"type\":\"string\"}]}"}' \
  http://127.0.0.1/compatibility/subjects/payments-value/versions/latest

This returns {"is_compatible":false} along with the offending field name, reference. That is the check you run in CI before a producer change is merged.

It is worth knowing which direction each rule protects, because the intuition is easy to get backwards. Under BACKWARD, removing a field is allowed, since a consumer on the new schema simply ignores the extra data it finds in old records; it is adding a mandatory field that breaks, as shown above. Under FORWARD the reverse holds. Use FULL when you need both to be safe.

To change the rule, for a single subject or globally, PUT a new compatibility level. Valid values are BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE and NONE:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X PUT \
  -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data '{"compatibility": "FULL"}' \
  http://127.0.0.1/config/payments-value

Registering an Avro schema for the payments-value subject and receiving a schema id, listing the subjects, reading the schema back with its version and id, then a compatibility check where adding a field with a default is accepted as compatible and adding a mandatory field without a default is correctly rejected as incompatible

Step 7 - Use the Kafka REST proxy

The REST proxy on port 8082 exposes the bundled broker over plain HTTP, so a client with no native Kafka driver can still work with topics. List the topics:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1:8082/topics

Inspect a single topic's partitions:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" http://127.0.0.1:8082/topics/_schemas

To produce a message, the topic must already exist: the REST proxy does not create it for you, and producing to an unknown topic returns {"error_code":40401,"message":"Topic payments not found"}. Create it first with the bundled Kafka tools:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 \
  --create --if-not-exists --topic payments --partitions 1 --replication-factor 1

Now produce a JSON message to it:

KPASS=$(sudo awk -F= '/^karapace_password=/{print $2; exit}' /root/karapace-info.txt)
curl -s -u "karapace:$KPASS" -X POST \
  -H 'Content-Type: application/vnd.kafka.json.v2+json' \
  --data '{"records":[{"value":{"id":"pay-001","amount":42.5}}]}' \
  http://127.0.0.1:8082/topics/payments

The response reports the partition and offset each record landed at:

{"key_schema_id":null,"offsets":[{"offset":0,"partition":0}],"value_schema_id":null}

The Kafka REST proxy listing the topics on the bundled broker over plain HTTP, the _schemas topic described from the Kafka command line tools showing it really holds the registry's data, the installed Karapace, Apache Kafka, Java and base OS versions, and a security summary reporting zero held packages, zero pending security updates and no Go compiler left in the image

Step 8 - Work with the bundled broker from the command line

The standard Apache Kafka tools are installed at /opt/kafka/bin and point at the loopback broker. Confirm that the registry's data really is a Kafka topic:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list

_schemas is the compacted topic Karapace stores every schema in. Describe it:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --describe --topic _schemas

Create your own topic:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server 127.0.0.1:9092 \
  --create --if-not-exists --topic orders --partitions 3 --replication-factor 1

Because the broker is single node, the replication factor must be 1.

Step 9 - Point Karapace at an external Kafka cluster

The bundled broker makes the image self contained, but Karapace is equally happy talking to a cluster you already run. Karapace is configured entirely through KARAPACE_ prefixed environment variables in its systemd units.

Rather than editing the shipped unit files, add a systemd drop-in for each service, which survives future package changes. Create /etc/systemd/system/karapace-registry.service.d/external-kafka.conf with your own bootstrap address:

[Service]
Environment=KARAPACE_BOOTSTRAP_URI=kafka-1.example.internal:9092,kafka-2.example.internal:9092
Environment=KARAPACE_REPLICATION_FACTOR=3

Raising KARAPACE_REPLICATION_FACTOR to 3 matters on a real cluster: it is the replication factor Karapace uses when it creates the _schemas topic, and a single replica would put your entire schema history on one broker. Create the matching /etc/systemd/system/karapace-rest.service.d/external-kafka.conf with the same KARAPACE_BOOTSTRAP_URI line.

If your cluster requires TLS or SASL, set KARAPACE_SECURITY_PROTOCOL (for example SSL or SASL_SSL) together with the matching KARAPACE_SSL_CAFILE, KARAPACE_SSL_CERTFILE, KARAPACE_SSL_KEYFILE, or KARAPACE_SASL_MECHANISM, KARAPACE_SASL_PLAIN_USERNAME and KARAPACE_SASL_PLAIN_PASSWORD variables in the same drop-in.

Apply the change, then confirm the registry reports itself ready against the new cluster:

sudo systemctl daemon-reload
sudo systemctl restart karapace-registry.service karapace-rest.service
curl -s http://127.0.0.1:8081/_health | python3 -m json.tool | head -12

Once you are using an external cluster, the bundled broker is redundant and can be switched off to free its memory:

sudo systemctl disable --now kafka.service

Step 10 - Security model

  • No shipped credential. The nginx password file is locked in the image, so no known or blank password authenticates. On each VM's first boot, karapace-firstboot.service generates a unique password, writes it as a bcrypt hash into /etc/nginx/karapace.htpasswd, and records it in /root/karapace-info.txt with mode 600. Because the locked file is the shipped state, the appliance is fail closed: if first boot has not completed, every request is refused rather than served anonymously.
  • Authentication cannot be bypassed. Karapace serves both APIs with no authentication of its own here, so both processes are bound to 127.0.0.1. The only way in is through nginx, which requires HTTP Basic authentication on every path except /healthz.
  • The broker is never exposed. The bundled Kafka broker listens on 127.0.0.1:9092 and 127.0.0.1:9093 only. Its sole client is Karapace on the same host, which is what makes an unauthenticated PLAINTEXT broker safe here. Do not change its listeners to a public address without adding Kafka level authentication.
  • Each VM has its own cluster. The image ships with the KRaft storage wiped, so every VM formats a fresh, empty cluster on first boot. No two deployments share a cluster identity, and no schemas from the build ever reach a customer instance.
  • No build toolchain. Karapace is built from source at image build time, which needs a Go and C toolchain; both are removed from the image afterwards, so no compiler ships to customers.
  • Rotate the password at any time. htpasswd -B prompts for the new password twice and writes a bcrypt hash; reload nginx to pick it up, then record the new password somewhere safe, because /root/karapace-info.txt still shows the original:
sudo htpasswd -B /etc/nginx/karapace.htpasswd karapace
sudo systemctl reload nginx
  • Put TLS in front. Karapace serves plain HTTP. For anything beyond a trusted network, terminate TLS at nginx with your own certificate, or place the VM behind an Azure Application Gateway or Load Balancer that does.

Step 11 - Base image and versions

/opt/karapace/venv/bin/python -c 'from karapace.version import __version__; print("Karapace", __version__)'

ls -d /opt/kafka_* && java -version 2>&1 | head -1

The base is fully patched at build time, with unattended security upgrades left enabled for the life of the instance. Confirm there is nothing held back:

apt-mark showhold | wc -l

This prints 0.

Managing the services

systemctl status kafka.service karapace-registry.service karapace-rest.service nginx.service --no-pager

Follow the registry's log:

journalctl -u karapace-registry.service -n 50 --no-pager

The services start automatically on boot. karapace-firstboot.service runs once, before the others, and is skipped on later boots because it leaves a sentinel at /var/lib/cloudimg/karapace-firstboot.done.

Support

This image is published and supported by cloudimg with 24/7 support. It is not affiliated with or endorsed by Aiven or the Karapace project; Karapace and Apache Kafka are used here under the Apache License 2.0, and their names are used only to identify the software included in this image.

For help with this image, contact support@cloudimg.co.uk.