Streaming Azure

Apache Kafka 4.2 on Ubuntu 22.04 on Azure User Guide

| Product: Apache Kafka 4.2 on Ubuntu 22.04 on Azure

Overview

This image runs Apache Kafka 4.2.0 as a single node KRaft broker on Ubuntu 22.04 LTS. The broker process combines the controller and broker roles in one JVM. Authentication is enforced on the client listener using SASL/SCRAM SHA 512. A unique admin password is generated on the first boot of every deployed virtual machine, so two virtual machines launched from the same gallery image never share credentials.

The image is intended for teams that want a production posture single broker on day one, without spending hours on packaging, systemd plumbing, JAAS files, or KRaft storage formatting. It is not a multi broker production cluster, it is not TLS encrypted out of the box, and it does not ship with Schema Registry, ksqlDB, Kafka Connect workers, or MirrorMaker. Section 14 documents the recommended path for adding TLS before you put real production traffic through the broker.

The brand is lowercase cloudimg throughout this guide. All cloudimg URLs in this guide use the form https://www.cloudimg.co.uk.

Prerequisites

Before you deploy this image you need:

  • A Microsoft Azure subscription where you can create resource groups, virtual networks, and virtual machines
  • Azure role permissions equivalent to Contributor on the target resource group
  • An SSH public key for first login to the build user account
  • A virtual network and subnet in the same region as the Azure Compute Gallery the image is published into, with an associated network security group
  • The Azure CLI (az version 2.50 or later) installed locally if you intend to use the CLI deployment path in Section 4
  • The cloudimg Apache Kafka 4.2 offer enabled on your tenant in Azure Marketplace

Step 1: Deploy the Virtual Machine from the Azure Portal

Navigate to Marketplace in the Azure Portal, search for Apache Kafka 4.2, and select the cloudimg publisher entry. Click Create to begin the wizard.

On the Basics tab choose your subscription, target resource group, and region. The region must match the region your Azure Compute Gallery exposes the image in. Set the virtual machine name. Choose SSH public key as the authentication type, set the username to a name of your choice, and paste your SSH public key.

On the Disks tab the recommended OS disk type is Premium SSD. Leave the OS disk size at the default. You can attach a separate Premium SSD data disk now if you intend to move the Kafka log directory to it later, or you can do that after the broker is running by following Section 13.

On the Networking tab select your existing virtual network and subnet. Attach a network security group that allows inbound TCP 22 from your management IP range and inbound TCP 9093 only from the virtual network CIDR. Do not expose 9093 to the public internet. The broker uses SASL_PLAINTEXT, which authenticates clients but does not encrypt traffic, so any exposure beyond a private network requires the TLS upgrade described in Section 14.

On the Management, Monitoring, and Advanced tabs the defaults are appropriate. Click Review + create, wait for validation to pass, then click Create. Deployment takes around three minutes.

Step 2: Deploy the Virtual Machine from the Azure CLI

If you prefer the command line, use the gallery image resource identifier as the source. The exact resource identifier is published on your Partner Center plan. A representative invocation:

RG="kafka-prod"
LOCATION="eastus"
VM_NAME="kafka-broker-1"
ADMIN_USER="kafkaops"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/apache-kafka-4-2-ubuntu-22-04/versions/1.0.20260414"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"

az group create --name "$RG" --location "$LOCATION"

az network vnet create \
  --resource-group "$RG" \
  --name kafka-vnet \
  --address-prefix 10.20.0.0/16 \
  --subnet-name kafka-subnet \
  --subnet-prefix 10.20.1.0/24

az network nsg create --resource-group "$RG" --name kafka-nsg

az network nsg rule create \
  --resource-group "$RG" --nsg-name kafka-nsg \
  --name allow-ssh-mgmt --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" \
  --destination-port-ranges 22 --access Allow --protocol Tcp

az network nsg rule create \
  --resource-group "$RG" --nsg-name kafka-nsg \
  --name allow-kafka-vnet --priority 110 \
  --source-address-prefixes 10.20.0.0/16 \
  --destination-port-ranges 9093 --access Allow --protocol Tcp

az vm create \
  --resource-group "$RG" \
  --name "$VM_NAME" \
  --image "$GALLERY_IMAGE_ID" \
  --size Standard_D4s_v3 \
  --admin-username "$ADMIN_USER" \
  --ssh-key-values "$SSH_KEY" \
  --vnet-name kafka-vnet --subnet kafka-subnet \
  --nsg kafka-nsg \
  --public-ip-address "" \
  --os-disk-size-gb 64

The --public-ip-address "" flag keeps the broker off the public internet. Use a bastion host or your existing private connectivity to reach it.

Step 3: Connect via SSH

After deployment, find the private IP of the new virtual machine. From a host inside the same virtual network:

ssh kafkaops@<private-ip>

The first login may take a few seconds while cloud init finalises. Once you have a shell, the broker has already been started by systemd and the first boot oneshot has already generated the per VM credentials.

Step 4: Retrieve the SASL Credentials

The admin SASL credentials are written by the kafka-firstboot.service systemd oneshot the very first time the virtual machine boots. They live in a single file:

sudo cat /stage/scripts/kafka-credentials.log

You will see something like:

username=admin
password=4f8aB2dQ9pK1xVnE7sT3uYrL6cZw0jHm
bootstrap=10.20.1.4:9093
mechanism=SCRAM-SHA-512
security.protocol=SASL_PLAINTEXT

These credentials are unique to this virtual machine. Store them somewhere safe immediately, because the next thing you do should be to rotate them per Section 11. The first boot oneshot does not run again on subsequent reboots, so this file is your only readable copy of the generated password until you rotate it.

Step 5: Server Components

The deployed image contains the following components:

Component Version Purpose
Apache Kafka 4.2.0 (Scala 2.13) Single node KRaft broker, controller and broker in one JVM
OpenJDK 17 (headless) JVM runtime
Ubuntu 22.04 LTS Base operating system
systemd units kafka.service, kafka-firstboot.service Process supervision

The broker process is kafka.Kafka running under the dedicated kafka system user, with no login shell, started by /opt/kafka/bin/kafka-server-start.sh /etc/kafka/server.properties.

Step 6: Filesystem Layout

Path Owner Purpose
/opt/kafka-4.2.0/ root:root Unpacked Apache Kafka tarball
/opt/kafka symlink Stable upgrade path that points at the active version
/etc/kafka/server.properties kafka:kafka 0640 Broker and KRaft controller configuration
/etc/kafka/kafka_server_jaas.conf kafka:kafka 0640 JAAS file consumed by the broker JVM, populated on first boot
/etc/default/kafka root:kafka 0640 Environment variables (heap size, JVM flags, log directory)
/etc/systemd/system/kafka.service root:root 0644 Main broker unit
/etc/systemd/system/kafka-firstboot.service root:root 0644 First boot oneshot unit
/usr/local/sbin/kafka-firstboot.sh root:root 0750 First boot logic (credential generation, KRaft format)
/var/lib/kafka/data/ kafka:kafka 0750 Kafka log directories
/var/log/kafka/ kafka:kafka 0750 Server and GC logs
/stage/scripts/kafka-credentials.log kafka:kafka 0600 Generated admin credentials, readable by root only

Step 7: Start, Stop, and Check Status

The broker is started by systemd at boot. Manage it as follows:

# Status
sudo systemctl status kafka.service

# Stop
sudo systemctl stop kafka.service

# Start
sudo systemctl start kafka.service

# Restart
sudo systemctl restart kafka.service

# Tail live logs
sudo journalctl -u kafka.service -f

The on disk log file /var/log/kafka/kafka.log is appended to by the systemd unit and contains the same content as journalctl. To check the first boot oneshot:

sudo systemctl status kafka-firstboot.service
sudo journalctl -u kafka-firstboot.service

The oneshot is expected to remain in the active (exited) state after a successful first boot. It is gated by /var/lib/kafka/.firstboot-done and will not run again.

Step 8: Connect a Kafka Client

Build a client properties file from your generated credentials. Replace the password with the value from /stage/scripts/kafka-credentials.log:

PRIVATE_IP="$(hostname -I | awk '{print $1}')"
ADMIN_PW="$(sudo awk -F= '/^password=/ {print $2}' /stage/scripts/kafka-credentials.log)"

cat > /tmp/client.properties <<EOF
security.protocol=SASL_PLAINTEXT
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="${ADMIN_PW}";
EOF
chmod 600 /tmp/client.properties

List topics:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --list

The same client.properties file works as --producer.config and --consumer.config for the console producer and consumer, and is the equivalent of the consumer.properties and producer.properties overrides you would set in a Java or Python client.

A minimal Java producer config block:

Properties props = new Properties();
props.put("bootstrap.servers", "10.20.1.4:9093");
props.put("security.protocol", "SASL_PLAINTEXT");
props.put("sasl.mechanism", "SCRAM-SHA-512");
props.put("sasl.jaas.config",
  "org.apache.kafka.common.security.scram.ScramLoginModule required " +
  "username=\"admin\" password=\"<your-password>\";");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");

The equivalent block for the confluent-kafka Python client:

from confluent_kafka import Producer
producer = Producer({
    "bootstrap.servers": "10.20.1.4:9093",
    "security.protocol": "SASL_PLAINTEXT",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "admin",
    "sasl.password": "<your-password>",
})

Step 9: Create Your First Topic and Round Trip a Message

# Create a topic with 6 partitions
/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --create --topic orders --partitions 6 --replication-factor 1

# Produce a message
echo "hello cloudimg" | /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --producer.config /tmp/client.properties \
  --topic orders

# Consume it back
/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --consumer.config /tmp/client.properties \
  --topic orders --from-beginning --max-messages 1 --timeout-ms 10000

You should see hello cloudimg echoed back. Replication factor is fixed at 1 because this is a single broker image.

Step 10: Rotate the Admin Password

The admin password generated on first boot should be rotated before you put any real workload through the broker. Generate a new password, alter the SCRAM credential, update the JAAS file, update the credentials file, and restart:

NEW_PW="$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)"

/opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --alter \
  --add-config "SCRAM-SHA-512=[password=${NEW_PW}]" \
  --entity-type users --entity-name admin

# Update the broker JAAS file with the new password
sudo tee /etc/kafka/kafka_server_jaas.conf >/dev/null <<EOF
KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required
    username="admin"
    password="${NEW_PW}";
};
EOF
sudo chown kafka:kafka /etc/kafka/kafka_server_jaas.conf
sudo chmod 0640 /etc/kafka/kafka_server_jaas.conf

# Update the operator credentials file
sudo sed -i "s|^password=.*|password=${NEW_PW}|" /stage/scripts/kafka-credentials.log

sudo systemctl restart kafka.service

After the restart, regenerate /tmp/client.properties with the new password and confirm kafka-topics.sh --list still works.

Step 11: Add Application Users and ACLs

The shipped image has only the admin user, with no ACLs configured (Kafka ACLs are off by default in server.properties). To add an application user with a generated password:

APP_PW="$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)"

/opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --alter \
  --add-config "SCRAM-SHA-512=[password=${APP_PW}]" \
  --entity-type users --entity-name app1

echo "app1 password: ${APP_PW}"

To turn ACLs on, add the following two lines to /etc/kafka/server.properties, then restart:

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
super.users=User:admin

Then grant app1 produce and consume rights on the orders topic:

/opt/kafka/bin/kafka-acls.sh \
  --bootstrap-server "${PRIVATE_IP}:9093" \
  --command-config /tmp/client.properties \
  --add --allow-principal User:app1 \
  --producer --consumer --group '*' --topic orders

Step 12: Move Data to an Attached Premium Disk

By default, the broker writes its log directories to /var/lib/kafka/data on the OS disk. For any non trivial workload you should attach a Premium SSD data disk and move the log directory onto it.

# Stop the broker
sudo systemctl stop kafka.service

# (In Azure) attach a new Premium SSD via the portal or `az vm disk attach`,
# then on the VM identify the new device. It will typically be /dev/sdc.
lsblk

# Format and mount
sudo mkfs.xfs /dev/sdc
sudo mkdir -p /data
echo "/dev/sdc /data xfs defaults,nofail 0 2" | sudo tee -a /etc/fstab
sudo mount /data
sudo mkdir -p /data/kafka
sudo chown kafka:kafka /data/kafka
sudo chmod 0750 /data/kafka

# Move existing data (cheap on a freshly deployed broker)
sudo rsync -aHAX /var/lib/kafka/data/ /data/kafka/
sudo mv /var/lib/kafka/data /var/lib/kafka/data.old

# Point Kafka at the new directory
sudo sed -i 's|^log.dirs=.*|log.dirs=/data/kafka|' /etc/kafka/server.properties

sudo systemctl start kafka.service
sudo systemctl status kafka.service

After confirming the broker comes back up cleanly and your topics still list, you can sudo rm -rf /var/lib/kafka/data.old to reclaim the OS disk space.

Step 13: Enable TLS Before Production

The shipped image is SASL_PLAINTEXT only. The wire is authenticated but not encrypted. Do not expose the broker beyond a private network until you have completed this step.

Adding TLS to a Kafka broker is a multi step exercise that depends on whether you want a self signed certificate, a private CA issued certificate, or a public CA issued certificate. The high level steps are:

  1. Generate or obtain a server certificate and private key for the broker, plus a truststore for clients
  2. Add a SASL_SSL listener to /etc/kafka/server.properties on a new port (typically 9094) and update advertised.listeners accordingly
  3. Configure ssl.keystore.location, ssl.keystore.password, ssl.key.password, and (if using a private CA) ssl.truststore.location and ssl.truststore.password on the broker
  4. Update the network security group to allow the new port and remove the SASL_PLAINTEXT port from the allow list
  5. Distribute the truststore (or root CA certificate) to all your clients and update their security.protocol from SASL_PLAINTEXT to SASL_SSL
  6. Restart the broker and validate the new listener with a test client before cutting traffic over

The Apache Kafka security documentation at https://kafka.apache.org/documentation/#security_overview is the authoritative reference. cloudimg does not ship a one click TLS enabler in this image, on purpose: certificate lifecycle should be owned by your security team, not by the marketplace publisher.

Step 14: Troubleshooting

The broker will not start. Check sudo journalctl -u kafka.service -n 200 --no-pager. The most common cause is a syntax error in /etc/kafka/server.properties after a manual edit. Compare against the original which is reproduced verbatim in this guide.

First boot oneshot never ran. Check sudo systemctl status kafka-firstboot.service and ls -la /var/lib/kafka/.firstboot-done. If the sentinel is present and the credentials file is empty or contains the placeholder text, something interrupted the oneshot. Remove the sentinel with sudo rm /var/lib/kafka/.firstboot-done and run sudo systemctl start kafka-firstboot.service to retry.

Client authentication fails with "Authentication failed during authentication due to invalid credentials". The single most common cause is a sasl.mechanism mismatch. The broker uses SCRAM-SHA-512, and your client must use exactly the same string. A client configured with SCRAM-SHA-256 or PLAIN will not authenticate against this broker.

Client connects locally but a remote client gets "Connection refused". Check ss -tlnp | grep 9093 on the broker. The listener should be bound to 0.0.0.0. Also check the network security group rule that controls inbound 9093 traffic. The image ships with listeners=SASL_PLAINTEXT://0.0.0.0:9093, so 0.0.0.0 is correct.

Producer fails with "Bootstrap broker returned wrong host". This means advertised.listeners is wrong. The first boot oneshot rewrites advertised.listeners to the value of hostname -I | awk '{print $1}'. If your VM has multiple network interfaces the wrong one may have been picked. Edit /etc/kafka/server.properties, fix the advertised.listeners line, and sudo systemctl restart kafka.service.

Out of disk on /var/lib/kafka/data. Either move the log directory to a data disk per Section 12, or reduce log.retention.hours in /etc/kafka/server.properties and restart.

Step 15: Security Recommendations

  • Rotate the admin password immediately after first boot (Section 10)
  • Restrict the network security group on port 9093 to the smallest possible source CIDR
  • Add TLS before exposing the broker beyond a private network (Section 13)
  • Use the application user pattern in Section 11 rather than reusing the admin user for application clients
  • Enable ACLs and set super.users=User:admin so application users have the minimum rights they need
  • Take regular snapshots of /var/lib/kafka/data (or your attached data disk) using Azure Disk Snapshots
  • Subscribe to the Apache Kafka security mailing list at https://kafka.apache.org/contact and apply security patches as they are published

Step 16: Support and Licensing

Apache Kafka is licensed under the Apache License 2.0. The full text is reproduced in /opt/kafka/LICENSE on the deployed image. cloudimg distributes the unmodified upstream Apache Kafka 4.2.0 binary tarball published by the Apache Software Foundation, with cloudimg authored configuration, systemd units, and first boot tooling layered on top.

For support with the cloudimg image itself contact support@cloudimg.co.uk or visit https://www.cloudimg.co.uk/support. For Apache Kafka questions outside the scope of cloudimg packaging, the Apache Kafka community resources at https://kafka.apache.org/contact are the authoritative reference.