Kroxylicious on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and configuration of Kroxylicious on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. Kroxylicious is an open source protocol aware proxy for Apache Kafka. It sits between your Kafka clients and your Kafka brokers and understands the Kafka wire protocol itself, so it can inspect, rewrite and police traffic as it passes. Because the policy lives in the proxy rather than in each application, you can apply encryption, multi tenancy, filtering and access policy across every client at once without changing a single line of client code.

A proxy is meaningless without a broker behind it, so this image is self contained. It ships Kroxylicious 0.23.0 alongside a working single node Apache Kafka 4.3.1 broker running in KRaft mode, with no ZooKeeper. You can produce and consume through the proxy the moment the VM finishes booting, and when you are ready to put the proxy in front of your own cluster you change one line of configuration. Both are installed from the official upstream release artifacts, pinned and verified by SHA 512 at install time.

The broker is only reachable through the proxy. The backing Kafka broker binds to loopback only, on ports 9092 and 9093. Nothing outside the VM can open a socket to it. Kroxylicious is the single network facing entry point, on port 9192 for bootstrap and 9193 for the broker node. This is a deliberate topology: a proxy that can be trivially bypassed enforces nothing.

Secure by default, with no shipped credential. Client facing authentication is SASL SCRAM SHA 512 and it is enforced twice over. The backing broker requires SASL, and Kroxylicious additionally runs the SaslInspection filter with requireAuthentication set to true, so the proxy itself refuses to forward any request other than the authentication exchange until authentication has succeeded. The SCRAM credential is generated uniquely on each VM at first boot and written straight into Kafka cluster metadata at format time, so there is no default or bootstrap password in the image and no window in which a published credential is live.

What is included:

• Kroxylicious 0.23.0 from the official GitHub release (SHA 512 verified), run under systemd as kroxylicious.service

• Apache Kafka 4.3.1 (Scala 2.13) in KRaft mode as the backing broker, run under systemd as kafka.service, bound to loopback only

• Client facing SASL SCRAM SHA 512 authentication, enforced at both the proxy and the broker

• A per VM SCRAM credential generated at first boot, written to /stage/scripts/kroxylicious-credentials.log with mode 0600

• A pre created demo topic called cloudimg, created through the proxy on first boot

• A Prometheus metrics endpoint for the proxy on loopback 127.0.0.1:9190

• An on image self test, /usr/local/sbin/kroxylicious-selftest.sh, that proves end to end traffic really traverses the proxy

• OpenJDK 21, required by Kroxylicious 0.23.0 and supported by Kafka 4.3

Prerequisites

• Active Azure subscription, SSH public key, VNet and subnet in your target region

• A subscription to the Kroxylicious listing on Azure Marketplace

• Network Security Group rules allowing TCP 22 from your admin network. Kafka clients inside the VNet need TCP 9192 and 9193 to the VM

Recommended virtual machine size: Standard_B2s (2 vCPU, 4 GB RAM). This image runs two JVMs, so both heaps are sized explicitly for that shape: Kafka is capped at 896 MB and Kroxylicious at 640 MB, with the proxy's direct memory capped at 192 MB. A freshly booted VM has roughly 2.9 GB still available. Size up if you intend to carry heavy sustained throughput or point the proxy at a large external cluster.

Step 1: Deploy from the Azure Portal

1. Open the Kroxylicious offer on the Azure Marketplace and choose Get It Now, then Create.
2. Select your subscription, resource group and region, and choose the Standard_B2s size.
3. Provide your SSH public key for the azureuser account.
4. In the Network Security Group, allow inbound TCP 22 from your admin network, and TCP 9192 and 9193 from the subnet your Kafka clients run in.
5. Create the VM.

Step 2: Deploy from the Azure CLI

az vm create \
  --resource-group my-rg \
  --name kroxylicious-01 \
  --image <kroxylicious-marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

Step 3: First boot

On the first boot only, a one shot unit called kroxylicious-firstboot.service runs before either product service starts. It generates this VM's unique SCRAM credential, writes it directly into Kafka cluster metadata as the storage is formatted, resolves the address the proxy will advertise to clients, and writes the credentials file. Both kafka.service and kroxylicious.service are gated on the marker that this unit creates when it finishes, so neither can start against half prepared state.

First boot completes in a few seconds. Connect with SSH:

ssh azureuser@<vm-ip>

Step 4: Confirm both services are running

sudo systemctl is-active kroxylicious.service kafka.service

Both report active. Check the listening topology at the same time, because it is the thing that makes this deployment safe:

sudo ss -ltn | grep -E ':(9092|9093|9190|9192|9193)'

The proxy listens on *:9192 and *:9193. The broker appears only on 127.0.0.1:9092 and 127.0.0.1:9093, and the proxy's metrics endpoint only on 127.0.0.1:9190. Anything reaching Kafka from off the box must therefore pass through Kroxylicious.

Step 5: Read this VM's credentials

The per VM SCRAM credential is written at first boot, readable only by root:

sudo cat /stage/scripts/kroxylicious-credentials.log

The file records the bootstrap address clients should use, the SASL user cloudimg, this VM's generated password, and the security protocol and mechanism. A ready made client configuration containing the same credential is also written to /etc/kroxylicious/client.properties, again root owned and mode 0600, and the examples below use it directly.

Step 6: Produce and consume through the proxy

The image pre creates a topic called cloudimg. Produce a message to it through the proxy on port 9192:

echo "hello from cloudimg" | sudo /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server 127.0.0.1:9192 \
  --command-config /etc/kroxylicious/client.properties \
  --topic cloudimg

Read it back, again through the proxy:

sudo timeout 40 /opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 127.0.0.1:9192 \
  --command-config /etc/kroxylicious/client.properties \
  --topic cloudimg --from-beginning --max-messages 1

--max-messages 1 makes the consumer exit cleanly as soon as it has read one message. Without it the consumer waits for a timeout and then reports a TimeoutException, which is normal behaviour but looks like a fault.

Note that on Kafka 4 the console tools take --command-config. The older --producer.config and --consumer.config flags are deprecated and print warnings that corrupt captured output.

You can confirm the client is genuinely being steered to the proxy rather than to the broker. Ask the cluster which nodes it advertises:

sudo /opt/kafka/bin/kafka-broker-api-versions.sh \
  --bootstrap-server 127.0.0.1:9192 \
  --command-config /etc/kroxylicious/client.properties | awk 'NR==1'

The node reported is the proxy's node port, 9193, on this VM's advertised address. It is never the broker's 9092. That is Kroxylicious rewriting the cluster metadata as it passes, which is what keeps clients on the proxy for the whole session rather than only for the initial bootstrap.

Step 7: Verify the authentication wall

Client facing authentication is on by default. A client presenting the wrong password is rejected during the SASL exchange:

sudo grep -E 'type: SaslInspection|enabledMechanisms|requireAuthentication' /etc/kroxylicious/config.yaml

requireAuthentication is set to true, so Kroxylicious will not forward any request beyond the authentication exchange until a client has successfully authenticated. The backing broker independently enforces SASL SCRAM SHA 512 as well, so removing the proxy filter would not open the cluster.

Step 8: Run the built in self test

The image ships a self test that proves the deployment end to end rather than merely checking that a process is running:

sudo /usr/local/sbin/kroxylicious-selftest.sh

It produces a uniquely tagged message through the proxy and consumes it back, asserting the payload matches byte for byte. It then reads the proxy's own upstream request counter and asserts it moved, which is what distinguishes traffic that traversed the proxy from traffic that reached the broker directly. It confirms the client is steered to the proxy node port, checks that a wrong credential is explicitly rejected, then stops the proxy and confirms the round trip breaks, and finally restarts the proxy and re proves the round trip so that a passing negative result cannot be an artefact of a broken system. It exits non zero if any of those fail.

Step 9: Prometheus metrics

Kroxylicious publishes Prometheus metrics on loopback port 9190:

curl -s http://127.0.0.1:9190/metrics | awk '/^kroxylicious_(client_to_proxy|proxy_to_server)/ && ++n <= 8'

The client_to_proxy families count what clients sent to the proxy, and the proxy_to_server families count what the proxy forwarded upstream to the broker. Comparing the two is the most direct way to confirm traffic is flowing through the proxy in production. To scrape these from elsewhere, bind the management endpoint to a routable address by editing bindAddress under management in /etc/kroxylicious/config.yaml, and restrict access with a Network Security Group rule.

Step 10: Point the proxy at your own Kafka cluster

The bundled broker exists so the image is useful immediately. In production you will usually put Kroxylicious in front of a cluster you already run, which is a single configuration change.

Edit /etc/kroxylicious/config.yaml and set bootstrapServers under targetCluster to your own cluster, then restart the proxy. If your cluster has more than one broker, widen nodeIdRanges so every broker node id maps to a proxy port, and open those ports in your Network Security Group. If your cluster requires TLS or a different SASL mechanism upstream, add the matching tls and authentication stanzas to targetCluster.

Once the proxy is serving your own cluster you can stop and disable the bundled broker so it is not consuming memory, with sudo systemctl disable --now kafka.service. Because the bundled broker only ever listened on loopback, leaving it running is not an exposure, but disabling it returns roughly 370 MB of RAM to the proxy.

Step 11: Exposing the proxy beyond the VNet

By default the proxy advertises this VM's private address to clients. That is deliberate. A Kafka proxy terminating client connections should not default to being reachable from the internet, and the default keeps the deployment safe until you make an explicit decision.

To serve clients from outside the VNet, set the advertised address to a name or address those clients can actually reach, then restart the proxy:

sudo sed -i 's|advertisedBrokerAddressPattern: .*|advertisedBrokerAddressPattern: <your-domain>|' \
  /etc/kroxylicious/config.yaml
sudo systemctl restart kroxylicious.service

Replace <your-domain> with the hostname your clients will use, for example kafka.internal.example.com.

A DNS name is strongly preferred over a bare public IP, because it survives the address changing and it is what you will need anyway when you add TLS. Whichever you choose, the value must be the address the client will use, since it is handed to clients in the cluster metadata. You must also open TCP 9192 and 9193 in the Network Security Group, and you should restrict the source range to the networks your clients run in rather than leaving it open to the internet.

Step 12: Record encryption and the other filters

Kroxylicious ships a set of filters beyond the SASL inspection filter enabled here, including record encryption, multi tenancy, record validation, entity isolation and authorization. They are present in the distribution and can be enabled by adding them to filterDefinitions and defaultFilters in /etc/kroxylicious/config.yaml.

Record encryption is deliberately not enabled in this image. The filter encrypts record payloads transparently, but it requires a key management service to hold the key material, and every production backend it supports (Azure Key Vault, AWS KMS, HashiCorp Vault, Fortanix DSM and Thales CipherTrust) is an external service. The only backend that runs standalone on a single VM is intended for testing and holds key material in the proxy's memory. Shipping that would mean advertising encryption backed by a test double, so the image ships the filter unconfigured instead. The practical consequence is that there is no encryption key material in this image, and therefore nothing sensitive baked into it.

On Azure the natural choice is the Azure Key Vault provider. Create a vault and a key, grant this VM's managed identity permission to use it, then add the record encryption filter to filterDefinitions referencing that vault. Consult the Kroxylicious documentation for the current configuration schema for your version.

Server Components

Network Ports

Scripts and Log Files

Managing the services

sudo systemctl status kroxylicious.service
sudo systemctl restart kroxylicious.service
sudo journalctl -u kroxylicious.service -n 50 --no-pager

The same commands work for kafka.service. Restarting the proxy does not disturb the broker, and the two are ordered but not tightly coupled, so the proxy can be restarted independently while you iterate on filter configuration.

Troubleshooting

Clients connect but then time out

Almost always the advertised address. Clients bootstrap against port 9192, then reconnect to whatever address the proxy advertises. If that address is not reachable from the client, the bootstrap succeeds and everything afterwards hangs. Check what is advertised with sudo grep advertisedBrokerAddressPattern /etc/kroxylicious/config.yaml and confirm the client can reach that address on port 9193. See Step 11.

Authentication failures

Confirm the client is using SASL SCRAM SHA 512 over SASL_PLAINTEXT, and that the password matches the one in /stage/scripts/kroxylicious-credentials.log. The password is unique to this VM and is not recoverable if the credentials file is deleted, though you can create additional SCRAM users with kafka-configs.sh against the broker on loopback.

A service will not start

Both product services are gated on the first boot marker at /var/lib/cloudimg/kroxylicious-firstboot.done. If that file is missing the services will remain inactive by design rather than start against unprepared state. Check sudo journalctl -u kroxylicious-firstboot.service for why first boot did not finish.

Memory pressure

This image runs two JVMs on a 4 GB VM. If you have raised throughput or pointed the proxy at a large cluster, check free -m and consider either raising the VM size or disabling the bundled broker as described in Step 10. Heaps are set in /etc/default/kroxylicious and /etc/default/kafka.

Security Recommendations

• Restrict TCP 9192 and 9193 in the Network Security Group to the subnets your Kafka clients run in. Do not expose them to the internet.

• Leave the backing broker on loopback. Binding it to a routable address would let clients bypass the proxy and everything the proxy enforces.

• Add TLS on the client facing gateway before carrying production data across any untrusted network. Kroxylicious terminates TLS on the gateway and can use a separate TLS configuration upstream to the broker.

• Create per application SCRAM users rather than sharing the built in cloudimg credential, and use the authorization filter to constrain what each identity may do.

• Keep the VM patched. Unattended upgrades are enabled in the image.

Support and Licensing

Kroxylicious is open source software licensed under the Apache License 2.0, and Apache Kafka is likewise licensed under the Apache License 2.0. cloudimg packages, hardens and supports the image. Your use of the software remains subject to the upstream licences.

For assistance, contact cloudimg support:

• Email: support@cloudimg.co.uk
• Response Time: 24/7 with guaranteed 24 hour response SLA
• Website: https://cloudimg.co.uk