R0
Streaming & Messaging AWS

rumqttd 0.20.0 MQTT Broker on AWS User Guide

| Product: rumqttd 0.20.0 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the rumqttd 0.20.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe.

rumqttd is the open source (Apache-2.0) MQTT broker of the rumqtt project, written in Rust. It serves MQTT 3.1.1, the publish/subscribe protocol used by IoT devices, sensors, vehicles and mobile apps. Clients publish messages to topics (such as sensors/17/temperature) and the broker delivers them to every client subscribed to a matching topic filter (such as sensors/+/temperature or sensors/#). Any MQTT 3.1.1 client works with it: the Mosquitto command-line tools mosquitto_pub and mosquitto_sub, Eclipse Paho (Python, Java, C, Go, JavaScript), MQTT.js and others.

What the cloudimg image adds:

  • MQTT over TLS only, terminated by HAProxy. Clients reach the broker on 8883 (MQTT over TLS 1.3 or 1.2 with AEAD cipher suites only). HAProxy 2.8 (Ubuntu's own package) terminates TLS there and passes each connection to rumqttd, which listens in clear text on the loopback address 127.0.0.1:1883 only. Nothing else answers on the network: rumqttd's clear-text port 1883, its console (3030), its metrics exporter (9042) and WebSockets are not reachable from another machine, even if a security group allows them (Step 11).
  • A handshake timeout and connection caps. rumqttd 0.20.0 performs the TLS handshake of a new client before it accepts the next one, with no time limit, so a single client that connects and never speaks TLS would hold up every other client. HAProxy closes a client that has not finished its TLS handshake within 10 seconds, allows at most 64 connections from one source address and 300 new connections per 10 seconds from one address, and 4,096 connections in all. You can tune every value (Step 9).
  • A certificate made on your instance. At first boot HAProxy gets a TLS key and an end-entity certificate for this instance's own names and addresses. It can only vouch for this broker and can never sign other certificates. You pin it in your clients (Step 5), or replace it with one from your own CA or Let's Encrypt (Step 13).
  • No anonymous access and no default password. Every client logs in. First boot generates a login unique to your instance, cloudimg-app, with a 128-bit random password. rumqttd refuses anonymous clients, wrong passwords and unknown users.
  • Users you can manage safely. rumqttd-user adds a login with its password read from standard input (never a command line) (Step 8).
  • A broker that refuses to run unsafely. Before every start a configuration gate refuses an unsafe change to rumqttd or HAProxy (a listener on a public address, a second listener, no logins, a console or metrics port, a HAProxy stats page or socket, TLS 1.0 or 1.1, a longer handshake timeout, a key readable by others). After every start a guard checks the running services' sockets, logins and network rules, and stops them if anything is wrong. If first boot cannot create the login or the certificate, HAProxy never listens (Step 10).
  • No telemetry, no instance metadata, no outbound network. rumqttd and this HAProxy configuration have no telemetry. Both run with the EC2 instance metadata address blocked; rumqttd's processes may exchange packets with the loopback address only, and a firewall table lets the rumqttd and haproxy users answer their clients and nothing more: HAProxy may open one kind of connection, to rumqttd on 127.0.0.1:1883 (Step 12).
  • A pinned supply chain. The rumqttd binary comes from the rumqtt project's GitHub release rumqttd-0.20.0 (rumqttd-0.20.0-linux-musl), checked against its SHA-256 digest, with every shipped file recorded in /usr/share/doc/cloudimg-rumqttd/PROVENANCE. HAProxy comes from Ubuntu's main archive. Ubuntu 24.04 LTS is fully patched at build time.

Two things to know before you start. rumqttd has no topic access rules: every login may publish and subscribe to every topic, including #. Give each application its own login so you can revoke it, but use a separate broker for parties that must not see each other's messages. And because HAProxy terminates TLS without the PROXY protocol, rumqttd sees every client as 127.0.0.1: client addresses appear in HAProxy's log, and the per-source caps live in HAProxy, not in rumqttd.

What is on the image: rumqttd 0.20.0 (/usr/local/bin/rumqttd; configuration and logins in /etc/rumqttd/rumqttd.toml), HAProxy 2.8 (/etc/haproxy/haproxy.cfg, rendered from /etc/haproxy/rumqttd-limits.conf; certificate and key in /etc/haproxy/cloudimg/), and the cloudimg helpers rumqttd-probe (an MQTT test client), rumqttd-selfcheck, rumqttd-user, rumqttd-limits, rumqttd-set-hostname, rumqttd-install-cert and rumqttd-config-gate.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows:
Port Protocol Source Purpose
22 TCP your admin network SSH
8883 TCP the networks your MQTT clients connect from MQTT over TLS

Nothing else needs to be open. Ports 1883 (clear-text MQTT), 3030 (console), 9042 (metrics) and 8404 (HAProxy statistics) are not served to the network, so opening them achieves nothing.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick rumqttd 0.20.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger for many thousands of connections or high message rates), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=rumqttd-broker}]'

The image requires IMDSv2 (session tokens). First boot takes under a minute after the instance is running: the boot service reads the instance's EC2 names, creates the login and the certificate, then rumqttd and HAProxy start and the guard checks them.

Step 2: Connect to your instance

Version SSH user
rumqttd 0.20.0 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

The login banner shows where the login and the certificate are:

  rumqttd 0.20.0 on Ubuntu 24.04 (cloudimg)
  Login, endpoint, certificate fingerprint:  sudo cat /root/rumqttd-credentials.txt
  Clients: MQTT over TLS only, mqtts://<this host>:8883 (pin /etc/haproxy/cloudimg/rumqttd.crt; TLS by HAProxy)
  Broker:  systemctl status rumqttd haproxy   Verify: sudo rumqttd-selfcheck   Users: sudo rumqttd-user list
  User guide: https://www.cloudimg.co.uk/guides/rumqttd-0-20-0-on-ubuntu-24-04-aws/

Step 3: Check the services and the listening set

systemctl is-active rumqttd-metadata-names rumqttd-firstboot rumqttd-egress-guard rumqttd haproxy
sudo ss -tlnp | grep -E ':(22|80|443|1883|3030|8080|8404|8883|9042) '
sudo ss -ulnp | grep -E 'rumqttd|haproxy' || echo 'no UDP socket'

active
active
active
active
active
LISTEN 0      1024       127.0.0.1:1883      0.0.0.0:*    users:(("rumqttd",pid=13609,fd=6))
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=891,fd=3),("systemd",pid=1,fd=258))
LISTEN 0      4096         0.0.0.0:8883      0.0.0.0:*    users:(("haproxy",pid=13921,fd=5))
LISTEN 0      4096            [::]:22           [::]:*    users:(("sshd",pid=891,fd=4),("systemd",pid=1,fd=259))
no UDP socket

rumqttd-metadata-names reads this instance's EC2 names for the certificate, rumqttd-firstboot prepares the login and the certificate, rumqttd-egress-guard loads the firewall table, rumqttd is the broker and haproxy is its TLS front. HAProxy listens on 8883 (every IPv4 address) for clients; rumqttd listens on 127.0.0.1:1883 only, where nothing outside the instance can reach it.

Step 4: Your endpoint and login

Everything generated for your instance is in /root/rumqttd-credentials.txt (readable by root only). View it all with sudo cat /root/rumqttd-credentials.txt. The lines that are not secret are:

sudo grep -E '^RUMQTTD_(HOST|URL|PORT|PRIVATE_IP|USER|TLS_CA_FILE|TLS_CERT_SHA256|TLS_PUBKEY_SHA256)=' /root/rumqttd-credentials.txt

RUMQTTD_HOST=3.89.221.187
RUMQTTD_URL=mqtts://3.89.221.187:8883
RUMQTTD_PORT=8883
RUMQTTD_PRIVATE_IP=172.31.88.68
RUMQTTD_USER=cloudimg-app
RUMQTTD_TLS_CA_FILE=/etc/haproxy/cloudimg/rumqttd.crt
RUMQTTD_TLS_CERT_SHA256=BB:3E:77:5A:1A:24:AA:8C:7F:F4:C9:20:F6:7B:06:61:B1:C9:56:66:F2:C7:F7:C6:AF:D2:88:50:8A:F0:AC:E9
RUMQTTD_TLS_PUBKEY_SHA256=ae72fdda1faf340dc4ff6e00a919bf42013b0aaa5206c013aa726d738d4b47cb

The file also holds RUMQTTD_PASSWORD, the password of cloudimg-app. Give each device type or application a login of its own (Step 8) rather than sharing this one. Keep passwords in files with mode 0600, not in shell history or on command lines (a command line is visible to every user on a machine through ps).

Step 5: Check the certificate and pin it in your client

The broker's certificate is self-signed, made on your instance:

sudo openssl x509 -in /etc/haproxy/cloudimg/rumqttd.crt -noout -subject -enddate -ext subjectAltName,basicConstraints
sudo openssl x509 -in /etc/haproxy/cloudimg/rumqttd.crt -noout -fingerprint -sha256

subject=CN = rumqttd-broker, O = cloudimg
notAfter=Feb 10 13:04:43 2029 GMT
X509v3 Subject Alternative Name:
    IP Address:3.89.221.187, DNS:ec2-3-89-221-187.compute-1.amazonaws.com, IP Address:172.31.88.68, DNS:ip-172-31-88-68.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
sha256 Fingerprint=BB:3E:77:5A:1A:24:AA:8C:7F:F4:C9:20:F6:7B:06:61:B1:C9:56:66:F2:C7:F7:C6:AF:D2:88:50:8A:F0:AC:E9

CA:FALSE means it is an end-entity certificate that cannot sign anything else. The SHA-256 fingerprint is the RUMQTTD_TLS_CERT_SHA256 value in your credentials file. Copy the certificate (it is public; the private key never leaves the instance) to each client machine:

client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/haproxy/cloudimg/rumqttd.crt' > rumqttd.crt
client-host$ openssl x509 -in rumqttd.crt -noout -fingerprint -sha256

Compare the fingerprint with RUMQTTD_TLS_CERT_SHA256 before you trust the file. Then give your client only this file as its CA file (--cafile for mosquitto_pub and mosquitto_sub, tls_set(ca_certs=...) in Paho), with host name checking on. Do not add it to the operating system's trust store: the client should trust this one broker, not every program on the machine. Your client will then refuse any other server, including one in the middle that presents a different certificate. A client that uses only its system trust store is refused: Paho raises CERTIFICATE_VERIFY_FAILED and mosquitto_pub prints Error: Protocol error.

The certificate covers the instance's public and private IPv4 addresses and EC2 DNS names, localhost and 127.0.0.1. An EC2 stop and start gives the instance a new public IP. At the next boot the certificate is re-issued for the new names with the same key: clients that pin the public key (RUMQTTD_TLS_PUBKEY_SHA256) keep working, while clients that pin the certificate file need the new copy.

Step 6: Test on the instance

rumqttd-probe is a small MQTT client. It reads passwords from a file (or from the credentials file with --identity app) and trusts only the broker's certificate:

sudo rumqttd-probe connect --identity app
sudo rumqttd-probe roundtrip --identity app --topic guide/test
sudo rumqttd-probe connect --anonymous || true
sudo rumqttd-probe plain || true
sudo rumqttd-probe connect --anonymous --no-tls --port 1883 || true

CONNACK-OK proto=4 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256
ROUNDTRIP OK topic=guide/test qos=1 bytes=43 proto=4 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256
CLOSED-NO-CONNACK
NOT-MQTT-PLAIN no-bytes
CLOSED-NO-CONNACK

The login connects over TLS 1.3, subscribes to guide/test, publishes a message there and receives it back. A client without a login, with a wrong password or with an unknown user name is closed without a CONNACK: rumqttd does not say which part was wrong. Clear-text MQTT sent to 8883 gets a TLS alert, never an MQTT answer. On the loopback address rumqttd itself refuses the same anonymous client. rumqttd-probe exits non-zero on a refusal, hence || true.

The broker speaks MQTT 3.1.1. rumqttd 0.20.0's MQTT 5 support is experimental and is not enabled on this image, so configure your client for protocol version 3.1.1 (-V mqttv311 for the Mosquitto tools, protocol=mqtt.MQTTv311 in Paho).

Step 7: Connect a client from another machine

On the client machine, install the Mosquitto command-line clients (sudo apt-get install mosquitto-clients on Ubuntu or Debian, brew install mosquitto on macOS) and copy the certificate as in Step 5. Then put the connection settings in an options file that only you can read. mosquitto_pub and mosquitto_sub read their default options, one per line, from mosquitto_pub and mosquitto_sub files in the directory named by XDG_CONFIG_HOME (normally ~/.config), so the password is never on a command line. Keeping them in a directory of their own lets you have one per broker and login:

client-host$ umask 077; mkdir -p ~/mqtt-app
client-host$ printf -- '-h <public-ip>\n-p 8883\n-V mqttv311\n--cafile %s/rumqttd.crt\n-u cloudimg-app\n' "$PWD" > ~/mqtt-app/mosquitto_pub
client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> "sudo sed -n 's/^RUMQTTD_PASSWORD=/-P /p' /root/rumqttd-credentials.txt" >> ~/mqtt-app/mosquitto_pub
client-host$ cp ~/mqtt-app/mosquitto_pub ~/mqtt-app/mosquitto_sub

Subscribe in one terminal and publish from another:

client-host$ XDG_CONFIG_HOME=~/mqtt-app mosquitto_sub -t 'demo/#' -q 1 -v
client-host$ XDG_CONFIG_HOME=~/mqtt-app mosquitto_pub -t demo/hello -q 1 -m 'hello from mosquitto_pub'

demo/hello hello from mosquitto_pub

Use a name the certificate covers as -h: the public or private IP, an EC2 DNS name, or your own DNS name once you have added it (Step 13). A wrong or missing password, or an unknown user, is refused: rumqttd closes the connection without a CONNACK, and mosquitto_pub and mosquitto_sub print Error: The connection was lost. A client that does not use TLS, or trusts only the system certificate store, gets Error: Protocol error.

The same from Python with Eclipse Paho (sudo apt-get install python3-paho-mqtt, or pip install paho-mqtt):

# paho_demo.py - publish and receive through your cloudimg rumqttd broker over MQTT/TLS (paho-mqtt 1.6 or 2.x).
# Usage: python3 paho_demo.py <broker-host> <rumqttd.crt> <user> <password-file> <topic>
# The password is read from a file, never the command line; TLS trusts ONLY the broker's own certificate.
import ssl
import sys
import threading

import paho.mqtt.client as mqtt

host, cafile, user, pwfile, topic = sys.argv[1:6]
with open(pwfile) as f:
    password = f.read().strip()

got = threading.Event()
try:
    c = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, protocol=mqtt.MQTTv311)
except AttributeError:                       # paho-mqtt 1.x
    c = mqtt.Client(protocol=mqtt.MQTTv311)
c.username_pw_set(user, password)
c.tls_set(ca_certs=cafile, tls_version=ssl.PROTOCOL_TLS_CLIENT)   # verify against this one certificate, host name checked
c.on_connect = lambda cl, *a: cl.subscribe(topic, qos=1)
c.on_subscribe = lambda cl, *a: cl.publish(topic, b"hello from paho", qos=1)
c.on_message = lambda cl, u, m: (print("received:", m.topic, m.payload.decode()), got.set())
c.connect(host, 8883)
c.loop_start()
ok = got.wait(15)
c.loop_stop()
c.disconnect()
sys.exit(0 if ok else 1)

client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> "sudo sed -n 's/^RUMQTTD_PASSWORD=//p' /root/rumqttd-credentials.txt" > app.password
client-host$ python3 paho_demo.py <public-ip> rumqttd.crt cloudimg-app app.password demo/paho
received: demo/paho hello from paho

The URL form is mqtts://<public-ip>:8883. Some clients accept the login inside the URL (mqtts://cloudimg-app:<password>@...); build it inside your program from the password file rather than writing it in a configuration file or on a command line.

Step 8: Add and remove users

Give each device type or application its own login rather than sharing one. rumqttd-user reads the new password from standard input (or a hidden prompt), adds the login to rumqttd's configuration through the configuration gate and restarts rumqttd and HAProxy, so connected clients reconnect:

umask 077
openssl rand -hex 16 > ~/sensors.password
sudo rumqttd-user add sensors < ~/sensors.password
sudo rumqttd-user list
sudo rumqttd-probe roundtrip --user sensors --password-file ~/sensors.password --topic sensors/17/temperature

added sensors
rumqttd and its TLS front restarted (connected clients reconnect)
cloudimg-app (publish and subscribe to any topic: rumqttd has no topic ACLs)
sensors (publish and subscribe to any topic: rumqttd has no topic ACLs)
ROUNDTRIP OK topic=sensors/17/temperature qos=1 bytes=43 proto=4 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256

User names are 1 to 32 characters (a-z, 0-9, _ and -); root, admin, sys, all, anonymous, rumqttd and haproxy are reserved, and an instance holds at most 64 logins. Passwords are 16 to 128 printable ASCII characters without a quote or a backslash. rumqttd keeps its logins in clear text in /etc/rumqttd/rumqttd.toml; that file is readable only by root and the broker (root:rumqttd, mode 0640), and rumqttd-user rewrites it whole, so never edit the logins by hand.

rumqttd has no topic access rules. sensors can publish and subscribe to every topic, including cloudimg-app's and #, exactly as cloudimg-app can. Treat every login as trusted with all of the broker's messages; run a separate broker for each party that must not see the others' traffic.

Hand ~/sensors.password to your application's secret store, then delete the local copy. sudo rumqttd-user passwd <name> changes a password (from standard input, as for add); the old one is refused from the next login. sudo rumqttd-user remove <name> deletes a login (the last login cannot be removed). To remove the example user again:

sudo rumqttd-user remove sensors
shred -u ~/sensors.password

Step 9: The handshake timeout and connection caps

HAProxy enforces the limits that keep slow, idle or misbehaving clients from shutting real clients out. Show them with rumqttd-limits:

sudo rumqttd-limits show

handshake_timeout_s            10   (2..10, default 10) seconds a client may take from TCP connect to a finished TLS handshake
idle_timeout_m                 15   (1..1440, default 15) minutes an established MQTT session may be silent in both directions
per_source_max_conn            64   (1..10000, default 64) concurrent connections from one source IPv4 address
per_source_conn_rate_10s      300   (1..100000, default 300) new connections from one source address per 10 seconds
max_conn                     4096   (64..10000, default 4096) concurrent connections in all (rumqttd itself takes at most 10010)
  • handshake_timeout_s (10 seconds): a client must finish its TLS handshake within this time of connecting, or HAProxy closes it. HAProxy 2.8 has no separate handshake timer, so this is its timeout client, which covers everything before the connection to rumqttd is made. It is capped at 10 seconds.
  • idle_timeout_m (15 minutes): once connected, a session that is silent in both directions for this long is closed (HAProxy's timeout tunnel). MQTT clients send a keep-alive ping well within this (60 seconds is the usual default); set the keep-alive of a client below this value.
  • per_source_max_conn (64) and per_source_conn_rate_10s (300): the connections one source IPv4 address may hold at once, and open in any 10 seconds. Clients behind one NAT gateway share one source address: raise these if a fleet of devices connects through NAT.
  • max_conn (4,096): the connections HAProxy accepts in all. rumqttd itself takes at most 10,010.

rumqttd sees every client as 127.0.0.1, so these limits are applied by HAProxy, before a client reaches rumqttd. You can watch them work: a client that connects and never starts TLS is closed after the handshake timeout, and while it waits a real login is still answered at once:

sudo rumqttd-probe idle --hold 20
sudo rumqttd-probe stallcheck --identity app --count 20

IDLE-DROPPED seconds=10.0
STALL-CONNACK n=20 seconds=0.0 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256

Change a limit with rumqttd-limits set. It checks the value, renders /etc/haproxy/haproxy.cfg again from /etc/haproxy/rumqttd-limits.conf and restarts HAProxy (connected clients reconnect):

sudo rumqttd-limits set per_source_max_conn 256

Do not edit /etc/haproxy/haproxy.cfg by hand: the configuration gate refuses any text other than the shipped configuration rendered with your limits, and HAProxy then does not start.

Step 10: The broker refuses to run unsafely

rumqttd.service and haproxy.service each run rumqttd-config-gate before every start and rumqttd-guard after it. Check the gates, and the whole broker as a client sees it, at any time:

sudo rumqttd-config-gate --unit rumqttd && sudo rumqttd-config-gate --unit haproxy && echo gate-ok
sudo rumqttd-selfcheck

rumqttd-config-gate: OK (rumqttd)
rumqttd-config-gate: OK (haproxy)
gate-ok
SELFCHECK OK login=cloudimg-app tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256; anonymous, wrong password, unknown user and clear text refused; 1883 refuses anonymous; STALL-CONNACK n=5 seconds=0.0 tls=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256

If you edit /etc/rumqttd/rumqttd.toml and move rumqttd's listener off 127.0.0.1, add a second listener, remove the logins, enable the console or the metrics exporter, lengthen the CONNECT timeout, or loosen the file's permissions, the gate refuses the change and rumqttd does not start. If HAProxy's configuration differs from the shipped one rendered with your limits (a stats page or socket, TLS 1.0 or 1.1, a longer handshake timeout, another listener), or its key is missing, does not match the certificate or is readable by others, HAProxy does not start. journalctl -u rumqttd -u haproxy shows the reason. If the running services ever listen anywhere they should not, accept an anonymous login or run without the firewall table, the guard stops them. HAProxy cannot run without rumqttd, and if first boot cannot create the login or the certificate, neither service starts and nothing listens on 8883 or 1883.

Step 11: Why 1883, the console and the metrics port are closed

rumqttd's example configuration opens clear-text MQTT on 1883 for every address, MQTT 5 on 1884, a console on 3030 and a Prometheus exporter on 9042. On a cloud instance each is a way around TLS or the logins: user names, passwords and every message cross the network in clear text on 1883, and the console reports the broker's state to anyone who asks. This image configures exactly one rumqttd listener, MQTT 3.1.1 on 127.0.0.1:1883, in rumqttd's own configuration rather than relying on a firewall, and HAProxy is the only service on 8883:

  • 1883 (MQTT): bound to 127.0.0.1 only, behind HAProxy. Every maintained MQTT client supports TLS on 8883.
  • 3030 (console) and 9042 (metrics): not configured, and the gate refuses them.
  • 8404 (HAProxy statistics): HAProxy has no stats page, stats socket or master socket.
  • Anonymous access: refused (every connection must carry a login from the configuration).

From another machine, even inside a security group that allows all traffic, a connection to 1883, 3030, 9042, 8404, 80 or 443 is refused, and clear-text MQTT sent to 8883 gets no MQTT answer. On the instance itself:

IP=$(sudo sed -n 's/^RUMQTTD_PRIVATE_IP=//p' /root/rumqttd-credentials.txt)
for p in 1883 3030 9042 8404 8883; do nc -z -w 3 "$IP" "$p" 2>/dev/null && echo "$p open" || echo "$p closed"; done

1883 closed
3030 closed
9042 closed
8404 closed
8883 open

Step 12: The instance metadata block and outbound traffic

The EC2 instance metadata service (169.254.169.254) hands out the instance's IAM role credentials to any process that asks. A network service must not be able to reach it. rumqttd, haproxy and rumqttd-firstboot run with the metadata addresses blocked by systemd (IPAddressDeny), and rumqttd may exchange packets with the loopback address only. Only rumqttd-metadata-names, which reads the instance's names for the certificate, may reach the metadata service, and it may reach nothing else:

for u in rumqttd haproxy rumqttd-firstboot rumqttd-metadata-names; do printf '%-24s deny=%s allow=%s\n' "$u" "$(systemctl show -p IPAddressDeny --value "$u")" "$(systemctl show -p IPAddressAllow --value "$u")"; done

rumqttd                  deny=0.0.0.0/0 ::/0 allow=127.0.0.0/8 ::1/128
haproxy                  deny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 allow=
rumqttd-firstboot        deny=fd00:ec2::254/128 169.254.169.254/32 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 168.63.129.16/32 allow=
rumqttd-metadata-names   deny=0.0.0.0/0 ::/0 allow=fd00:ec2::254/128 169.254.169.254/32

On top of that, the nftables table inet cloudimg_rumqttd_egress denies all outbound traffic, IPv4 and IPv6, for the users rumqttd and haproxy. Only their answers to their clients and HAProxy's connections to rumqttd on 127.0.0.1:1883 are allowed. Neither can reach the metadata service, the internet, your VPC or DNS, whatever its configuration says. Other users are unaffected:

IP=$(sudo sed -n 's/^RUMQTTD_PRIVATE_IP=//p' /root/rumqttd-credentials.txt)
for u in rumqttd haproxy; do for t in http://169.254.169.254/latest/meta-data/ https://1.1.1.1/ "http://$IP:22/"; do sudo -u "$u" curl -s -m 5 -o /dev/null -w "$u -> $t: %{http_code} %{errormsg}\n" "$t" || true; done; done
curl -s -m 5 -o /dev/null -w "ubuntu -> https://1.1.1.1/: %{http_code}\n" https://1.1.1.1/

rumqttd -> http://169.254.169.254/latest/meta-data/: 000 Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to server
rumqttd -> https://1.1.1.1/: 000 Failed to connect to 1.1.1.1 port 443 after 0 ms: Couldn't connect to server
rumqttd -> http://172.31.88.68:22/: 000 Failed to connect to 172.31.88.68 port 22 after 0 ms: Couldn't connect to server
haproxy -> http://169.254.169.254/latest/meta-data/: 000 Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to server
haproxy -> https://1.1.1.1/: 000 Failed to connect to 1.1.1.1 port 443 after 0 ms: Couldn't connect to server
haproxy -> http://172.31.88.68:22/: 000 Failed to connect to 172.31.88.68 port 22 after 0 ms: Couldn't connect to server
ubuntu -> https://1.1.1.1/: 301

rumqttd-egress-guard.service loads the table before rumqttd and HAProxy start, and both require it: if the table is removed, the guard stops them.

Step 13: Use your own DNS name or a certificate from your CA

Point a DNS name at the instance (an Elastic IP keeps the address across stop and start), then add the name to the certificate:

sudo rumqttd-set-hostname <dns-name>

This re-issues the self-signed certificate with the same key to cover the name, updates the endpoint in /root/rumqttd-credentials.txt and restarts HAProxy. Copy the new certificate to clients that pin the certificate file.

To use a certificate from your CA (your own, or a public one such as Let's Encrypt), copy the chain (server certificate first) and its unencrypted PEM key to the instance, then run:

sudo rumqttd-install-cert <certificate-chain.pem> <private-key.pem> <dns-name>

Port 8883 then serves your certificate, and clients trust your CA instead of pinning this instance. rumqttd-selfcheck checks it under the name you gave. The boot-time renewal never replaces an installed certificate. sudo rumqttd-set-hostname <name> goes back to a new self-signed certificate.

Logs, files and limits

  • HAProxy log: journalctl -u haproxy, one line per connection with the client's real address, and one per failed TLS handshake (SSL handshake failure). Broker log: journalctl -u rumqttd (connections and refused logins, such as Error while handling MQTT connect packet, error: InvalidAuth; never a password; every client appears as 127.0.0.1). First boot: journalctl -u rumqttd-firstboot.
  • Configuration: /etc/rumqttd/rumqttd.toml (root:rumqttd 0640, logins in clear text, checked by the gate on every start; change logins with rumqttd-user), /etc/haproxy/rumqttd-limits.conf (change with rumqttd-limits). Certificate and key: /etc/haproxy/cloudimg/ (the key is root:haproxy 0640).
  • A message payload is at most 256 KB, and a client must send its MQTT CONNECT within 10 seconds of finishing the TLS handshake.
  • Sessions, queued messages and retained messages are kept in memory: they do not survive a broker restart or a reboot (and adding or removing a user restarts the broker). Applications that need durable messages should republish their retained state when they reconnect.
  • rumqttd runs as the unprivileged rumqttd user and HAProxy as haproxy, each in a systemd sandbox with no capabilities (a read-only system, a system-call filter, core dumps off).
  • This image runs a single rumqttd node. Clustering is not configured.

Support

cloudimg provides support for the image: launch problems, the first-boot setup, the HAProxy TLS front and the cloudimg helpers. Contact support@cloudimg.co.uk. rumqttd itself is documented at github.com/bytebeamio/rumqtt.