Apache Storm 3.1.0 on AWS User Guide

Overview

This guide covers launching and using the Apache Storm 3.1.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Apache Storm is a distributed real-time computation system: you write a topology, a graph of spouts (sources of streams, such as a Kafka topic) and bolts (processing steps), and Storm runs it continuously, processing every tuple as it arrives, with guaranteed processing, back pressure and per-component parallelism. Storm 3.1.0 is the current release line (it needs Java 25; the 2.x line has reached end of life).

The image runs a complete single-node cluster: Nimbus (the master that accepts topologies), one Supervisor with four worker slots, ZooKeeper 3.9.6 for coordination and the Storm UI.

What the cloudimg image adds:

• Nothing reachable off the instance except SSH and the UI over HTTPS. Nimbus (6627), the Supervisor (6628), every worker's messaging port (6700 to 6703), ZooKeeper (2181) and the Storm UI's own port (8080) listen on 127.0.0.1 only. Storm 3.1.0 has no setting that binds its daemons to one address (they all listen on every interface by default), so the image adds a small, open-source bind shim that turns those listeners into loopback listeners, checks it every time a service starts (a service that would listen on a network address is stopped), and adds a host-firewall table that resets any connection to the Storm and ZooKeeper ports that does not come from the instance itself. Storm's worker transport and Nimbus accept unauthenticated connections by design, so this is what keeps them private.
• The Storm UI over HTTPS only, behind a login. The Storm UI has no authentication of its own. nginx serves it on port 443 (port 80 only redirects) with HTTP Basic authentication for the user admin. The password is generated on your instance at first boot and stored only as a SHA-512 hash; failed logins are rate limited. Requests for host names the instance does not serve are dropped.
• Guarded topology actions. The UI can deactivate, rebalance and kill topologies, change log levels, enable debugging and run profilers. Those requests are refused unless the browser proves they come from the Storm UI page itself (a request with another site's Origin, a cross-site request, a link opened from outside the browser or a request without any browser signal gets a 403), so another web page cannot use your browser's saved login against them. WebSocket upgrades and ; path parameters are refused too.
• No logviewer. Storm's logviewer reads log and dump files on the server for any request; it is not installed, and its routes are refused. Read worker logs over SSH (Step 13).
• No default secret. The admin password and the TLS private key are generated on your instance at first boot; the image ships neither, and nginx will not start until first boot has created them. ZooKeeper and Storm start with empty state on your instance.
• HTTPS from the first boot. The certificate is generated on your instance for its own names and addresses. It is an end-entity certificate: it can only vouch for this instance and can never sign other certificates. Install your own CA-issued certificate with one command (Step 11); the boot-time renewal never replaces it.
• Instance metadata blocked. ZooKeeper, Nimbus and the UI may reach nothing but localhost; the Supervisor (and so every worker it starts, which is where your topology code runs), nginx and the boot service are blocked from the instance metadata service, so topology code or a request to the UI can never read the instance's IAM credentials. Workers can still reach your data sources (Kafka, databases, HTTP APIs). The AMI requires IMDSv2.
• Hardened defaults. Every Storm and ZooKeeper service runs as an unprivileged user under systemd sandboxing, nginx hands Storm the real client address and the configured host name (never values a client sends), SSH accepts keys only, and Ubuntu 24.04 LTS is fully patched at build time with unattended security upgrades switched on.

What is on the image: Apache Storm 3.1.0 and Apache ZooKeeper 3.9.6 (the official Apache binary releases, verified against their published SHA-512 digests and their release signatures), the Eclipse Temurin 25.0.5 Java runtime (verified against its published SHA-256), flux-core 3.1.0 with a WordCount demo topology (from Maven Central, verified against its release signature), nginx 1.24 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH) and TCP 443 (the Storm UI) from your admin network, plus TCP 80 if you want plain HTTP requests redirected to HTTPS. There is nothing else to open: Storm and ZooKeeper listen on the loopback address only and refuse connections from other machines even if a security group allows them.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Apache Storm 3.1.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-storm}]'

First boot takes about a minute after the instance is running: the boot service generates the admin password and the certificate, ZooKeeper, Nimbus, the Supervisor and the UI start, then nginx.

Step 2: Connect to your instance

Version	SSH user
Apache Storm 3.1.0 on Ubuntu 24.04	ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services and the listening set

systemctl is-active storm-metadata-names storm-firstboot storm-firewall zookeeper storm-nimbus storm-supervisor storm-ui nginx
sudo test -f /var/lib/cloudimg/storm-firstboot.done && echo FIRSTBOOT_DONE

All eight units are active and the first boot marker is present:

active
active
active
active
active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network addresses; ZooKeeper (2181), Nimbus (6627), the Supervisor (6628) and the Storm UI's own port (8080) listen on 127.0.0.1:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:2181 users:(("java"))
127.0.0.1:6627 users:(("java"))
127.0.0.1:6628 users:(("java"))
127.0.0.1:8080 users:(("java"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

Step 4: Get the Storm UI URL, admin login and certificate fingerprint

The credentials file is readable by root only. It holds the UI URL, the admin user and password and the SHA-256 fingerprint of this instance's certificate:

sudo cat /root/storm-credentials.txt

# Apache Storm 3.1.0 on Ubuntu 24.04 - generated on first boot by storm-firstboot.service.
# The Storm UI admin password is unique to this instance; none was baked into the image. Keep this file secret.
# nginx stores it only as a SHA-512-crypt hash (/etc/nginx/storm-htpasswd). The Storm UI is served over HTTPS only
# (port 443; port 80 redirects). The certificate is self signed (minted on this instance): before you sign in,
# check that the SHA-256 fingerprint your browser shows matches STORM_TLS_CERT_SHA256 below - or install a
# CA-issued certificate with
#   sudo storm-install-cert <chain.pem> <key.pem>
# Change the served name: sudo storm-set-hostname <dns-name>      Re-verify: sudo storm-selfcheck
STORM_UI_URL=https://<public-ip>/
STORM_UI_HOST=<public-ip>
STORM_ADMIN_USER=admin
STORM_ADMIN_PASSWORD=<STORM_ADMIN_PASSWORD>
STORM_TLS_CERT_SHA256=<sha256-fingerprint>
STORM_TLS_CERT_FILE=/etc/cloudimg/storm-ui.pem
# Nimbus, the Supervisor, its workers and ZooKeeper listen on 127.0.0.1 only: submit topologies on this instance with
# the storm CLI (storm jar ...). REST API with the admin login: sudo curl -K /root/storm-curl.conf ...
# User guide: https://www.cloudimg.co.uk/guides/apache-storm-3-1-0-on-ubuntu-24-04-aws/

The URL, the user and the fingerprint (without the password):

sudo grep -E '^STORM_(UI_URL|ADMIN_USER|TLS_CERT_SHA256)=' /root/storm-credentials.txt

STORM_UI_URL=https://<public-ip>/
STORM_ADMIN_USER=admin
STORM_TLS_CERT_SHA256=<sha256-fingerprint>

The host is the instance's public IP address from the EC2 instance metadata (the private address when the instance has no public one). The certificate also names the public DNS name, the private IP address and the private DNS name. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it (unless you installed your own, Step 11).

The certificate's fingerprint and names, read from the instance:

sudo openssl x509 -in /etc/cloudimg/storm-ui.pem -noout -fingerprint -sha256
openssl x509 -in /etc/cloudimg/storm-ui.pem -noout -ext subjectAltName,basicConstraints,keyUsage,extendedKeyUsage

sha256 Fingerprint=<sha256-fingerprint>
X509v3 Subject Alternative Name:
    IP Address:<public-ip>, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<private-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication

Step 5: Open the Storm UI

Browse to the STORM_UI_URL from Step 4. The certificate is self signed and unique to this instance, so the browser warns about it the first time. Check the certificate before you continue: open the browser's certificate viewer for the page and compare its SHA-256 fingerprint with STORM_TLS_CERT_SHA256 from the credentials file. Continue only if they match, and only for this site. Never add this certificate to an operating-system or browser trust store; for a certificate browsers trust without a warning, install a CA-issued one (Step 11).

The browser then asks for a user name and password: sign in as admin with the password from the credentials file. To print just the password on the instance:

sudo awk -F= '$1=="STORM_ADMIN_PASSWORD" {print $2}' /root/storm-credentials.txt

The Cluster Summary shows Storm 3.1.0, one supervisor with four slots and the Nimbus leader on 127.0.0.1:6627 (screenshot at the top of this guide). Once you have submitted a topology (Step 6) it appears under Topology Summary; its page shows the spouts and bolts with their emitted, transferred and acked counts, the worker it runs in and the topology actions:

Each component has its own page with per-executor statistics and errors:

The Supervisor page lists the slots and the workers running in them:

The UI's links to worker and daemon log files point at Storm's logviewer, which this image does not run (it serves files from the server to anyone who can reach it). Read the logs over SSH instead (Step 13).

Step 6: Submit a topology with the storm CLI

Topologies are submitted on the instance, over SSH: Nimbus listens on 127.0.0.1:6627 and the storm command (/usr/local/bin/storm) talks to it there. The image includes a small demo topology in /opt/storm/examples: a Flux definition of a word spout feeding a counting bolt (both classes ship with Storm), and the flux-core runner jar.

cat /opt/storm/examples/wordcount.yaml

# cloudimg demo topology for Apache Storm 3.1.0 (Flux): a word spout and a counting bolt, both from storm-client.
name: "cloudimg-wordcount"
config:
  topology.workers: 1
spouts:
  - id: "words"
    className: "org.apache.storm.testing.TestWordSpout"
    parallelism: 1
bolts:
  - id: "count"
    className: "org.apache.storm.testing.TestWordCounter"
    parallelism: 2
streams:
  - name: "words --> count"
    from: "words"
    to: "count"
    grouping:
      type: FIELDS
      args: ["word"]

Submit it as the ubuntu user:

storm jar /opt/storm/examples/flux-core-3.1.0.jar org.apache.storm.flux.Flux --remote /opt/storm/examples/wordcount.yaml 2>&1 | grep -E 'Submitting topology|Finished submitting'

05:40:15.357 [main] INFO  o.a.s.StormSubmitter - Submitting topology cloudimg-wordcount in distributed mode with conf {"topology.workers":1,"storm.zookeeper.topology.auth.scheme":"digest","storm.zookeeper.topology.auth.payload":"*****"}
05:40:15.667 [main] INFO  o.a.s.StormSubmitter - Finished submitting topology: cloudimg-wordcount

After a few seconds the topology is active and its worker runs in slot 6700, which also listens on the loopback address only:

sleep 30
storm list 2>&1 | grep -vE '^Running:| (INFO|WARN) '
sudo ss -lntp 'sport = :6700' | awk 'NR>1 {print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g'

Topology_name        Status     Num_tasks  Num_workers  Uptime_secs  Topology_Id          Owner
----------------------------------------------------------------------------------------
cloudimg-wordcount   ACTIVE     4          1            50           cloudimg-wordcount-1-1793252415 storm
127.0.0.1:6700 users:(("java"))

Your own topologies work the same way: build a jar with your spouts and bolts (with storm-client 3.1.0 as a provided dependency and Java 25 bytecode or older), copy it to the instance and submit it with its main class:

scp -i <your-key.pem> target/my-topology-1.0.jar ubuntu@<public-ip>:
storm jar my-topology-1.0.jar com.example.MyTopology my-topology

storm list, storm deactivate <name>, storm activate <name>, storm rebalance <name> -n <workers> and storm kill <name> manage topologies from the command line. Worker processes run as the unprivileged storm user.

Step 7: The Storm UI REST API

The UI's REST API (/api/v1/...) is behind the same login. /root/storm-curl.conf holds the admin login as a curl config, so the password never appears on a command line. On the instance:

H=$(sudo sed -n 's/^STORM_UI_HOST=//p' /root/storm-credentials.txt)
sudo curl -s -K /root/storm-curl.conf --cacert /etc/cloudimg/storm-ui.pem --connect-to "$H:443:127.0.0.1:443" "https://$H/api/v1/cluster/summary" | python3 -m json.tool | grep -E '"(stormVersion|supervisors|slotsTotal|topologies)"'

    "stormVersion": "3.1.0",
    "slotsTotal": 4,
    "topologies": 1,
    "supervisors": 1

Without the login, or with a wrong password, nginx answers 401:

H=$(sudo sed -n 's/^STORM_UI_HOST=//p' /root/storm-credentials.txt)
curl -s -o /dev/null -w '%{http_code}\n' --cacert /etc/cloudimg/storm-ui.pem --connect-to "$H:443:127.0.0.1:443" "https://$H/api/v1/cluster/summary"

401

The topology actions (activate, deactivate, rebalance, kill, debug, log levels and the profiler) are POST requests and are refused unless they carry proof that they come from the Storm UI page itself. A script calling them must send the headers a browser would send for a same-origin request (Sec-Fetch-Site: same-origin and this instance's Origin); the same request marked cross-site is refused with 403 and changes nothing:

H=$(sudo sed -n 's/^STORM_UI_HOST=//p' /root/storm-credentials.txt)
TID=$(storm list 2>/dev/null | awk '$1=="cloudimg-wordcount" {print $6}')
api() { sudo curl -s -K /root/storm-curl.conf --cacert /etc/cloudimg/storm-ui.pem --connect-to "$H:443:127.0.0.1:443" "$@"; }
api -o /dev/null -w 'cross-site deactivate: %{http_code}\n' -X POST -H 'Sec-Fetch-Site: cross-site' "https://$H/api/v1/topology/$TID/deactivate"
api -X POST -H 'Sec-Fetch-Site: same-origin' -H "Origin: https://$H" "https://$H/api/v1/topology/$TID/deactivate"; echo
api -X POST -H 'Sec-Fetch-Site: same-origin' -H "Origin: https://$H" "https://$H/api/v1/topology/$TID/activate"; echo

cross-site deactivate: 403
{"topologyId":"cloudimg-wordcount-1-1793252415","topologyOperation":"deactivate","status":"success"}
{"topologyId":"cloudimg-wordcount-1-1793252415","topologyOperation":"activate","status":"success"}

Step 8: Kill the topology

storm kill cloudimg-wordcount -w 5 2>&1 | grep -E 'Killed topology'
sleep 15
storm list 2>&1 | grep -vE '^Running:| (INFO|WARN) '

05:41:10.834 [main] INFO  o.a.s.c.KillTopology - Killed topology: cloudimg-wordcount
No topologies running.

-w 5 gives the topology five seconds to drain before its workers stop. The worker slot is free again and port 6700 is closed.

Step 9: Security checks

storm-selfcheck re-proves the whole appliance with this instance's credentials. It checks the units and the cluster, the exact listening set (TCP and UDP), the bind shim in every Storm JVM and the firewall table, and ZooKeeper's four-letter words. It checks the login wall (anonymous and wrong-password requests refused, the admin login accepted) and the redirect. It confirms that other host names, foreign origins, WebSocket upgrades, ; path parameters, other HTTP methods, the logviewer routes and encoded path traversal are all refused. It runs the cross-site guard on every topology action against Storm, the metadata block inside every service, and checks that the certificate served on 443 is this instance's end-entity certificate:

sudo storm-selfcheck

Apache Storm 3.1.0 appliance self-check (https://<public-ip>/)
  [ok] units active; Storm 3.1.0 cluster: Nimbus leader 127.0.0.1:6627, 1 supervisor with 4 slots on 127.0.0.1
  [ok] listening set: 22, 80, 443 public; ZooKeeper 2181, Nimbus 6627, Supervisor 6628, UI 8080 (and workers 6700-6703) on 127.0.0.1 only; no logviewer, DRPC or Pacemaker
  [ok] the loopback-bind shim is mapped into all 3 Storm JVMs; nft table inet cloudimg_storm resets off-host connections to the Storm/ZooKeeper ports
  [ok] ZooKeeper 127.0.0.1:2181: ruok -> imok; stat refused (whitelist ruok only)
  [ok] HTTPS UI/API: anonymous 401 (Basic realm "Apache Storm"), wrong password refused, admin 200 (Storm 3.1.0, certificate pinned); /health 200 body 'ok'
  [ok] front door: http -> 301 https://<public-ip>/; unknown Host -> 444; foreign Origin 403 (own 200); WebSocket 403; ';' 400; PUT/DELETE/TRACE/OPTIONS 405; logviewer routes 403; encoded traversal 400
  [ok] cross-site guard: 15 guarded routes (13 POST + GET lag/logconfig): same-origin reaches Storm; cross-site, same-site, Sec-Fetch-Site none, no signal, foreign/look-alike Origin and foreign Referer 403
  [ok] instance metadata refused inside the zookeeper, nimbus, supervisor (and its workers), ui and nginx cgroups (loopback control 200)
  [ok] TLS certificate: end-entity (CA:FALSE, serverAuth, no certificate signing), names <public-ip>; served on 443; SHA-256 <sha256-fingerprint>
SELFCHECK OK - Storm 3.1.0 single-node cluster on loopback (shim + nft layer), HTTPS UI behind a per-instance login, origin and cross-site guards, exact listening set, metadata block verified.

Step 10: How the Storm ports stay on the instance

Storm 3.1.0's UI, its Nimbus and Supervisor thrift servers and every worker's Netty messaging server open their listening sockets on all network interfaces, and Storm has no setting to change that. The worker transport and Nimbus do not authenticate their peers. Left on a network address, anyone who can reach them could submit topologies (run code) or inject tuples. The image handles this in three layers:

1. The bind shim. Every Storm service starts with LD_PRELOAD=/usr/local/lib/cloudimg/libcloudimg-loopback-bind.so. When a Storm process asks to listen on "every interface", the shim makes it listen on 127.0.0.1 instead; a request for a specific address is left alone. The Supervisor passes it on to every worker. The source is on the instance (/usr/share/doc/cloudimg/apache-storm/loopback-bind.c, about 50 lines) with the library's SHA-256.
2. A start-up check. Each Storm and ZooKeeper unit runs storm-assert-loopback after it starts. If any socket of the service listens off the loopback address, or a Storm JVM runs without the shim, the unit fails and stops.
3. A firewall table. storm-firewall.service loads the nft table inet cloudimg_storm, which resets every connection to the Storm and ZooKeeper ports that does not arrive on the loopback interface. Every Storm and ZooKeeper unit requires it: stop it and they stop too.

for u in storm-nimbus storm-supervisor storm-ui; do p=$(systemctl show -p MainPID --value $u); printf '%s: shim mapped %s time(s)\n' $u "$(sudo grep -c libcloudimg-loopback-bind /proc/$p/maps)"; done
sudo nft list table inet cloudimg_storm

storm-nimbus: shim mapped 5 time(s)
storm-supervisor: shim mapped 5 time(s)
storm-ui: shim mapped 5 time(s)
table inet cloudimg_storm {
    chain input {
        type filter hook input priority filter - 5; policy accept;
        iifname "lo" accept
        tcp dport { 2181, 2888, 3772-3774, 3888, 6627-6628, 6699-6799, 8000, 8080, 8089 } counter packets 0 bytes 0 reject with tcp reset
    }
}

This is a single-node cluster by design. To run topologies across several machines, build a multi-node Storm cluster with authentication (Kerberos/SASL and TLS between the daemons); this image is not meant to be joined to one.

Step 11: Your own DNS name and a CA-issued certificate

Point a DNS record at the instance, then tell the appliance to serve that name. This updates nginx, issues a new self signed end-entity certificate naming it, accepts https://<name> as the UI's origin and updates the URL and the fingerprint in the credentials file:

sudo storm-set-hostname storm.<your-domain>

The name is saved in /etc/cloudimg/storm-hostname and survives reboots. For a certificate browsers trust without a warning, install one from a certificate authority (your own CA, or Let's Encrypt). The chain file holds the server certificate first, then any intermediates; the key is PEM and unencrypted:

sudo storm-install-cert /path/to/fullchain.pem /path/to/privkey.pem storm.<your-domain>

storm-install-cert checks that the key matches the certificate and that the certificate covers the name, installs it for nginx and reloads it. The boot-time renewal never replaces a certificate installed this way, even if the instance's address changes. Running storm-set-hostname again goes back to a self signed certificate. Renew a CA certificate by running storm-install-cert again with the new files.

Step 12: Worker slots and memory

The Supervisor offers four worker slots (ports 6700 to 6703), and each worker gets a 768 MB heap by default. The daemons use about 2.3 GB together (Nimbus 1 GB, UI 768 MB, Supervisor 256 MB, ZooKeeper 256 MB), which fits an m5.large with three or four workers. To change the slots or the worker heap, edit /etc/storm/storm.yaml (a link to /opt/storm/current/conf/storm.yaml), for example:

sudoedit /etc/storm/storm.yaml
# supervisor.slots.ports:
#   - 6700
#   - 6701
#   - 6702
#   - 6703
#   - 6704
#   - 6705
# worker.heap.memory.mb: 1536
sudo systemctl restart storm-supervisor

Keep the slot ports between 6700 and 6799: that range is covered by the firewall table. A topology can ask for more heap per worker with topology.worker.max.heap.size.mb in its own configuration. On an instance with more memory, raise the daemons' heaps in the *.childopts settings of the same file and restart the service.

Step 13: Logs, versions and file locations

The daemon logs are in /var/log/storm, each worker's log in /var/log/storm/workers-artifacts/<topology-id>/<port>/worker.log:

sudo ls /var/log/storm

access-nimbus.log
access-supervisor.log
access-ui.log
access-web-nimbus.log
access-web-supervisor.log
access-web-ui.log
nimbus.log
nimbus.log.metrics
supervisor.log
supervisor.log.metrics
ui.log
ui.log.metrics
workers-artifacts

Versions and licences:

readlink -f /opt/storm/current /opt/zookeeper/current
sha256sum /opt/storm/current/LICENSE /opt/zookeeper/current/LICENSE.txt
/opt/storm/jre/bin/java -version 2>&1 | head -1
storm version 2>&1 | grep '^Storm'

/opt/storm/apache-storm-3.1.0
/opt/zookeeper/apache-zookeeper-3.9.6-bin
fb5d8293b02cce4a10abc1d59a1b85062147cb702b0aecc538c3ba67984cd46a  /opt/storm/current/LICENSE
cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30  /opt/zookeeper/current/LICENSE.txt
openjdk version "25.0.5" 2026-10-20 LTS
Storm 3.1.0

Path	What it is
/etc/storm/storm.yaml	the Storm configuration (single node, loopback, slots, heaps)
/etc/zookeeper/zoo.cfg	the ZooKeeper configuration (127.0.0.1:2181, admin server off, ruok only)
/var/lib/storm, /var/lib/zookeeper	Storm state (Nimbus blob store with your topology jars, Supervisor and worker state) and ZooKeeper data
/var/log/storm, /var/log/zookeeper	logs; also journalctl -u zookeeper -u storm-nimbus -u storm-supervisor -u storm-ui
/root/storm-credentials.txt, /root/storm-curl.conf	the UI URL, admin password and fingerprint; curl login (root only)
/etc/nginx/storm-htpasswd	the admin password's SHA-512 hash (root and nginx only)
/etc/cloudimg/storm-ui.pem	the public certificate
/etc/nginx/sites-available/cloudimg-storm	the HTTPS front door; certificate and key in /etc/nginx/ssl/
/opt/storm/examples	the Flux WordCount demo
/usr/local/lib/cloudimg/libcloudimg-loopback-bind.so	the bind shim (source in /usr/share/doc/cloudimg/apache-storm/)

Step 14: The instance metadata block

ZooKeeper, Nimbus and the UI may reach nothing but localhost. The Supervisor (and every worker it starts), nginx and the boot service are denied the instance metadata service (169.254.169.254) by systemd IPAddressDeny rules, so neither topology code nor a request to the UI can read the instance's IAM credentials. The only service allowed to reach the metadata service is storm-metadata-names, which runs once at boot, reads the instance's host names and addresses (no credentials) for the certificate, and may reach nothing else:

for u in zookeeper storm-nimbus storm-ui storm-supervisor nginx storm-firstboot storm-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u.service)" "$(systemctl show -p IPAddressAllow --value $u.service)"; done

zookeeper: deny=0.0.0.0/0 ::/0 allow=127.0.0.0/8 ::1/128
storm-nimbus: deny=0.0.0.0/0 ::/0 allow=127.0.0.0/8 ::1/128
storm-ui: deny=0.0.0.0/0 ::/0 allow=127.0.0.0/8 ::1/128
storm-supervisor: deny=169.254.169.254/32 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 fd00:ec2::254/128 168.63.129.16/32 allow=
nginx: deny=168.63.129.16/32 ::ffff:169.254.169.254/128 fd00:ec2::254/128 169.254.169.254/32 ::ffff:168.63.129.16/128 allow=
storm-firstboot: deny=::ffff:168.63.129.16/128 168.63.129.16/32 ::ffff:169.254.169.254/128 fd00:ec2::254/128 169.254.169.254/32 allow=
storm-metadata-names: deny=0.0.0.0/0 ::/0 allow=169.254.169.254/32 fd00:ec2::254/128

Consequence: topologies cannot use the instance's IAM role. A spout or bolt that needs AWS credentials (for example to read from Amazon Kinesis or write to Amazon S3) must be given them through its own configuration, such as access keys from AWS Secrets Manager that you pass to the topology, or run on a cluster built for that.

Step 15: Backup and maintenance

• Your topologies are your own jars: keep them (and their configuration) in your build system. To move a running workload, submit the same jar to the new instance. To keep the cluster's state across instances, back up /var/lib/storm and /var/lib/zookeeper with the services stopped (sudo systemctl stop storm-ui storm-supervisor storm-nimbus zookeeper), plus /etc/cloudimg and /etc/nginx/ssl.
• Ubuntu security updates install automatically (unattended-upgrades). Storm, ZooKeeper and the Java runtime live in /opt; to move to a newer release, launch the newer version from the listing and resubmit your topologies.
• Storm keeps worker logs and their artifacts in /var/log/storm/workers-artifacts; the log4j2 configuration in /opt/storm/current/log4j2 rotates them. For larger log volumes, attach an EBS volume and mount it on /var/log/storm.

Step 16: Security recommendations

• Limit TCP 443 and 22 in the security group to your admin network. Anyone with the admin login can kill and rebalance topologies and change their log levels; treat the UI as an administrative interface.
• Anyone who can log in to the instance over SSH can submit topologies, because Nimbus trusts its loopback clients. Give SSH access only to the people who should run code on the cluster.
• Serve the UI under your own DNS name with a certificate from a trusted CA (Step 11). Until then, check the self signed certificate's fingerprint (Step 4) before you sign in.
• To change the admin password, write a new SHA-512 hash to /etc/nginx/storm-htpasswd without putting the password on a command line (for example printf '%s\n' "$NEW" | openssl passwd -6 -stdin, after reading NEW with read -s), update /root/storm-credentials.txt and /root/storm-curl.conf, and reload nginx.
• Keep Storm and ZooKeeper on the loopback address: do not remove the bind shim or the firewall table, and do not add a listener for them in nginx.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk. Apache Storm and Apache ZooKeeper are licensed under the Apache License 2.0 (their LICENSE and NOTICE files are on the instance in /usr/share/doc/apache-storm and /usr/share/doc/apache-zookeeper), the Eclipse Temurin runtime under the GNU General Public License version 2 with the Classpath Exception, and nginx under the 2-clause BSD licence. Apache, Apache Storm, Storm, Apache ZooKeeper and ZooKeeper are trademarks of the Apache Software Foundation; cloudimg is not affiliated with or endorsed by the Apache Software Foundation. Eclipse Temurin is a trademark of the Eclipse Foundation.