Apache Storm on AWS User Guide

Overview

This image runs Apache Storm, a distributed, fault-tolerant, real-time computation system. Storm processes unbounded streams of data with low latency by running topologies - directed graphs of spouts (sources that emit tuples) and bolts (operators that transform, aggregate or route them). It is used for real-time analytics, continuous ETL, online machine learning, distributed RPC and event processing.

A production Storm deployment has several moving parts, and this image runs them all on a single instance so the cluster is useful the moment it boots:

• Nimbus - the master, or control plane. It accepts topology submissions, distributes code, assigns work and monitors for failures. Its Thrift API listens on 127.0.0.1:6627.
• Supervisor - the worker daemon. It runs the topology's tasks in worker slots (this image is configured with four, on ports 6700-6703).
• Storm UI - the web dashboard, which reports cluster, topology, spout and bolt statistics. It listens on 127.0.0.1:8080 and is published on port 80 by nginx.
• Apache ZooKeeper - the coordination service Storm requires for cluster state. A single-node instance is bundled and binds 127.0.0.1:2181.

The Storm UI and the Nimbus Thrift API ship upstream with no authentication, and Nimbus can submit or kill topologies - which is arbitrary code execution on the cluster. Treat Nimbus as a privileged control plane and keep it private. This image does that for you: the Storm UI, Nimbus and ZooKeeper all bind to the loopback interface, a loopback-only firewall keeps their ports unreachable from off the instance regardless of your security group, and nginx publishes only the Storm UI on port 80 behind HTTP Basic authentication. The only network-reachable surfaces are SSH on port 22 and the Storm UI on port 80.

Storm's local working state (storm.local.dir), the ZooKeeper data and transaction logs, and the Storm and worker logs all live at /var/lib/storm, which is a dedicated, independently resizable EBS data volume kept off the operating system disk.

The administrator password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/storm-credentials.txt with mode 0600, so only the root user can read it. First boot also submits the bundled storm-starter WordCount topology, so the Storm UI shows a genuine running topology processing tuples immediately.

Connecting to your instance

Connect over SSH as the default login user for the operating system variant you launched. This listing currently ships the following variant:

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the Storm UI
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Apache Storm. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. The instance runs several Java virtual machines - Nimbus, the Supervisor, the Storm UI and ZooKeeper - so smaller burstable types are not recommended. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the Storm UI. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a short time after the instance state becomes Running and the status checks pass, while the image generates its administrator password and submits the demonstration topology. The Storm and ZooKeeper JVMs take a little longer to become fully ready than a typical web app.

Step 2: Launch the Instance from the AWS CLI

You can launch the same image from the command line. Replace the AMI ID with the one shown on the Marketplace listing for your Region, and set your key pair, subnet and security group.

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type m5.large \
  --key-name your-key \
  --subnet-id subnet-xxxxxxxx \
  --security-group-ids sg-xxxxxxxx \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-storm}]'

Step 3: Retrieve the Administrator Password

SSH into the instance as ubuntu, then read the per-instance credentials file. It is mode 0600 and owned by root, so use sudo:

sudo cat /root/storm-credentials.txt

The file contains the Storm UI URL, the administrator user (admin) and the generated password:

STORM_URL=http://<public-ip>/
STORM_ADMIN_USER=admin
STORM_ADMIN_PASSWORD=<generated-per-instance>

Step 4: Sign In to the Storm UI

Open your browser to the instance's public address on port 80:

http://<public-ip>/

nginx prompts for HTTP Basic authentication. Sign in as admin with the password from the credentials file. The Storm UI dashboard opens on the Cluster Summary, showing the Storm version, the number of supervisors and worker slots, and the Nimbus leader.

The full cluster view lists the Nimbus leader on port 6627, the Owner and Topology summaries, the Supervisor and its worker slots, and the complete Nimbus configuration.

Step 5: Explore the Running Topology

Every instance boots with the bundled storm-starter WordCount topology already submitted. Click the wordcount topology in the Topology Summary to open its detail page. You can see the topology is ACTIVE and watch its statistics climb: the topology-wide emitted and transferred tuple counts, and the per-spout and per-bolt breakdown. WordCount is a three-stage pipeline - a spout that emits random sentences, a split bolt that splits them into words, and a count bolt that counts word occurrences - so you can see tuples flowing from stage to stage.

From this page you can also Activate, Deactivate, Rebalance or Kill the topology, and open the visualization and per-component views.

Step 6: Verify the Cluster from the Command Line

You can confirm the whole stack is healthy over SSH. Check that every service is active:

systemctl is-active zookeeper storm-nimbus storm-supervisor storm-ui nginx

nginx serves an unauthenticated health endpoint used for monitoring:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1/health

The Storm UI REST API is reachable through nginx with the administrator credential. A request without credentials is rejected with 401, and a request with the per-instance password returns the cluster summary as JSON. Read the generated password from the credentials file and pass it to the API:

PW=$(sudo grep '^STORM_ADMIN_PASSWORD=' /root/storm-credentials.txt | cut -d= -f2-)
curl -s -u "admin:$PW" http://127.0.0.1/api/v1/cluster/summary

The response includes the stormVersion and the supervisor and slot counts.

Step 7: Submit and Manage Your Own Topologies

Storm is operated with the storm command-line tool at /opt/storm/current/bin/storm, run as the storm service user. The examples below list running topologies, submit a packaged topology jar to Nimbus, and kill a topology by name. Package your topology as a jar with its main class, copy it to the instance, then:

# Run these on your instance after connecting: ssh ubuntu@<public-ip>
# List running topologies
sudo -u storm /opt/storm/current/bin/storm list

# Submit a topology jar to Nimbus (main class then topology name and args)
sudo -u storm /opt/storm/current/bin/storm jar /path/to/your-topology.jar com.example.MyTopology my-topology

# Kill a topology by name (Storm drains it gracefully)
sudo -u storm /opt/storm/current/bin/storm kill my-topology

Nimbus assigns the topology to worker slots on the Supervisor, and it appears in the Storm UI within a few seconds. To scale beyond a single node, add more Supervisor instances pointed at this instance's ZooKeeper and Nimbus - contact cloudimg support for a multi-node reference architecture.

Security Model

Understanding what is exposed matters for a stream-processing control plane:

• Nimbus is privileged. Anyone who can reach the Nimbus Thrift API (127.0.0.1:6627) can submit or kill topologies, which runs arbitrary code on the cluster. This image binds Nimbus to loopback and a firewall keeps port 6627 unreachable from off the instance. Do not expose it.
• The Storm UI has no native authentication. It binds to 127.0.0.1:8080; nginx fronts it on port 80 with HTTP Basic authentication, and the per-instance password is generated at first boot. There is no shared or default credential in the AMI.
• ZooKeeper binds 127.0.0.1:2181 and is firewalled to loopback.
• Your security group should open only port 22 (from your management network) and port 80 (to your users). Never open 6627, 8080 or 2181.

For production, terminate TLS in front of the Storm UI. The simplest path is to point a DNS name at the instance and configure a certificate (for example with Let's Encrypt via certbot) on the nginx server block that currently proxies port 80 to 127.0.0.1:8080. cloudimg support can provide a tested nginx TLS configuration.

Data Volume

Storm's local state, the ZooKeeper data and transaction logs, and the Storm and worker logs live on a dedicated 30 GiB gp3 EBS volume mounted at /var/lib/storm, separate from the operating system disk and independently resizable. It is mounted by filesystem UUID with the nofail option, so the layout is reproduced on every instance launched from the image.

findmnt /var/lib/storm

To grow it, modify the EBS volume in the console, then extend the filesystem with resize2fs - contact support if you need the exact procedure for your instance.

Backup and Maintenance

• Topology code is your source of truth - keep your topology jars in version control and redeploy them with storm jar.
• Operating state (which topologies are running, ZooKeeper state) lives under /var/lib/storm; snapshot the EBS volume for point-in-time recovery.
• Service management uses systemctl - the units are zookeeper, storm-nimbus, storm-supervisor, storm-ui and nginx.
• OS updates apply with sudo apt-get update && sudo apt-get upgrade in the usual way.

Support

Every cloudimg deployment is backed by 24/7 support via email and live chat at support@cloudimg.co.uk, with a one-hour average response for critical issues. Support covers deployment and instance sizing, scaling to a multi-node Storm and ZooKeeper cluster, topology submission and parallelism tuning, Storm UI authentication and TLS termination, ZooKeeper configuration, and Apache Storm and ZooKeeper version upgrades.

Apache Storm and Apache ZooKeeper are trademarks of the Apache Software Foundation. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.