Apache StreamPipes on AWS User Guide

Overview

Apache StreamPipes is a self service toolbox that lets domain experts connect, analyse and explore continuous data streams without writing code. A visual pipeline editor, a library of connectors for industrial and IoT protocols, a catalogue of data processors and sinks, and live dashboards are all reached through a single web interface, so you can go from a raw sensor feed to a working alert or dashboard entirely in a browser.

This image runs the official StreamPipes self hosted stack under Docker. Six containers make up the deployment: the StreamPipes core backend, the web user interface, a CouchDB store that holds users and pipeline and adapter metadata, an InfluxDB time series sink, a NATS message broker, and the full industrial IoT extensions library that supplies the adapters, processors and sinks. The whole stack is managed as a single systemd unit, so it starts and stops with the instance.

Only SSH on port 22 and HTTP on port 80 are published. The message broker and both databases are reachable only on the internal Docker network, and the user interface container binds to the loopback interface, with nginx publishing it on port 80 and serving an unauthenticated health endpoint for load balancer checks.

The administrator password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/streampipes-credentials.txt with mode 0600 so that only the root user can read it. On first boot the image also generates a fresh client secret, encryption passcode and token signing secret, all unique to that instance, and provisions the administrator against an empty user store, which is why the interactive setup wizard is already completed when you first open the interface.

What is included:

• The official Apache StreamPipes 0.98.0 self hosted stack running under Docker: backend core, web interface, CouchDB, InfluxDB, NATS and the full industrial IoT extensions library
• nginx on port 80 as a reverse proxy to the StreamPipes interface, with an unauthenticated static health endpoint for load balancers
• A per instance administrator generated on first boot, recorded in a root only file, so every instance is secured independently
• The Docker image store and every container volume relocated onto a dedicated 60 GiB gp3 data volume at /var/lib/streampipes, independently resizable and separate from the operating system disk
• The whole multi container stack managed as one streampipes.service systemd unit, alongside docker.service and nginx.service, all enabled
• The message broker and both databases bound to the internal Docker network only, with just ports 22 and 80 exposed
• 24/7 cloudimg support

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target region for SSH access
• A security group allowing inbound TCP 22 from your management network and inbound TCP 80 from the clients that will use the interface

The recommended instance type is m5.large (2 vCPU, 8 GiB RAM). The six container stack idles at roughly 2 GiB of memory, so m5.large leaves comfortable headroom for pipelines and adapters. Increase the instance size if you plan to run many concurrent pipelines or high throughput adapters.

StreamPipes serves plain HTTP on port 80. For production, terminate TLS in front of it with your own domain, as described under Enabling HTTPS below.

Step 1 - Launch from the AWS Marketplace

Sign in to the AWS Marketplace, search for Apache StreamPipes by cloudimg, and choose Continue to Subscribe, then Continue to Configuration. Select the software version and your region, then Continue to Launch.

Choose Launch through EC2 for full control over networking and storage. Pick the m5.large instance type, select your key pair, choose a VPC and subnet, and attach a security group that allows inbound TCP 22 and TCP 80. The dedicated 60 GiB data volume is part of the image and appears automatically under Storage. Review the settings and choose Launch instance.

Step 2 - Launch from the AWS CLI

Replace the AMI id, key name, security group and subnet with your own values.

aws ec2 run-instances \
  --image-id <ami-id-from-the-marketplace-listing> \
  --instance-type m5.large \
  --key-name <your-key-pair> \
  --security-group-ids <your-security-group-id> \
  --subnet-id <your-subnet-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=streampipes}]'

First boot takes a few minutes: the one shot initialisation service generates this instance's secrets, provisions the administrator against an empty user store, and waits for the backend to report healthy before it finishes.

Connecting to your instance

Connect over SSH on port 22 using the key pair you selected at launch. The login user depends on the operating system variant you launched:

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Step 3 - Retrieve the administrator credentials

The administrator email, password and interface URL are written to a root only file on first boot. Read it over SSH:

sudo cat /root/streampipes-credentials.txt

The file records STREAMPIPES_URL, STREAMPIPES_ADMIN_EMAIL and STREAMPIPES_ADMIN_PASSWORD. The administrator email is admin@cloudimg.local. Treat the password as a secret and change it from the interface once you have signed in.

You can confirm the file exists and is protected without displaying its contents:

sudo test -f /root/streampipes-credentials.txt && echo "credentials file present (mode $(sudo stat -c %a /root/streampipes-credentials.txt), owner $(sudo stat -c %U /root/streampipes-credentials.txt))"

credentials file present (mode 600, owner root)

You can also verify that the generated password authenticates, and that a wrong one is refused, without printing the secret:

sudo bash -c 'PW=$(grep "^STREAMPIPES_ADMIN_PASSWORD=" /root/streampipes-credentials.txt | cut -d= -f2-); [ -n "$PW" ] && echo "password present, length ${#PW}"; curl -s -o /dev/null -w "correct password -> HTTP %{http_code}\n" -X POST -H "Content-Type: application/json" -d "{\"username\":\"admin@cloudimg.local\",\"password\":\"$PW\"}" http://127.0.0.1/streampipes-backend/api/v2/auth/login; curl -s -o /dev/null -w "wrong password   -> HTTP %{http_code}\n" -X POST -H "Content-Type: application/json" -d "{\"username\":\"admin@cloudimg.local\",\"password\":\"definitely-wrong\"}" http://127.0.0.1/streampipes-backend/api/v2/auth/login'

password present, length 24
correct password -> HTTP 200
wrong password   -> HTTP 401

Step 4 - Sign in to the web interface

Browse to http://<public-ip>/ and sign in with the administrator email and the password from the credentials file. Because the administrator is provisioned automatically on first boot, you are taken straight to the sign in page rather than an installation wizard.

After signing in you land on the home view, which summarises your pipelines, adapters, dashboards and charts, and links to each StreamPipes module.

The first time you sign in a welcome dialog offers an interactive tour. Choose Start Tour to take it, or Do not show again to dismiss it permanently.

Step 5 - Connect a data source

Choose Connect in the left sidebar, then New adapter. The bundled industrial IoT extensions library supplies adapters for a wide range of protocols and systems, including Apache Kafka, Apache Pulsar, Apache RocketMQ, MQTT, OPC UA, BACnet/IP, Allen Bradley ETH and generic HTTP and file sources.

Pick the adapter that matches your source, choose Create, and work through the configuration wizard. StreamPipes samples the incoming data, infers a schema, and lets you adjust field names, data types and units before you save. Once saved, the adapter produces a data stream that any pipeline can consume.

Step 6 - Build a pipeline

Choose Pipelines & Functions in the sidebar, then New Pipeline. The editor shows your data streams and the catalogue of data processors and sinks on the left, and a canvas on the right.

Drag a data stream onto the canvas, drag one or more processors after it, such as a filter, aggregation or boolean rule, and finish with a sink such as the live dashboard, the data lake or a notification. Connect the elements, configure each one in the dialog that opens, then choose Save and start the pipeline. Live results appear under Dashboard, and historical data under Data Explorer.

Verifying the deployment

Confirm the three services that make up the deployment are running:

systemctl is-active docker.service nginx.service streampipes.service

active
active
active

Confirm the unauthenticated health endpoint responds, which is the check to point a load balancer at:

curl -s -o /dev/null -w 'health endpoint: HTTP %{http_code}\n' http://127.0.0.1/health

health endpoint: HTTP 200

Confirm the StreamPipes release this image ships:

grep '^SP_VERSION=' /opt/streampipes/compose/.env

SP_VERSION=0.98.0

Storage layout

The whole StreamPipes footprint lives on a dedicated gp3 data volume mounted at /var/lib/streampipes, separate from the operating system disk and independently resizable. Both the Docker data root and the containerd image store are relocated there before any container image is pulled, so the container image layers, the CouchDB user and metadata store, the InfluxDB time series data and the backend assets all sit on that volume.

findmnt -no SOURCE,TARGET,FSTYPE /var/lib/streampipes; df -h --output=target,size,used,avail /var/lib/streampipes | tail -1

/dev/nvme1n1 /var/lib/streampipes ext4
/var/lib/streampipes   59G  2.0G   54G

The mount is recorded in /etc/fstab by filesystem UUID rather than by device name, so it survives reboots and instance type changes. To grow the volume, expand it in the EC2 console, then run sudo resize2fs /dev/nvme1n1 on the instance.

Operating the stack

The six containers are managed as one systemd unit. The compose project lives at /opt/streampipes/compose.

Stopping streampipes.service brings the whole stack down cleanly; starting it brings every container back up. Because all state is on the data volume, stopping and starting the stack, or rebooting the instance, does not lose pipelines, adapters or recorded data.

Enabling HTTPS

The image serves plain HTTP on port 80. For production, put your own certificate in front of it. Point a DNS record at the instance, open TCP 443 in the security group, then install certbot and obtain a certificate for your domain:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

Certbot edits the nginx site at /etc/nginx/sites-available/cloudimg-streampipes in place, adds the TLS listener and sets up automatic renewal. The proxy headers already present in that site are the ones StreamPipes needs, so no further changes are required.

Alternatively, terminate TLS at an Application Load Balancer with a certificate from AWS Certificate Manager, and restrict the instance security group so port 80 only accepts traffic from the load balancer.

Backup and maintenance

Everything worth preserving is on the data volume at /var/lib/streampipes, so the simplest backup is a scheduled EBS snapshot of that volume, which you can automate with Amazon Data Lifecycle Manager. Take snapshots with the stack stopped, or accept crash consistent snapshots for a running stack.

Keep the operating system patched with sudo apt-get update && sudo apt-get upgrade. To move to a newer StreamPipes release, edit SP_VERSION in /opt/streampipes/compose/.env, then restart the stack with sudo systemctl restart streampipes; review the upstream release notes first, since some releases require migration steps.

Troubleshooting

The interface does not load. Check that the security group allows inbound TCP 80 from your address, then confirm the health endpoint responds locally with the check under Verifying the deployment. If the health endpoint returns 200 but the interface does not load, the nginx proxy is up but the interface container is not ready; give it a minute and check systemctl status streampipes.

The credentials file only contains a comment. First boot has not finished. The initialisation service waits for the backend to report healthy, which takes a few minutes on first launch. Check its progress with systemctl status apache-streampipes-firstboot, then read the file again.

Sign in is rejected. Confirm you are using admin@cloudimg.local and the password from /root/streampipes-credentials.txt on that specific instance. Each instance generates its own password, so a password from another instance will not work.

An adapter or pipeline will not start. Check the backend and extensions containers are running with systemctl status streampipes, then inspect the relevant container's recent output with sudo docker logs --tail 50 streampipes-extensions-all-iiot-1.

Support

This image is published and supported by cloudimg with 24/7 technical support by email and chat, covering deployment, adapter and pipeline configuration, sizing, persistence and backup, TLS termination and upgrades.

Apache, Apache StreamPipes, StreamPipes and the Apache StreamPipes logo are trademarks of The Apache Software Foundation. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.