Arroyo 0.15.0 on AWS User Guide

Overview

This guide covers launching and using the Arroyo 0.15.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Arroyo is a stream processing engine that runs SQL over real-time data. You write a query (with windows, joins and aggregates over event time) and Arroyo turns it into a stateful, fault-tolerant pipeline that processes events as they arrive, with exactly-once checkpointing. It has sources and sinks for Kafka, Kinesis, MQTT, NATS, RabbitMQ, Redis, server-sent events, WebSockets, HTTP polling, webhooks, files and object storage, plus Python user-defined functions (UDFs).

The image runs Arroyo as a complete single-node cluster (arroyo cluster): the API with the embedded web console, the controller, the compiler and an admin service, with each running pipeline in its own worker process. Pipeline definitions are kept in an embedded SQLite database and checkpoints on the instance's disk.

What the cloudimg image adds:

• Nothing reachable off the instance except SSH and HTTPS. Upstream Arroyo listens on every network interface and has no login. In this image the API (5115), the admin service (5114), the controller (5116), the compiler (5117) and every pipeline worker listen on 127.0.0.1 only; nginx is the only service on a network address.
• The console and REST API over HTTPS only, behind a login. nginx serves them on port 443 (port 80 only redirects) with HTTP Basic authentication for the user admin. The password is generated on your instance at first boot and stored only as a bcrypt hash; failed logins are rate limited. Requests for host names the instance does not serve are dropped.
• Arroyo's own API key switched on. Arroyo's API and admin services require a bearer API key that is generated on your instance at first boot. nginx presents it for signed-in users (the key never reaches your browser), and anything on the instance that talks to 127.0.0.1:5115 or :5114 directly needs the key too. The engine refuses to start without it.
• Guarded server-side requests. Testing a connection, autocompleting a connection profile, validating a query or UDF, starting a preview and streaming a pipeline's output make the server connect to other systems or return pipeline data. Those requests are refused unless the browser proves they come from the console page itself, so another web page cannot use your browser's saved login to make your instance fetch addresses inside your network. Other sites' Origin headers, WebSocket upgrades, form posts and ; path parameters are refused too.
• A sandboxed engine. Signing in lets you run SQL connectors and Python code on the instance, so the engine, its workers and your UDFs run as the unprivileged arroyo user with systemd sandboxing: the file system is read-only except Arroyo's own state directory, and the credentials file, the TLS key and the login hash are unreadable. They are also blocked from the instance metadata service, so pipeline code can never read the instance's IAM credentials.
• No default secret. The login, the API key and the TLS private key are generated on your instance at first boot; the image ships none of them, and Arroyo starts with an empty metadata database.
• HTTPS from the first boot. The certificate is generated on your instance for its own names and addresses. It is an end-entity certificate: it can only vouch for this instance and can never sign other certificates. Install your own CA-issued certificate with one command (Step 14); the boot-time renewal never replaces it.
• Hardened defaults. Telemetry to the Arroyo project is off, nginx hands Arroyo the real client address and the configured host name (never values a client sends) and strips Arroyo's permissive CORS headers, SSH accepts keys only, and Ubuntu 24.04 LTS is fully patched at build time with unattended security upgrades switched on.

What is on the image: Arroyo 0.15.0 (the official Python-enabled linux-x86_64 release, verified against the SHA-256 digest GitHub publishes for it), Python 3.12 from Ubuntu 24.04 for Python UDFs, nginx 1.24 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent. No compiler toolchain is installed.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH) and TCP 443 (the console and API) from your admin network, plus TCP 80 if you want plain HTTP requests redirected to HTTPS. There is nothing else to open: Arroyo's own ports listen on the loopback address only and refuse connections from other machines even if a security group allows them. Pipelines open outbound connections to your sources and sinks (Kafka brokers, databases, HTTP endpoints), so allow that outbound traffic.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Arroyo 0.15.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger for heavy pipelines), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=arroyo}]'

First boot takes about a minute after the instance is running: the boot service generates the login, the API key and the certificate, then Arroyo and nginx start.

Step 2: Connect to your instance

Version	SSH user
Arroyo 0.15.0 on Ubuntu 24.04	ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services and the listening set

systemctl is-active arroyo-metadata-names arroyo-firstboot arroyo nginx
sudo test -f /var/lib/cloudimg/arroyo-firstboot.done && echo FIRSTBOOT_DONE

All four units are active and the first boot marker is present:

active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network addresses; Arroyo's admin service (5114), API (5115), controller (5116) and compiler (5117) listen on 127.0.0.1:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:5114 users:(("arroyo"))
127.0.0.1:5115 users:(("arroyo"))
127.0.0.1:5116 users:(("arroyo"))
127.0.0.1:5117 users:(("arroyo"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

While a pipeline runs, its worker process adds a few more listeners on random ports, also on 127.0.0.1 only.

Step 4: Get the console URL, login, API key and certificate fingerprint

The credentials file is readable by root only. It holds the console URL, the admin user and password, the Arroyo API key and the SHA-256 fingerprint of this instance's certificate:

sudo cat /root/arroyo-credentials.txt

# Arroyo 0.15.0 on Ubuntu 24.04 - generated on first boot by arroyo-firstboot.service.
# The console login and the Arroyo API key are unique to this instance; none was baked into the image. Keep this file
# secret. Signing in to the console lets you run SQL pipelines and Python UDFs, which is code execution on this
# instance: share the login only with people you would give a shell to.
# nginx stores the password only as a bcrypt hash (/etc/nginx/arroyo-htpasswd). The console is served over HTTPS only
# (port 443; port 80 redirects). The certificate is self signed (minted on this instance): before you sign in, check
# that the SHA-256 fingerprint your browser shows matches ARROYO_TLS_CERT_SHA256 below - or install a CA-issued
# certificate with
#   sudo arroyo-install-cert <chain.pem> <key.pem>
# Change the served name: sudo arroyo-set-hostname <dns-name>      Re-verify: sudo arroyo-selfcheck
ARROYO_URL=https://<public-ip>/
ARROYO_HOST=<public-ip>
ARROYO_ADMIN_USER=admin
ARROYO_ADMIN_PASSWORD=<ARROYO_ADMIN_PASSWORD>
ARROYO_API_KEY=<api-key>
ARROYO_TLS_CERT_SHA256=<sha256-fingerprint>
ARROYO_TLS_CERT_FILE=/etc/cloudimg/arroyo-ui.pem
# REST API through nginx with the login:  sudo curl -K /root/arroyo-curl.conf --cacert /etc/cloudimg/arroyo-ui.pem https://<public-ip>/api/v1/pipelines
# REST API on this instance with the key: sudo curl -K /root/arroyo-api.curl http://127.0.0.1:5115/api/v1/pipelines
# User guide: https://www.cloudimg.co.uk/guides/arroyo-0-15-0-on-ubuntu-24-04-aws/

The URL, the user and the fingerprint (without the password or the key):

sudo grep -E '^ARROYO_(URL|ADMIN_USER|TLS_CERT_SHA256)=' /root/arroyo-credentials.txt

ARROYO_URL=https://<public-ip>/
ARROYO_ADMIN_USER=admin
ARROYO_TLS_CERT_SHA256=<sha256-fingerprint>

The host is the instance's public IP address from the EC2 instance metadata (the private address when the instance has no public one). The certificate also names the public DNS name, the private IP address and the private DNS name. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it (unless you installed your own, Step 14).

The certificate's names and constraints, read from the instance:

openssl x509 -in /etc/cloudimg/arroyo-ui.pem -noout -ext subjectAltName,basicConstraints,keyUsage,extendedKeyUsage

X509v3 Subject Alternative Name:
    IP Address:<public-ip>, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<private-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication

Step 5: Open the console

Browse to the ARROYO_URL from Step 4. The certificate is self signed and unique to this instance, so the browser warns about it the first time. Check the certificate before you continue: open the browser's certificate viewer for the page and compare its SHA-256 fingerprint with ARROYO_TLS_CERT_SHA256 from the credentials file. Continue only if they match, and only for this site. Never add this certificate to an operating-system or browser trust store; for a certificate browsers trust without a warning, install a CA-issued one (Step 14).

The browser then asks for a user name and password: sign in as admin with the password from the credentials file. To print just the password on the instance:

sudo awk -F= '$1=="ARROYO_ADMIN_PASSWORD" {print $2}' /root/arroyo-credentials.txt

Pipelines lists your pipelines with their state (screenshot at the top of this guide). Create Pipeline opens the SQL editor: write a query, press Preview to run it against live data without saving it, and Launch to start it as a pipeline. Here a Nexmark benchmark source (built into Arroyo) feeds a query that also calls a Python UDF (Step 8):

Each pipeline has its own page with the operator graph, outputs, checkpoints, the query, its UDFs and any errors:

Connections creates reusable connection profiles and tables for the external systems your pipelines read and write:

If a preview stays on Loading..., press Stop and Preview again: the console sometimes subscribes to the output a moment before the job is ready.

Step 6: The REST API

Everything the console does goes through Arroyo's REST API (/api/v1/...). There are two ways to call it:

• Through nginx, with the login, from anywhere that can reach port 443. /root/arroyo-curl.conf holds the login as a curl config file, so the password never appears on a command line. nginx replaces your login with the API key on the way to Arroyo.
• On the instance, with the API key, directly on 127.0.0.1:5115. /root/arroyo-api.curl holds the key as a curl header line.

On the instance the examples pin the instance's public name to 127.0.0.1 with --resolve, so they work whatever the security group allows, and check the server against this instance's certificate:

H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem https://$H/api/v1/ping; echo
curl -s -o /dev/null -w 'without the login: %{http_code}\n' --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem https://$H/api/v1/pipelines
curl -s -o /dev/null -w 'direct, without the key: %{http_code}\n' http://127.0.0.1:5115/api/v1/pipelines
sudo curl -s -K /root/arroyo-api.curl http://127.0.0.1:5115/api/v1/connectors | python3 -c 'import json,sys; print(" ".join(sorted(c["id"] for c in json.load(sys.stdin)["data"])))'

"Pong"
without the login: 401
direct, without the key: 401
blackhole confluent delta filesystem fluvio iceberg impulse kafka kinesis mqtt nats nexmark polling_http preview rabbitmq redis single_file_source sse stdout webhook websocket

From your workstation, copy the certificate (Step 4) and keep the login in a file only you can read, then call the API the same way:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/cloudimg/arroyo-ui.pem arroyo.pem
openssl x509 -in arroyo.pem -noout -fingerprint -sha256      # must match ARROYO_TLS_CERT_SHA256
( umask 077; printf 'user = "admin:%s"\n' "<your-password>" > arroyo-login.conf )
curl -K arroyo-login.conf --cacert arroyo.pem https://<public-ip>/api/v1/pipelines

Requests that stream a pipeline's output or make the server connect to another system (connection tests, autocomplete, query and UDF validation, previews) must carry the browser's same-origin signal; scripts add -H 'Sec-Fetch-Site: same-origin', as in Step 7. The full API description is at /api/v1/api-docs/openapi.json (behind the same login).

Step 7: Create and run a SQL pipeline

This pipeline reads Arroyo's built-in Nexmark source (a generated stream of auctions, bids and people) and counts the bids per auction in 5-second tumbling windows. Create it through the API (the console's Create Pipeline does the same):

mkdir -p ~/arroyo-demo && cd ~/arroyo-demo
cat > bids.sql <<'SQL'
CREATE TABLE nexmark WITH (connector = 'nexmark', event_rate = '200');
SELECT bid.auction AS auction, count(*) AS bids
FROM nexmark
WHERE bid IS NOT NULL
GROUP BY bid.auction, tumble(interval '5 seconds');
SQL
python3 -c 'import json; print(json.dumps({"name": "nexmark-bids", "query": open("bids.sql").read(), "parallelism": 1}))' > pipeline.json
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @pipeline.json https://$H/api/v1/pipelines \
  | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["id"], p["name"])' | tee pipeline.id

pl_<id> nexmark-bids

Wait for its job to run:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt); P=$(cut -d' ' -f1 pipeline.id)
for i in $(seq 1 30); do
  sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem https://$H/api/v1/pipelines/$P/jobs \
    | python3 -c 'import json,sys; j=json.load(sys.stdin)["data"][0]; print(j["id"], j["state"])' > job.id
  grep -q Running job.id && break; sleep 2
done
cat job.id

job_<id> Running

Read a few seconds of its live output. The output is a server-sent events stream of record batches; this decodes them:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt); P=$(cut -d' ' -f1 pipeline.id); J=$(cut -d' ' -f1 job.id)
{ sudo timeout 15 curl -s -N -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
  -H 'Sec-Fetch-Site: same-origin' https://$H/api/v1/pipelines/$P/jobs/$J/output || true; } | python3 -c '
import json, sys
n = 0
for line in sys.stdin:
    if line.startswith("data:") and n < 6:
        for r in json.loads(json.loads(line[5:])["batch"])[:3]:
            print("auction %(auction)s: %(bids)s bids" % r); n += 1'

auction 1632: 8 bids
auction 1600: 403 bids
auction 1558: 2 bids
auction 1673: 5 bids
auction 1700: 478 bids
auction 1712: 5 bids

A pipeline with no sink of its own writes to the preview sink you just read. For production, end the query with an INSERT INTO a sink table (Kafka, a file system or object store, a webhook and so on; Step 10 shows Kafka).

Step 8: Python UDFs

Python user-defined functions work out of the box: the image's Arroyo build embeds Python 3.12 from Ubuntu and needs no compiler. A UDF is a Python function with type annotations and the @udf decorator. Register one for the whole cluster (it then appears under UDFs in the editor's side panel) and use it in a query:

cd ~/arroyo-demo
cat > price_band.py <<'PY'
from arroyo_udf import udf

@udf
def price_band(price: int) -> str:
    if price < 1000:
        return "low"
    if price < 10000:
        return "medium"
    return "high"
PY
python3 -c 'import json; print(json.dumps({"prefix": "", "language": "python", "definition": open("price_band.py").read(), "description": "Bid price band"}))' > udf.json
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @udf.json https://$H/api/v1/udfs | python3 -c 'import json,sys; u=json.load(sys.stdin); print(u["name"], u["language"])'
cat > bands.sql <<'SQL'
CREATE TABLE nexmark WITH (connector = 'nexmark', event_rate = '200');
SELECT price_band(bid.price) AS band, count(*) AS bids
FROM nexmark
WHERE bid IS NOT NULL
GROUP BY price_band(bid.price), tumble(interval '5 seconds');
SQL
python3 -c 'import json; print(json.dumps({"name": "nexmark-price-bands", "query": open("bands.sql").read(), "parallelism": 1}))' > bands.json
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @bands.json https://$H/api/v1/pipelines \
  | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["id"], p["name"])' | tee bands.id

price_band python
pl_<id> nexmark-price-bands

After a few seconds its output shows the bid count per price band for each 5-second window:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt); B=$(cut -d' ' -f1 bands.id)
for i in $(seq 1 30); do
  BJ=$(sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem https://$H/api/v1/pipelines/$B/jobs \
    | python3 -c 'import json,sys; j=json.load(sys.stdin)["data"][0]; print(j["id"] if j["state"] == "Running" else "")')
  [ -n "$BJ" ] && break; sleep 2
done
{ sudo timeout 15 curl -s -N -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
  -H 'Sec-Fetch-Site: same-origin' https://$H/api/v1/pipelines/$B/jobs/$BJ/output || true; } | python3 -c '
import json, sys
n = 0
for line in sys.stdin:
    if line.startswith("data:") and n < 6:
        for r in json.loads(json.loads(line[5:])["batch"]):
            print("%(band)s: %(bids)s bids" % r); n += 1'

high: 615 bids
medium: 154 bids
low: 151 bids
medium: 155 bids
low: 159 bids
high: 606 bids

Security note. A UDF is arbitrary Python code that runs inside the pipeline's worker process on this instance, and SQL connectors can read and write files and open connections to any address the instance can reach. Anyone with the console login can execute code on the instance, so treat the login like a shell login. The image confines that code: it runs as the unprivileged arroyo user, may write only to /var/lib/arroyo, cannot read the credentials file, the TLS key or the login hash, and cannot reach the instance metadata service (Step 12 proves it from inside a running pipeline). The API key itself must be readable by the engine, so code in a pipeline can read it; that grants nothing the login does not already grant.

Only the Python standard library is available to UDFs. Rust UDFs need a Rust toolchain, which the image does not ship: Arroyo's compiler can download one at start-up if you set install-rustc = true and install-clang = true in /etc/arroyo/config.toml and restart Arroyo, but that downloads and runs a compiler from the internet as the arroyo user; Python UDFs cover most needs without it.

Step 9: Stop and delete pipelines

Stop a pipeline with a final checkpoint (or "immediate"), then delete it:

cd ~/arroyo-demo
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
for f in pipeline.id bands.id; do
  P=$(cut -d' ' -f1 $f)
  sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
    -X PATCH -H 'Content-Type: application/json' --data '{"stop": "checkpoint"}' https://$H/api/v1/pipelines/$P | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["name"], "stop:", p["stop"])'
done
sleep 15
for f in pipeline.id bands.id; do
  P=$(cut -d' ' -f1 $f)
  sudo curl -s -o /dev/null -w "delete $P: %{http_code}\n" -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem -X DELETE https://$H/api/v1/pipelines/$P
done

nexmark-bids stop: checkpoint
nexmark-price-bands stop: checkpoint
delete pl_<id>: 200
delete pl_<id>: 200

{"stop": "none"} restarts a stopped pipeline from its last checkpoint. In the console, use Stop on the pipeline's page and the delete icon in the pipelines list.

Step 10: Read from Kafka over TLS

Create a Kafka connection profile (console: Connections, Create, Kafka), then tables that use it. For a broker with a TLS listener, set the bootstrap servers and these Connection Properties (librdkafka options):

Property	Value
security.protocol	SSL (or SASL_SSL with SASL authentication)
ssl.ca.location	a PEM file on the instance, readable by Arroyo, that holds the CA (or the broker certificate) to trust

Arroyo reads files under /etc and /var/lib/arroyo, so place the trust file in /etc/arroyo/kafka/. If the broker uses a self-signed certificate, pin it for this client only: copy the broker certificate to the instance, check its fingerprint against the one the broker administrator gives you, and point ssl.ca.location at that copy. Never add a broker's self-signed certificate to the instance's system trust store.

sudo install -d -m 0755 /etc/arroyo/kafka
sudo install -m 0644 broker.pem /etc/arroyo/kafka/broker.pem
openssl x509 -in /etc/arroyo/kafka/broker.pem -noout -fingerprint -sha256     # compare with the broker administrator's value

The same profile created through the API (bootstrap servers in angle brackets):

cat > kafka-profile.json <<'JSON'
{"name": "kafka-tls", "connector": "kafka",
 "config": {"bootstrapServers": "<broker-1>:9093,<broker-2>:9093",
            "authentication": {},
            "schemaRegistryEnum": {},
            "connectionProperties": {"security.protocol": "SSL", "ssl.ca.location": "/etc/arroyo/kafka/broker.pem"}}}
JSON
H=$(sudo sed -n 's/^ARROYO_HOST=//p' /root/arroyo-credentials.txt)
sudo curl -s -K /root/arroyo-curl.conf --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/arroyo-ui.pem \
  -H 'Content-Type: application/json' --data-binary @kafka-profile.json https://$H/api/v1/connection_profiles

A query then reads a topic through the profile and writes results back to another topic:

CREATE TABLE orders (order_id TEXT, amount DOUBLE)
WITH (connector = 'kafka', connection_profile = 'kafka-tls', topic = 'orders', type = 'source', format = 'json');

CREATE TABLE order_totals (orders BIGINT, total DOUBLE)
WITH (connector = 'kafka', connection_profile = 'kafka-tls', topic = 'order-totals', type = 'sink', format = 'json');

INSERT INTO order_totals
SELECT count(*) AS orders, sum(amount) AS total
FROM orders GROUP BY tumble(interval '1 minute');

Keep SASL passwords out of saved queries: connection fields marked var-str accept {{ NAME }} placeholders that Arroyo fills from its environment. Put the values in a root-only file and add it to the service with sudo systemctl edit arroyo ([Service] then EnvironmentFile=/etc/arroyo/secrets.env). The instance metadata block (Step 12) means Arroyo cannot use the instance's IAM role, so Amazon MSK IAM authentication and the Kinesis connector need explicit credentials.

Step 11: Pipeline state and checkpoints

Arroyo keeps its metadata database (pipelines, connection profiles, UDFs) and the pipelines' checkpoints in /var/lib/arroyo, the only directory the engine may write:

sudo ls -l /var/lib/arroyo | awk 'NR>1 {print $1, $3, $4, $9}'
grep -E '^(checkpoint-url|default-checkpoint-interval)' /etc/arroyo/config.toml

drwxr-x--- arroyo arroyo artifacts
drwxr-x--- arroyo arroyo checkpoints
-rw-r----- arroyo arroyo config.sqlite
-rw-r----- arroyo arroyo config.sqlite-shm
-rw-r----- arroyo arroyo config.sqlite-wal
checkpoint-url = "/var/lib/arroyo/checkpoints"
default-checkpoint-interval = "10s"

The directory starts empty on every new instance. For large state, attach an EBS volume, stop Arroyo (sudo systemctl stop arroyo), copy /var/lib/arroyo onto the volume, mount it on /var/lib/arroyo (owner arroyo, mode 0750) and start Arroyo again. To keep checkpoints in Amazon S3 instead, set checkpoint-url = "s3://<bucket>/<prefix>" in /etc/arroyo/config.toml and give Arroyo credentials through an EnvironmentFile as in Step 10 (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION); the instance's IAM role is not reachable from the engine.

Step 12: Security checks

arroyo-selfcheck re-proves the whole appliance with this instance's credentials. It checks the units and the per-instance files, the served certificate, the login (anonymous and wrong-password requests refused, the login accepted) and the API key on 127.0.0.1 (missing and wrong keys refused), the redirect, unknown host names, foreign origins, WebSocket upgrades, form posts, ; path parameters and encoded traversal, and the exact listening set. It then runs a real SQL pipeline and checks its output, and from inside running pipelines it proves the sandbox: a Python UDF cannot read the credentials file, the TLS key, the login hash or nginx's key file and cannot reach the instance metadata service (while it can reach a loopback address), and an HTTP-source connector pointed at 169.254.169.254 fails while the same connector reads a local test endpoint. It cleans up after itself:

sudo arroyo-selfcheck

  ok  units active; per-instance credentials, login hash, API key and TLS key present with their permissions
  ok  TLS: the served certificate is this instance's end-entity certificate (CA:FALSE, serverAuth; SHA-256 <first-bytes>...)
  ok  front door: HTTPS only (80 -> 301), unknown Host dropped, login 401/401/200 on the console, API and output stream, foreign Origin 403, WebSocket 403, OPTIONS 405, form POST 415, ';' 400, encoded traversal 400, CORS stripped, basename header blanked; API key on 127.0.0.1:5115/:5114 401/401/200
  ok  public listeners 22/80/443 only; every Arroyo listener on 127.0.0.1
  ..  400 live output records, 400 with transformed == counter*10+7
  ok  SQL pipeline: invalid query refused 400; impulse -> transform Running, live output verified; worker in arroyo's cgroup, every listener on loopback
  ..  25 UDF probe rows: uid=<arroyo-uid>;creds=EACCES;tlskey=EACCES;htpasswd=EACCES;bearer=EACCES;control=READ;imds=TIMEOUT;imds6=TIMEOUT;loopback=OPEN
  ok  Python UDF (in a running pipeline, uid arroyo): credentials file, TLS key, login hash and bearer include unreadable (EACCES; config control READ); 169.254.169.254 unreachable (v4 and v4-mapped), loopback control OPEN
  ok  HTTP-source connector (polling_http, in a running pipeline): http://169.254.169.254/ -> 'error sending request', 0 records; loopback control -> the nonce
SELFCHECK OK

The engine's sandbox settings:

systemctl show -p User,NoNewPrivileges,ProtectSystem,ProtectHome,PrivateTmp,ReadWritePaths arroyo.service

User=arroyo
ReadWritePaths=/var/lib/arroyo
PrivateTmp=yes
ProtectHome=yes
ProtectSystem=strict
NoNewPrivileges=yes

Step 13: The instance metadata block

Arroyo (and so every pipeline worker and UDF), nginx and the boot service are denied the instance metadata service (169.254.169.254) by systemd IPAddressDeny rules, so neither pipeline code nor a request to the console can read the instance's IAM credentials. The only service allowed to reach the metadata service is arroyo-metadata-names, which runs once at boot, reads the instance's host names and addresses (no credentials) for the certificate, and may reach nothing else:

for u in arroyo nginx arroyo-firstboot arroyo-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u.service)" "$(systemctl show -p IPAddressAllow --value $u.service)"; done

arroyo: deny=::ffff:168.63.129.16/128 169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32 ::ffff:169.254.169.254/128 allow=
nginx: deny=169.254.169.254/32 fd00:ec2::254/128 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 168.63.129.16/32 allow=
arroyo-firstboot: deny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32 ::ffff:169.254.169.254/128 ::ffff:168.63.129.16/128 allow=
arroyo-metadata-names: deny=::/0 0.0.0.0/0 allow=169.254.169.254/32 fd00:ec2::254/128

Pipelines can still reach your sources and sinks on the network and the internet; only the metadata address is blocked. Connectors that would normally use the instance's IAM role (Kinesis, S3 checkpoints, MSK IAM) need explicit credentials (Step 10).

Step 14: Your own DNS name and a CA-issued certificate

To serve the console under your own DNS name, point the name at the instance (an Elastic IP is recommended) and set it. This writes the name into nginx and issues a new self-signed certificate that covers it:

sudo arroyo-set-hostname arroyo.example.com

To install a certificate from your CA (or Let's Encrypt), copy the full chain (server certificate first) and the unencrypted private key to the instance, then:

sudo arroyo-install-cert /path/to/fullchain.pem /path/to/privkey.pem arroyo.example.com

The boot-time renewal never replaces a certificate installed this way. sudo arroyo-set-hostname <name> goes back to a self-signed certificate. Both commands update the URL and fingerprint in /root/arroyo-credentials.txt.

Step 15: Change the console password or the API key

The password is stored as a bcrypt hash in /etc/nginx/arroyo-htpasswd. To set a new one without putting it on a command line, let htpasswd prompt for it, then update the curl config and the credentials file:

sudo htpasswd -B -C 10 /etc/nginx/arroyo-htpasswd admin
sudo nano /root/arroyo-curl.conf /root/arroyo-credentials.txt
sudo systemctl reload nginx

To rotate the API key, write a new 64-character hex key (for example from openssl rand -hex 32) into the two api-key lines of /etc/arroyo/conf.d/90-cloudimg-auth.toml, into /etc/nginx/arroyo-upstream-auth.conf, /root/arroyo-api.curl and the credentials file, then sudo systemctl restart arroyo and sudo systemctl reload nginx. Edit the files with an editor rather than sed, so the key never appears on a command line.

Step 16: Logs, versions and file locations

arroyo --version
journalctl -u arroyo -b --no-pager -o cat | grep -E 'Starting (HTTP API|arroyo-controller|compiler)|cluster HTTP admin' | sed -E 's/^[0-9T:.-]+Z +//' | awk 'n < 4 {print; n++}'

arroyo 0.15.0
INFO arroyo_server_common: Starting cluster HTTP admin server on 127.0.0.1:5114
INFO arroyo_api: Starting HTTP API server on 127.0.0.1:5115
INFO arroyo_compiler_service: Starting compiler service at 127.0.0.1:5117
INFO arroyo_controller: Starting arroyo-controller on 127.0.0.1:5116

Path	Contents
/usr/local/bin/arroyo	the Arroyo 0.15.0 binary
/etc/arroyo/config.toml	Arroyo configuration (no secrets)
/etc/arroyo/conf.d/90-cloudimg-auth.toml	this instance's API key (root:arroyo 0640)
/var/lib/arroyo	metadata database, checkpoints, artifacts
/etc/nginx/sites-available/cloudimg-arroyo	the nginx front door
/root/arroyo-credentials.txt, /root/arroyo-curl.conf, /root/arroyo-api.curl	login, API key, curl configs (root only)
/usr/share/doc/arroyo	Arroyo's licences and the release provenance

Arroyo logs to the journal (journalctl -u arroyo), including every pipeline worker's log lines; nginx logs to /var/log/nginx/. Pipeline errors also appear on the pipeline's Errors tab and at /api/v1/pipelines/<id>/jobs/<job>/errors.

Step 17: Backup and maintenance

• Back up /var/lib/arroyo with Arroyo stopped (sudo systemctl stop arroyo) to keep pipelines, connection profiles, UDFs and checkpoints, plus /etc/arroyo/conf.d, /etc/cloudimg and /etc/nginx/ssl if you want to keep the key and certificate. Keep your SQL and UDF source in version control as well.
• Ubuntu security updates install automatically (unattended-upgrades). Arroyo is a single binary in /usr/local/bin; to move to a newer Arroyo release, launch the newer version from the listing and recreate your pipelines there (export the queries from the console or the API).
• Size the instance for your pipelines: each running pipeline is a worker process with its own memory for state. m5.large suits development and modest streams; use larger instances for high event rates or large windows.

Step 18: Security recommendations

• Limit TCP 443 and 22 in the security group to your admin network. The console login grants code execution on the instance (Step 8): share it only with people you would give a shell to.
• Serve the console under your own DNS name with a certificate from a trusted CA (Step 14). Until then, check the self signed certificate's fingerprint (Step 4) before you sign in.
• Keep Arroyo's services on the loopback address and its API key switched on: do not change the bind-address settings or remove the key file. The engine refuses to start with a wildcard bind or without the key.
• Give pipelines credentials of their own, scoped to the data they need, rather than broad keys. The instance's IAM role is deliberately unreachable from the engine.
• HTTP Basic authentication has no sign-out: close the browser to end a session on a shared computer.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk. Arroyo is licensed under the MIT license or the Apache License 2.0, at your option (both licence texts are on the instance in /usr/share/doc/arroyo); nginx is licensed under the 2-clause BSD licence and Python under the Python Software Foundation License. Arroyo is a trademark of its respective owner; cloudimg is not affiliated with or endorsed by the Arroyo project or its owners.