Streaming & Messaging AWS

Feldera 0.378.0 on AWS User Guide

| Product: Feldera 0.378.0 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Feldera 0.378.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Feldera is an incremental compute engine. You declare tables and views in standard SQL, and Feldera compiles the program into a running pipeline that keeps every view correct as rows are inserted, updated and deleted, doing only the work each change requires instead of recomputing from scratch. Data arrives through HTTP or connectors (Kafka, Debezium, PostgreSQL, Delta Lake, S3, files, URLs and more) and results leave the same ways, or you query materialized views ad hoc. Rust user-defined functions (UDFs) extend the SQL.

The image runs the MIT-licensed Feldera open-source edition the way upstream ships it: the official all-in-one pipeline-manager container (REST API, Web Console, the SQL-to-Rust compiler and the runner that executes compiled pipelines) with an official PostgreSQL container as its metadata store, both pinned by digest and started by systemd.

What the cloudimg image adds:

  • Nothing reachable off the instance except SSH and HTTPS. Upstream publishes the console on every interface with no login. Here Docker publishes the pipeline-manager to 127.0.0.1:8080 only, PostgreSQL has no network at all (the pipeline-manager reaches it through a unix socket), and compiled pipelines listen inside the container only. nginx is the only service on a network address.
  • The console and REST API over HTTPS only, behind a login or an API key. nginx serves them on port 443 (port 80 only redirects). The console uses HTTP Basic authentication for the user admin; scripts, the Python SDK and the fda CLI send this instance's API key as a bearer token. Both are generated on your instance at first boot; the password is stored only as a bcrypt hash and failed logins are rate limited. Requests for host names the instance does not serve are dropped. (Feldera's open-source edition has no login of its own without an external OIDC provider, so nginx is the authentication layer.)
  • Guarded streaming and query requests. Another web page cannot use your browser's saved login to run ad-hoc SQL, stream a pipeline's output or logs, or trigger profiles and support bundles: those requests are refused unless they come from the console page itself (or carry the API key). Other sites' Origin headers, cross-site WebSocket upgrades, CORS preflights and ; path parameters (plain or encoded as %3B) are refused too.
  • Compressed request bodies refused at the front door. Feldera 0.378.0 is built on actix-web 4.11.0 / actix-http 3.12.1, whose request-body decompression has no size limit (RUSTSEC-2026-0447, -0448 and -0455; no Feldera release carries a fix yet). No Feldera client compresses what it sends, so nginx answers 415 to any request whose Content-Encoding is not identity, before the body is read and even with a valid login. nginx is also the only HTTP parser a client reaches: it re-frames request bodies, forwards no request trailers and opens a fresh connection to Feldera for every request.
  • Sandboxed containers. Signing in lets you compile and run SQL pipelines and Rust code on the instance. The pipeline-manager runs as an unprivileged user with no Linux capabilities, no privilege escalation, a read-only root file system, Docker's default seccomp profile, no Docker socket and no host network. It cannot read the credentials file, the TLS key, the login hash or the PostgreSQL password, and pipelines cannot open connections to the instance metadata service (so they can never read the instance's IAM credentials) or to the instance's own services.
  • No default secret. The login, the API key, the PostgreSQL password and the TLS private key are generated on your instance at first boot; the image ships none of them, and PostgreSQL starts with an empty database.
  • HTTPS from the first boot. The certificate is generated on your instance for its own names and addresses. It is an end-entity certificate: it can only vouch for this instance and can never sign other certificates. Install your own CA-issued certificate with one command (Step 14).
  • Telemetry off. No analytics key is configured, the start gate refuses one, and the console's tracking pixel is blocked by the page's Content Security Policy. nginx hands Feldera the real client address and the configured host name (never values a client sends), SSH accepts keys only, and Ubuntu 24.04 LTS is fully patched at build time with unattended security upgrades switched on.

What is on the image: Feldera 0.378.0 (the official images.feldera.com/feldera/pipeline-manager:0.378.0 image, pinned by digest), PostgreSQL 16.15 (the official postgres:16.15-alpine3.24 image, pinned by digest), the fda CLI 0.378.0 (from the same Feldera image), Docker CE (from Docker's signed repository), nginx 1.24 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large (choose more memory for large pipelines) · Root volume: 30 GiB

The Feldera Web Console over HTTPS: the pipelines list

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows TCP 22 (SSH) and TCP 443 (the console and API) from your admin network, plus TCP 80 if you want plain HTTP requests redirected to HTTPS. There is nothing else to open: the pipeline-manager listens on the loopback address only and PostgreSQL has no network. Pipelines open outbound connections to your sources and sinks (Kafka brokers, databases, object storage, URLs), so allow that outbound traffic.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Feldera 0.378.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=feldera}]'

First boot takes about two minutes after the instance is running: the boot service generates the login, the API key, the PostgreSQL password and the certificate, PostgreSQL initialises its empty database, and the pipeline-manager prepares its state volume from the image's precompiled Rust dependencies, then nginx opens.

Step 2: Connect to your instance

Version SSH user
Feldera 0.378.0 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services and the listening set

systemctl is-active docker cloudimg-feldera-firewall feldera-metadata-names feldera-firstboot feldera-postgres feldera nginx
sudo test -f /var/lib/cloudimg/feldera-firstboot.done && echo FIRSTBOOT_DONE

active
active
active
active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network addresses; Docker publishes the pipeline-manager to 127.0.0.1:8080 and nothing else:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort
sudo docker port feldera-manager
sudo docker port feldera-postgres | wc -l

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:8080 users:(("docker-proxy"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))
8080/tcp -> 127.0.0.1:8080
0

Step 4: Get the console URL, login, API key and certificate fingerprint

The credentials file is readable by root only. It holds the console URL, the admin user and password, the API key and the SHA-256 fingerprint of this instance's certificate:

sudo cat /root/feldera-credentials.txt

FELDERA_URL=https://<public-ip>/
FELDERA_HOST=<public-ip>
FELDERA_UI_USER=admin
FELDERA_UI_PASSWORD=<FELDERA_UI_PASSWORD>
FELDERA_API_KEY=apikey:<your-api-key>
FELDERA_TLS_CERT_SHA256=<the certificate's SHA-256 fingerprint, AA:BB:...>
FELDERA_TLS_CERT_FILE=/etc/cloudimg/feldera-ui.pem

The file also carries comment lines with the same security note and the commands used in this guide.

The URL, the user and the fingerprint (without the password or the key):

sudo grep -E '^FELDERA_(URL|UI_USER|TLS_CERT_SHA256)=' /root/feldera-credentials.txt

FELDERA_URL=https://44.204.156.48/
FELDERA_UI_USER=admin
FELDERA_TLS_CERT_SHA256=6F:7D:44:91:92:C2:2F:86:7E:2E:A3:49:FA:5F:58:F6:F1:AD:32:D1:D1:2A:B0:38:5C:2E:B2:95:40:CA:B9:16

The host is the instance's public IP address from the EC2 instance metadata (the private address when the instance has no public one). The certificate also names the public DNS name, the private IP address and the private DNS name. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it (unless you installed your own, Step 14).

The certificate's names and constraints, read from the instance:

openssl x509 -in /etc/cloudimg/feldera-ui.pem -noout -ext subjectAltName,basicConstraints,keyUsage,extendedKeyUsage

X509v3 Subject Alternative Name: 
    IP Address:44.204.156.48, DNS:ec2-44-204-156-48.compute-1.amazonaws.com, IP Address:172.31.89.197, DNS:ip-172-31-89-197.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage: 
    TLS Web Server Authentication

Step 5: Open the Web Console

Browse to the FELDERA_URL from Step 4. The certificate is self signed and unique to this instance, so the browser warns about it the first time. Check the certificate before you continue: open the browser's certificate viewer for the page and compare its SHA-256 fingerprint with FELDERA_TLS_CERT_SHA256 from the credentials file. Continue only if they match, and only for this site. Never add this certificate to an operating-system or browser trust store; for a certificate browsers trust without a warning, install a CA-issued one (Step 14).

The browser then asks for a user name and password: sign in as admin with the password from the credentials file. To print just the password on the instance:

sudo awk -F= '$1=="FELDERA_UI_PASSWORD" {print $2}' /root/feldera-credentials.txt

The home page lists your pipelines and the bundled use cases and tutorials (screenshot at the top of this guide). New Pipeline opens the SQL editor: write tables and views, press Start to compile the program to a native pipeline and run it, and use the Runtime, Ad-Hoc Queries, Change Stream, Health and Logs tabs under the editor to watch it:

The orders pipeline from Step 7 open in the Feldera SQL editor, compiled and running

An ad-hoc SQL query in the console returning the customer_totals view of the running pipeline

Step 6: The REST API

Everything the console does goes through Feldera's REST API (/v0/..., described at /v0/... in the Feldera API reference). Through nginx it accepts either the console login or the API key:

  • /root/feldera-api.curl holds the API key as a curl header line, so the key never appears on a command line;
  • /root/feldera-curl.conf holds the console login the same way.

On the instance the examples pin the instance's public name to 127.0.0.1 with --resolve (so they work whatever the security group allows) and check the server against this instance's certificate:

H=$(sudo sed -n 's/^FELDERA_HOST=//p' /root/feldera-credentials.txt)
curl -s --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem https://$H/healthz
sudo curl -s -K /root/feldera-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem https://$H/v0/config \
  | python3 -c 'import json,sys; c=json.load(sys.stdin); print(c["version"], c["edition"])'
curl -s -o /dev/null -w 'without credentials: %{http_code}\n' --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem https://$H/v0/pipelines
curl -s -o /dev/null -w 'direct to 127.0.0.1:8080 as ubuntu: %{http_code}\n' -m 5 http://127.0.0.1:8080/v0/pipelines || true

ok
0.378.0 Open source
without credentials: 401
direct to 127.0.0.1:8080 as ubuntu: 000

127.0.0.1:8080 itself has no login (Feldera's own), so the instance refuses connections to it from every local user except root and nginx.

From your workstation, copy the certificate (Step 4), check its fingerprint and keep the API key in a file only you can read, then call the API the same way:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/cloudimg/feldera-ui.pem feldera.pem
openssl x509 -in feldera.pem -noout -fingerprint -sha256      # must match FELDERA_TLS_CERT_SHA256
( umask 077; printf 'header = "Authorization: Bearer %s"\n' "<your-api-key>" > feldera-api.curl )
curl -K feldera-api.curl --cacert feldera.pem https://<public-ip>/v0/pipelines

Requests made with the console login (rather than the API key) to the per-pipeline read routes, /v0/metrics and /v0/cluster must carry the browser's same-origin signal; such scripts add -H 'Sec-Fetch-Site: same-origin'.

Step 7: Create and run a SQL pipeline

This pipeline keeps a running total of orders per customer. Create it through the API (the console's New Pipeline does the same):

mkdir -p ~/feldera-demo && cd ~/feldera-demo
cat > orders.sql <<'SQL'
CREATE TABLE orders (id BIGINT NOT NULL PRIMARY KEY, customer VARCHAR NOT NULL, amount DECIMAL(10,2) NOT NULL)
  WITH ('materialized' = 'true');
CREATE MATERIALIZED VIEW customer_totals AS
  SELECT customer, COUNT(*) AS orders, SUM(amount) AS total FROM orders GROUP BY customer;
SQL
python3 -c 'import json; print(json.dumps({"name": "orders", "program_code": open("orders.sql").read(), "runtime_config": {"workers": 2}}))' > pipeline.json
H=$(sudo sed -n 's/^FELDERA_HOST=//p' /root/feldera-credentials.txt)
sudo curl -s -K /root/feldera-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem \
  -X PUT -H 'Content-Type: application/json' --data-binary @pipeline.json https://$H/v0/pipelines/orders \
  | python3 -c 'import json,sys; p=json.load(sys.stdin); print(p["name"], p["program_status"])'

orders Pending

Feldera compiles the SQL to Rust and the Rust to a native binary in the background: about a minute for a small program. On a new instance the first compile takes a few minutes longer, because the instance's disk loads its blocks from the AMI snapshot on first read (a first-boot service, feldera-prewarm, reads Feldera's compiler files in the background during the first minutes to speed this up). Wait for the compile, then start the pipeline:

H=$(sudo sed -n 's/^FELDERA_HOST=//p' /root/feldera-credentials.txt)
api() { sudo curl -s -K /root/feldera-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem "$@"; }
for i in $(seq 1 180); do S=$(api https://$H/v0/pipelines/orders | python3 -c 'import json,sys; print(json.load(sys.stdin)["program_status"])'); [ "$S" = Success ] && break; sleep 5; done
echo "program: $S"

program: Success

H=$(sudo sed -n 's/^FELDERA_HOST=//p' /root/feldera-credentials.txt)
api() { sudo curl -s -K /root/feldera-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem "$@"; }
api -X POST -o /dev/null -w 'start: %{http_code}\n' https://$H/v0/pipelines/orders/start
for i in $(seq 1 60); do D=$(api https://$H/v0/pipelines/orders | python3 -c 'import json,sys; print(json.load(sys.stdin)["deployment_status"])'); [ "$D" = Running ] && break; sleep 2; done
echo "pipeline: $D"

start: 202
pipeline: Running

Push rows in over HTTP (JSON, CSV, Avro and Parquet are supported), then query the view. Upserts and deletes update the totals incrementally:

cd ~/feldera-demo
H=$(sudo sed -n 's/^FELDERA_HOST=//p' /root/feldera-credentials.txt)
api() { sudo curl -s -K /root/feldera-api.curl --resolve "$H:443:127.0.0.1" --cacert /etc/cloudimg/feldera-ui.pem "$@"; }
printf '%s\n' '{"insert": {"id": 1, "customer": "acme", "amount": 120.50}}' '{"insert": {"id": 2, "customer": "acme", "amount": 79.50}}' \
  '{"insert": {"id": 3, "customer": "globex", "amount": 42.00}}' > batch.json
api -X POST -H 'Content-Type: application/json' --data-binary @batch.json -o /dev/null -w 'ingress: %{http_code}\n' \
  "https://$H/v0/pipelines/orders/ingress/orders?format=json&update_format=insert_delete"
printf '%s\n' '{"delete": {"id": 2, "customer": "acme", "amount": 79.50}}' > delete.json
api -X POST -H 'Content-Type: application/json' --data-binary @delete.json -o /dev/null -w 'delete: %{http_code}\n' \
  "https://$H/v0/pipelines/orders/ingress/orders?format=json&update_format=insert_delete"
sleep 2
api -G --data-urlencode 'sql=SELECT customer, orders, total FROM customer_totals ORDER BY customer' --data-urlencode 'format=text' \
  https://$H/v0/pipelines/orders/query; echo

ingress: 200
delete: 200
+----------+--------+--------+
| customer | orders | total  |
+----------+--------+--------+
| acme     | 1      | 120.50 |
| globex   | 1      | 42.00  |
+----------+--------+--------+

To follow a view's changes as they happen, POST /v0/pipelines/orders/egress/customer_totals?format=json streams every insert and delete; the console's Change Stream tab shows the same.

Step 8: The fda CLI on the instance

fda is Feldera's command-line client (the same release, copied from the Feldera image). It reads the API key from a file with --oidc-token-file (it sends the file's contents as the bearer token) and checks the server against this instance's certificate with --tls-cert, so no secret appears on the command line:

F() { sudo fda --host https://127.0.0.1 --tls-cert /etc/cloudimg/feldera-ui.pem --oidc-token-file /root/feldera-api-key "$@"; }
F --version
F pipelines
F query orders "SELECT customer, total FROM customer_totals ORDER BY customer"

fda 0.378.0
╭────────┬─────────╮
│ name   │ status  │
├────────┼─────────┤
│ orders │ Running │
╰────────┴─────────╯
+----------+--------+
| customer | total  |
+----------+--------+
| acme     | 120.50 |
| globex   | 42.00  |
+----------+--------+

fda can also create pipelines from SQL files (fda create <name> program.sql), start and stop them, stream changes (fda listen), tail logs and open an interactive SQL shell (fda shell <name>). From your workstation, download fda for your platform from the Feldera 0.378.0 release and point it at https://<public-ip> with the same three options.

Step 9: The Python SDK

The Feldera Python SDK (pip install feldera==0.378.0) talks to the same API. From your workstation, with the certificate copied as in Step 6 and the API key in a file only you can read:

python3 -m venv feldera-venv && . feldera-venv/bin/activate && pip install feldera==0.378.0
( umask 077; printf '%s\n' "<your-api-key>" > feldera-api-key )

from feldera import FelderaClient, PipelineBuilder

client = FelderaClient("https://<public-ip>",
                       api_key=open("feldera-api-key").read().strip(),   # the key never appears in your code
                       requests_verify="feldera.pem")                   # trust this instance's certificate only
print([p.name for p in client.pipelines()])

sql = """
CREATE TABLE readings (sensor VARCHAR NOT NULL, value DOUBLE NOT NULL) WITH ('materialized' = 'true');
CREATE MATERIALIZED VIEW sensor_avg AS SELECT sensor, AVG(value) AS avg_value FROM readings GROUP BY sensor;
"""
pipeline = PipelineBuilder(client, name="sensors", sql=sql).create_or_replace()
pipeline.start()
pipeline.input_json("readings", [{"sensor": "s1", "value": 20.5}, {"sensor": "s1", "value": 21.5}])
print(list(pipeline.query("SELECT * FROM sensor_avg")))
pipeline.stop(force=True)

requests_verify pins this instance's certificate for the SDK only; never point it at a system trust store.

Step 10: Connectors and the security note

Tables and views read and write external systems through connectors declared in SQL. For example, a table fed from a Kafka topic and a view written back to another one:

CREATE TABLE clicks (user_id BIGINT, url VARCHAR, ts TIMESTAMP) WITH (
  'connectors' = '[{"transport": {"name": "kafka_input", "config": {"bootstrap.servers": "<broker>:9092", "topics": ["clicks"]}},
                    "format": {"name": "json", "config": {"update_format": "raw"}}}]');
CREATE MATERIALIZED VIEW clicks_per_user WITH (
  'connectors' = '[{"transport": {"name": "kafka_output", "config": {"bootstrap.servers": "<broker>:9092", "topic": "clicks-per-user"}},
                    "format": {"name": "json"}}]')
AS SELECT user_id, COUNT(*) AS clicks FROM clicks GROUP BY user_id;

Other connectors cover Debezium and PostgreSQL CDC, Delta Lake and Iceberg, S3 and other object stores, files, HTTP URLs, Redis, NATS and more (see the connector reference). For a TLS Kafka broker with a self-signed certificate, pin that certificate in the connector's own ssl.ca.pem setting (the PEM text), never in a system trust store.

Security note. A pipeline is native code compiled from your SQL, Rust UDFs are arbitrary Rust, and connectors open connections to any address the instance can reach. Anyone with the console login or the API key can execute code on this instance and make it connect to systems in your network. Share both only with people you would give a shell to. The image confines that code (Step 13), but it is still code you run.

Rust UDFs: declare CREATE FUNCTION <name>(...) RETURNS ...; in the SQL and put the implementation in the pipeline's UDF tab (udf_rust in the API, plus crate dependencies in udf_toml); see Rust UDFs. Feldera fetches crates your UDFs add from crates.io at compile time; the image's precompiled dependencies cover Feldera's own.

Step 11: Where pipelines and data live

The pipeline-manager keeps pipeline definitions, program versions and runtime state in PostgreSQL, and compiled programs, checkpoints and pipeline storage in its own state volume. Both are Docker volumes that start empty on every new instance (the state volume is prepared from the image's precompiled Rust dependency cache at first start):

sudo docker volume ls --format '{{.Name}}' | sort
sudo docker inspect --format '{{range .Mounts}}{{.Name}} -> {{.Destination}}{{"\n"}}{{end}}' feldera-manager | sed '/^$/d'
sudo du -sh /var/lib/docker/volumes/feldera-pgdata /var/lib/docker/volumes/feldera-home | awk '{print $2, $1}' | sed 's#/var/lib/docker/volumes/##'

feldera-home
feldera-pgdata
feldera-pgsock
feldera-pgsock -> /run/postgresql
feldera-home -> /home/ubuntu/.feldera
feldera-pgdata 48M
feldera-home 6.0G

For large pipeline state, grow the root volume (EBS Modify volume, then sudo growpart /dev/nvme0n1 1 && sudo resize2fs /dev/nvme0n1p1) or move Docker's data to a dedicated EBS volume. PostgreSQL has no network: to inspect it, use sudo docker exec -it feldera-postgres psql -U postgres -d feldera.

Step 12: Security checks

feldera-selfcheck re-proves the whole appliance with this instance's credentials: the services and container hardening, the exact listening set, the network guards, telemetry off, the HTTPS front door (login and API key accepted; anonymous, wrong-password and wrong-key requests refused; unknown host names dropped; cross-origin requests refused; compressed request bodies refused with 415; ; and %3B path parameters refused with 400), the loopback owner rule (only root and nginx reach the pipeline-manager on 127.0.0.1:8080; other local users, including ubuntu, are refused on 127.0.0.1, ::1 and the container bridge), the certificate, and a real pipeline with a Rust UDF that probes its own sandbox from inside: the instance metadata service and the host's own services time out while a loopback control connects, and no secret file is visible. It takes about two minutes and deletes its pipeline afterwards:

sudo feldera-selfcheck

SELFCHECK OK: Feldera 0.378.0 open-source (telemetry keys empty); services active; containers uid 1000 / 70, read-only, no capabilities, no Docker socket; listening 22, 80 (301), 443, 127.0.0.1:8080 only; login + API key accepted, anonymous / wrong password / wrong key 401, unknown Host dropped, cross-origin 403, compressed request bodies 415, path parameters 400; loopback API owner-fenced (root + nginx only; nobody/ubuntu refused on 127.0.0.1, ::1 and the bridge); pipeline cloudimg_selfcheck_3418 compiled a Rust UDF and ran; metadata service TIMEOUT and host services TIMEOUT from inside the pipeline (loopback control OPEN), no secret file visible; certificate 6F:7D:44:91:92:C2:2F:86...

Step 13: The container sandbox and the metadata block

The pipeline-manager and PostgreSQL containers run unprivileged, with no capabilities, no privilege escalation and a read-only root file system:

for c in feldera-manager feldera-postgres; do sudo docker inspect --format "$c: user={{.Config.User}} readonly={{.HostConfig.ReadonlyRootfs}} privileged={{.HostConfig.Privileged}} network={{.HostConfig.NetworkMode}} capdrop={{.HostConfig.CapDrop}} secopt={{.HostConfig.SecurityOpt}}" $c; done

feldera-manager: user=1000:1000 readonly=true privileged=false network=feldera-net capdrop=[ALL] secopt=[no-new-privileges]
feldera-postgres: user=70:70 readonly=true privileged=false network=none capdrop=[ALL] secopt=[no-new-privileges]

Containers may not reach the instance metadata service (169.254.169.254, and 168.63.129.16) or open connections to this instance's own services; the host services that do not need the metadata service are denied it by systemd IPAddressDeny rules. Only feldera-metadata-names, which reads the instance's names for the certificate, may reach it:

sudo iptables -S DOCKER-USER | grep DROP
sudo iptables -S CLOUDIMG-FELDERA-IN
for u in feldera feldera-postgres nginx docker feldera-metadata-names; do printf '%s: deny=%s allow=%s\n' $u "$(systemctl show -p IPAddressDeny --value $u.service | tr ' ' '\n' | sort | tr '\n' ' ' | sed 's/ $//')" "$(systemctl show -p IPAddressAllow --value $u.service)"; done

-A DOCKER-USER -d 168.63.129.16/32 -j DROP
-A DOCKER-USER -d 169.254.169.254/32 -j DROP
-N CLOUDIMG-FELDERA-IN
-A CLOUDIMG-FELDERA-IN -i br-feldera -m conntrack --ctstate NEW -j DROP
feldera: deny=168.63.129.16/32 169.254.169.254/32 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 fd00:ec2::254/128 allow=
feldera-postgres: deny=168.63.129.16/32 169.254.169.254/32 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 fd00:ec2::254/128 allow=
nginx: deny=168.63.129.16/32 169.254.169.254/32 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 fd00:ec2::254/128 allow=
docker: deny=168.63.129.16/32 169.254.169.254/32 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 fd00:ec2::254/128 allow=
feldera-metadata-names: deny=0.0.0.0/0 ::/0 allow=fd00:ec2::254/128 169.254.169.254/32

Pipelines can still reach your sources and sinks on the network and the internet; only the metadata addresses and the host itself are blocked. Connectors that would normally use the instance's IAM role (S3, Delta Lake on S3, MSK IAM) need explicit credentials in their configuration.

Step 14: Your own DNS name and a CA-issued certificate

To serve the console under your own DNS name, point the name at the instance (an Elastic IP is recommended) and set it. This writes the name into nginx and issues a new self-signed certificate that covers it:

sudo feldera-set-hostname feldera.example.com

To install a certificate from your CA (or Let's Encrypt), copy the full chain (server certificate first) and the unencrypted private key to the instance, then:

sudo feldera-install-cert /path/to/fullchain.pem /path/to/privkey.pem feldera.example.com

The boot-time renewal never replaces a certificate installed this way. sudo feldera-set-hostname <name> goes back to a self-signed certificate. Both commands update the URL and fingerprint in /root/feldera-credentials.txt.

Step 15: Change the console password or the API key

The password is stored as a bcrypt hash in /etc/nginx/feldera-htpasswd. To set a new one without putting it on a command line, let htpasswd prompt for it, then update the curl config and the credentials file:

sudo htpasswd -B -C 10 /etc/nginx/feldera-htpasswd admin
sudo nano /root/feldera-curl.conf /root/feldera-credentials.txt
sudo systemctl reload nginx

To rotate the API key, write a new 64-character hex key (for example from openssl rand -hex 32) after apikey: in /etc/nginx/feldera-api-key.conf, /root/feldera-api.curl, /root/feldera-api-key and the credentials file with an editor, then sudo systemctl reload nginx.

Step 16: Logs, versions and file locations

sudo docker exec feldera-manager /home/ubuntu/feldera/build/pipeline-manager --version 2>/dev/null | tail -1
sudo docker exec feldera-postgres postgres --version
journalctl -u feldera -b --no-pager -o cat | grep -E '^Version:|starting service' | sed -E 's/^\[manager\] [0-9TZ:.-]+ +//' | awk 'n < 2 {print; n++}'

Pipeline manager CLI 0.378.0
postgres (PostgreSQL) 16.15
INFO actix_server::server:  starting service: "actix-web-service-0.0.0.0:8085", workers: 2, listening on: 0.0.0.0:8085
INFO actix_server::server:  starting service: "actix-web-service-0.0.0.0:8089", workers: 2, listening on: 0.0.0.0:8089

The two 0.0.0.0 listeners are inside the pipeline-manager container's own network namespace (its compiler and runner services). On the instance itself only 127.0.0.1:8080 is published, and only root and nginx may connect to it (Step 3).

Path Contents
/etc/systemd/system/feldera.service, feldera-postgres.service the two containers (images pinned by digest)
/etc/feldera/manager.env, /etc/feldera/postgres/ pipeline-manager environment (telemetry off), PostgreSQL access rules (no secrets)
/etc/feldera/secrets/postgres-password this instance's PostgreSQL superuser password (only the PostgreSQL container reads it)
Docker volumes feldera-pgdata, feldera-home metadata database; compiled programs and pipeline storage
/etc/nginx/sites-available/cloudimg-feldera the nginx front door
/root/feldera-credentials.txt, /root/feldera-curl.conf, /root/feldera-api.curl, /root/feldera-api-key login, API key, curl configs (root only)
/usr/share/doc/feldera Feldera's licence and the image provenance

The pipeline-manager logs to the journal (journalctl -u feldera), PostgreSQL to journalctl -u feldera-postgres, nginx to /var/log/nginx/. A pipeline's own log is on its Logs tab and at /v0/pipelines/<name>/logs.

Step 17: Backup and maintenance

  • Feldera can checkpoint pipeline state and resume from it. Back up the PostgreSQL metadata with sudo docker exec feldera-postgres pg_dump -U postgres feldera > feldera-metadata.sql, and keep your SQL programs and UDFs in version control: pipelines are re-created from them.
  • Ubuntu security updates install automatically (unattended-upgrades). The Feldera and PostgreSQL images are pinned by digest; to move to a newer Feldera release, launch the newer version from the listing and re-create your pipelines there.
  • Size the instance for your pipelines: compiling uses CPU and memory, and each running pipeline keeps its state in memory and storage. m5.large suits development and small pipelines; use larger instances for large state or high input rates.

Step 18: Security recommendations

  • Limit TCP 443 and 22 in the security group to your admin network. The console login and the API key grant code execution on the instance (Step 10): share them only with people you would give a shell to.
  • Serve the console under your own DNS name with a certificate from a trusted CA (Step 14). Until then, check the self-signed certificate's fingerprint (Step 4) before you sign in, and pin it in clients (--cacert, --tls-cert, requests_verify).
  • Keep the pipeline-manager on the loopback address: do not add other published ports to feldera.service or remove the network guards. The services refuse to start without them.
  • Give connectors credentials of their own, scoped to the data they need. The instance's IAM role is deliberately unreachable from pipelines.
  • HTTP Basic authentication has no sign-out: close the browser to end a session on a shared computer.

Support and licensing

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk. The Feldera open-source edition is licensed under the MIT License (the licence text is on the instance in /usr/share/doc/feldera/LICENSE; Feldera Enterprise features are not part of this image), PostgreSQL under the PostgreSQL License, nginx under the 2-clause BSD licence and Docker Engine under the Apache License 2.0. Feldera is a trademark of Feldera, Inc. This image is provided by cloudimg and is not affiliated with or endorsed by Feldera, Inc.