Streaming & Messaging Azure

Feldera on Ubuntu 24.04 on Azure User Guide

| Product: Feldera on Ubuntu 24.04 LTS on Azure

Overview

Feldera is an open source incremental compute engine for continuous analytics over data in motion. You write ordinary SQL, a table and a view over it, and Feldera compiles the program into a running pipeline that keeps the view's results correct as rows are inserted, updated and deleted, doing only the work each change requires instead of recomputing from scratch. The cloudimg image ships the MIT licensed Feldera open source edition, the all in one pipeline-manager (REST API, Web Console, SQL to Rust compiler and local runner) pointed at a bundled PostgreSQL for metadata, run the officially supported way as an upstream container orchestrated by Docker Compose under systemd and fronted by nginx. Both images are pinned by digest and captured into the VM, so your instance starts without pulling anything.

Feldera's open source build has no built in authentication, so cloudimg fronts the Console and REST API with nginx HTTP Basic Auth using a unique password generated for each VM on first boot, before the port is reachable. The pipeline manager and the bundled PostgreSQL are bound to the loopback interface only. Backed by 24/7 cloudimg support.

Feldera is a trademark of Feldera, Inc. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by Feldera, Inc. It ships only the MIT licensed Feldera open source edition.

The Feldera Web Console pipelines list showing a running streaming SQL pipeline

What is included:

  • Feldera open source edition 0.322.0 (the MIT licensed incremental view maintenance and streaming SQL engine, REST API, Web Console, SQL to Rust compiler and local runner), pinned by image digest
  • A bundled PostgreSQL 16 database for pipeline metadata, pinned by image digest, reachable only inside a private Docker network and never published to a host port
  • Docker Engine (Docker CE) with the pipeline manager published to the loopback interface only, fronted by nginx on port 80
  • feldera.service, feldera-firstboot.service and nginx.service as systemd units, enabled and active on boot
  • A unique Web Console password (nginx HTTP Basic Auth, bcrypt) and a unique PostgreSQL password generated per VM on first boot, never baked into the image
  • An 8 GB swap file provisioned on first boot so the runtime SQL to Rust pipeline compiler has headroom on smaller instances
  • A clean, freshly initialised metadata database on first boot: no default login, no shipped secret, no prior pipelines
  • 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region. Feldera compiles each SQL pipeline to native code at runtime, which is CPU and memory intensive: Standard_B2s (2 vCPU / 4 GiB RAM) runs the image and compiles a pipeline in a couple of minutes, but for production pipelines choose a larger size such as Standard_D4s_v3. NSG inbound: allow 22/tcp from your management network and 80/tcp (and 443/tcp once you add TLS) for the Console. The Console serves plain HTTP on port 80; for production, put it behind TLS with your own domain (see the final section).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Feldera by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Then Review + create and Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name feldera \
  --image <marketplace-image-urn> \
  --size Standard_D4s_v3 \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name feldera --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

On first boot the image provisions an 8 GB swap file, generates a unique Console password and database password, initialises a clean PostgreSQL metadata database and starts the stack. Confirm the systemd services are active:

systemctl is-active docker feldera nginx

All three services report active. The public liveness endpoint returns 200 with no credentials, while the pipeline manager and the bundled PostgreSQL stay bound to the loopback interface only:

curl -s -o /dev/null -w 'health: HTTP %{http_code}\n' http://127.0.0.1/healthz

The stack runs as two containers under one Docker Compose project (the pipeline manager and the bundled PostgreSQL):

The Feldera pipeline manager and bundled PostgreSQL containers running healthy under systemd, with a public health check

Step 5 - Secure by default: the Console password gates everything

Feldera's open source build has no built in authentication, so cloudimg fronts the Console and REST API with nginx HTTP Basic Auth. Any request without the exact per-VM credential is rejected with 401. Confirm it yourself; this request sends no credential:

curl -s -o /dev/null -w 'no credential: HTTP %{http_code}\n' http://127.0.0.1/v0/pipelines

It returns HTTP 401. Guessable defaults such as feldera:feldera are rejected too. Your unique credential authenticates and returns 200; the block below reads the password from the notes file (so it never appears on screen):

P=$(sudo grep '^FELDERA_UI_PASSWORD=' /root/feldera-notes.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'authorized: HTTP %{http_code}\n' -u "feldera:$P" http://127.0.0.1/v0/pipelines

The REST API rejects no credential and a guessable default with 401 and accepts only the per-VM credential with 200

Your unique Console user and password are written to /root/feldera-notes.txt, readable only by root:

sudo stat -c '%a %U:%G' /root/feldera-notes.txt

It reports 600 root:root. View the notes (which include the Console user, the Console password and the URL) with:

sudo cat /root/feldera-notes.txt

The per-VM access notes, with the Console password and database password masked

Step 6 - Open the Feldera Web Console

Browse to http://<vm-public-ip>/. Your browser prompts for the Basic Auth credential; enter the user feldera and the password from /root/feldera-notes.txt. The Console opens on the pipelines list, showing every pipeline and its status.

The Feldera Web Console pipelines list with a running pipeline

Step 7 - Build a streaming SQL pipeline

Create a new pipeline and write standard SQL: a CREATE TABLE for your input and one or more CREATE MATERIALIZED VIEW statements for the results you want kept up to date. When you start the pipeline, Feldera compiles the SQL to a native binary and runs it. The example below keeps a running revenue total per category, and a per hour revenue window, updated incrementally as transactions arrive.

The Feldera SQL editor showing a streaming pipeline that maintains category and hourly revenue views

You can also drive everything over the REST API. This creates and starts a trivial pipeline, then you can start it and feed it rows (the compile takes a moment the first time):

P=$(sudo grep '^FELDERA_UI_PASSWORD=' /root/feldera-notes.txt | cut -d= -f2-)
curl -s -u "feldera:$P" -X PUT http://127.0.0.1/v0/pipelines/demo \
  -H 'Content-Type: application/json' \
  -d '{"name":"demo","program_code":"CREATE TABLE t(x INT) WITH (\047materialized\047=\047true\047); CREATE MATERIALIZED VIEW v AS SELECT x, x*2 AS y FROM t;"}'
curl -s -u "feldera:$P" -X POST http://127.0.0.1/v0/pipelines/demo/start

Step 8 - Query and watch results incrementally

With a pipeline running, use the Ad-Hoc Queries tab to run SQL against the current state of any materialized view. Here a query returns the running totals per category:

An ad-hoc SQL query returning the running per category revenue totals from a materialized view

The Change Stream tab shows records being emitted as data flows through the pipeline. Select a table or view, then send more rows: each change appears as an Insert or Delete, so you can watch the incremental view maintenance happen in real time.

The Feldera Change Stream tab showing insert records flowing through the pipeline as new transactions arrive

To push data into a table over HTTP, POST newline delimited JSON to the ingress endpoint:

P=$(sudo grep '^FELDERA_UI_PASSWORD=' /root/feldera-notes.txt | cut -d= -f2-)
curl -s -u "feldera:$P" -X POST \
  'http://127.0.0.1/v0/pipelines/demo/ingress/t?force=true&format=json' \
  -H 'Content-Type: application/json' --data-binary '{"insert":{"x":21}}'

Step 9 - Rotate the Console password

The Console password is your access key. To set your own, rewrite the Basic Auth file and reload nginx:

sudo htpasswd -B /etc/nginx/feldera.htpasswd feldera
sudo systemctl reload nginx

Step 10 - Production: your own domain with TLS

For production, front the Console with your own domain and TLS. Point a DNS record at the VM public IP, then install a certificate with your preferred tool (for example Certbot with the nginx plugin) and reload nginx. Restrict inbound access in your NSG to 443/tcp (and 22/tcp from your management network) once TLS is in place, and choose a larger instance such as Standard_D4s_v3 for production pipelines. Keep the Console credential confidential; it grants full access to create and run pipelines and read all pipeline data.

Support

This image is supported 24/7 by cloudimg. For help with deployment or configuration, contact support through cloudimg.co.uk.