Prefect on Ubuntu 24.04 on Azure User Guide

Overview

Prefect is the modern Python workflow orchestration platform. Turn any Python function into an observable, schedulable workflow with retries, caching, concurrency limits and event-driven automation, and monitor every flow run, deployment and work pool from a rich web UI. It is the go-to tool for data engineers building and operating resilient data pipelines. The cloudimg image installs Prefect 3.7.5 in a dedicated Python virtualenv, runs the Prefect server as a systemd service bound to loopback behind an nginx reverse proxy on port 80 with HTTP Basic auth, persists orchestration state on a dedicated Azure data disk, and generates a unique login password on the first boot of every VM. Backed by 24/7 cloudimg support.

What is included:

• Prefect 3.7.5 in a Python virtualenv at /opt/prefect/venv
• The Prefect server (UI + API) on :80 via nginx, with HTTP Basic auth
• A per-VM admin password in a root-only file (Prefect Server has no built-in auth)
• A dedicated Azure data disk at /var/lib/prefect holding the SQLite state database (flow runs, deployments, config) - separate from the OS disk and re-provisioned with every VM
• prefect-server.service + nginx.service as systemd units, enabled and active
• 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet + subnet in the target region. Standard_B2ms (2 vCPU / 8 GiB RAM) is a good starting point; scale up for many concurrent flow runs. NSG inbound: allow 22/tcp from your management network and 80/tcp for the UI and API (front with TLS for public exposure - see Enabling HTTPS).

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Prefect by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name prefect \
  --image <marketplace-image-urn> \
  --size Standard_B2ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name prefect --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active prefect-server.service nginx.service

Both services report active. The Prefect server runs a database migration on first boot, then serves the UI and API.

Step 5 - Retrieve your password

The admin password is generated uniquely on the first boot of your VM and written to a root-only file:

sudo cat /root/prefect-credentials.txt

This file contains PREFECT_ADMIN_USER (admin), PREFECT_ADMIN_PASSWORD and the URLs. Store the password somewhere safe.

Step 6 - Check the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes:

curl -s http://localhost/health

It returns ok.

Step 7 - Open the Prefect UI

Browse to http://<vm-public-ip>/ and sign in as admin with the password from Step 5. The dashboard summarises recent flow-run activity; the Runs, Flows, Deployments and Work Pools pages let you drill into every pipeline.

The Runs page charts and lists every flow run with its state:

Open a run to see its task runs, timeline and logs:

Step 8 - Confirm the API from the command line

The Prefect API is available behind the same Basic auth. Confirm it is healthy and report how many flow runs it is tracking:

curl -s -u admin:<PREFECT_ADMIN_PASSWORD> http://localhost/api/health; echo
curl -s -u admin:<PREFECT_ADMIN_PASSWORD> -X POST -H 'Content-Type: application/json' -d '{}' http://localhost/api/flow_runs/count; echo

/api/health returns true, and the count endpoint returns the number of flow runs.

Step 9 - Point a client or worker at the server

On your laptop or a worker machine, install Prefect and point it at this server's API (the credentials go in the URL), then run a flow:

pip install prefect==3.7.5
prefect config set PREFECT_API_URL="http://admin:<password>@<vm-public-ip>/api"

# my_flow.py
from prefect import flow

@flow(log_prints=True)
def hello(name: str = "world"):
    print(f"Hello {name} from Prefect on Azure!")

if __name__ == "__main__":
    hello()

Run python my_flow.py and the run appears in the UI. Create a work pool and deploy flows to run them on a schedule with prefect deploy.

Step 10 - Confirm state lives on the dedicated disk

The Prefect home and its SQLite database are stored on the dedicated Azure data disk so they survive OS changes and can be resized independently:

findmnt /var/lib/prefect

The mount is backed by a separate Azure data disk captured into the image and re-provisioned on every VM.

Enabling HTTPS

The nginx reverse proxy terminates plain HTTP on port 80. For public exposure, put a certificate in front of it - add a DNS name for the VM and use the companion cloudimg nginx-ssl-certbot image as a TLS reverse proxy, or install certbot and extend the existing nginx site with a listen 443 ssl; server block. Keep the Prefect server bound to loopback so the only public surface is the authenticated, TLS-terminated proxy.

Maintenance

• Backups: snapshot the /var/lib/prefect data disk to back up flow runs and deployments.
• Database: the image uses SQLite, ideal for a single server; for high throughput, migrate PREFECT_HOME/the API to PostgreSQL.
• Upgrades: sudo /opt/prefect/venv/bin/pip install -U prefect then sudo systemctl restart prefect-server.
• Security patches: unattended-upgrades remains enabled so the OS continues to receive security updates automatically.

Support

cloudimg provides 24/7 expert support for this image. Contact support@cloudimg.co.uk.