Prefect on AWS User Guide

Overview

This image runs Prefect, the fast, open source Python framework for building, running and observing resilient data workflows. You turn ordinary Python functions into orchestrated flows with retries, scheduling, caching, concurrency limits and a rich observability UI, with no boilerplate.

Prefect 3.7.5 is installed into a dedicated Python virtual environment under /opt/prefect and run by an unprivileged prefect system account under a systemd service that starts the server on boot and restarts it on failure. The server data directory, PREFECT_HOME, holds the embedded SQLite metadata database, the server logs and storage. It lives at /var/lib/prefect, a dedicated, independently resizable EBS data volume that survives an operating system disk reimage. The single node server uses the embedded SQLite database, so no external database is required.

Prefect ships with no built in authentication in the open source server, so the server binds to the loopback interface only (127.0.0.1:4200) and is never exposed directly. An nginx reverse proxy publishes the server API and the web UI on port 80 behind HTTP Basic authentication, forwarding the WebSocket upgrade and connection headers that the Prefect UI needs for its live event and log streams. The admin password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/prefect-credentials.txt with mode 0600 so that only the root user can read it. The /api/health endpoint is left unauthenticated through the proxy so external health checks and load balancers can reach it.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the Prefect server and UI
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Prefect. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.medium or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the Prefect server and UI. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a short time after the instance state becomes Running and the status checks pass; this is when the per instance admin password is generated and the server runs its first time database migrations.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Prefect Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.medium \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=prefect}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

Step 4: Retrieve the Admin Password

The Prefect UI admin password is unique to your instance and was generated on first boot. Read it as root:

sudo cat /root/prefect-credentials.txt

The file lists the UI URL, the admin user (admin) and the generated password, along with a ready to paste PREFECT_API_URL for connecting workers and clients. Keep this password somewhere safe.

Step 5: Sign In to the Prefect UI

The Prefect server and UI are served on port 80 by nginx behind HTTP Basic authentication. In a browser, go to:

http://<instance-public-ip>/

You are prompted for credentials. Sign in as admin with the password from the credentials file. The dashboard gives you a live overview of your flow runs, task runs and work pools, updated in real time over the proxied WebSocket connection.

Open Runs in the left navigation to see every flow run and task run, filter by state, flow, deployment, work pool or tag, and drill into the logs and timeline of an individual run.

Open Work Pools to create and manage the pools that workers poll for scheduled deployment runs, so you can route work to the infrastructure you choose.

Step 6: Confirm Prefect Is Running

Over SSH, confirm the server and the nginx proxy are active, check the API health endpoint, and confirm the listening ports:

sudo systemctl is-active prefect nginx
curl -s http://127.0.0.1:4200/api/health
sudo ss -tlnp | grep -E ':(80|4200) '

You should see both services reported as active, the health endpoint returning true, the server listening on 127.0.0.1:4200 (loopback only), and nginx listening on port 80.

Confirm the installed Prefect version with the bundled virtual environment:

/opt/prefect/venv/bin/prefect version

Step 7: Write and Run Your First Flow

Prefect turns plain Python functions into orchestrated flows and tasks. Create a file etl.py on the instance and run it with the bundled virtual environment so the run is recorded against the local server. The following example defines a flow with three tasks and runs it three times:

cat > /tmp/etl.py <<'PYEOF'
from prefect import flow, task

@task
def extract():
    return [1, 2, 3, 4, 5]

@task
def transform(data):
    return [x * 10 for x in data]

@task
def load(data):
    return sum(data)

@flow(name="example-etl-pipeline")
def etl_pipeline():
    raw = extract()
    clean = transform(raw)
    return load(clean)

if __name__ == "__main__":
    print(etl_pipeline())
PYEOF
PREFECT_API_URL=http://127.0.0.1:4200/api /opt/prefect/venv/bin/python /tmp/etl.py

Each run appears immediately in the Runs view of the UI with its state, logs and timeline. Tasks gain retries, caching and concurrency limits by adding the matching arguments to the @task decorator, with no extra infrastructure.

Step 8: Connect Workers and Clients

To submit work from another machine or to run a worker, point the Prefect client at this server through the authenticating proxy. The credentials file prints the exact URL; it has the form below, where <password> is the generated admin password and <instance-public-ip> is this instance's public address:

prefect config set PREFECT_API_URL=http://admin:<password>@<instance-public-ip>/api

A worker started against a work pool then polls the server for scheduled deployment runs and executes them on its own infrastructure. Create a work pool from the Work Pools view in the UI, or with the bundled CLI on the instance, and start a worker against it from wherever your code should run.

Step 9: Install Extra Python Packages

Your flows often need extra libraries. Install them into the bundled virtual environment so they are importable by the server process and by flows run with /opt/prefect/venv/bin/python. The virtual environment lives at /opt/prefect/venv and is owned by the prefect service account. Confirm you are using the bundled pip with:

/opt/prefect/venv/bin/pip --version

Then install whatever your flows need, for example sudo /opt/prefect/venv/bin/pip install pandas requests. Because the virtual environment lives on the dedicated data volume, your installed packages persist independently of the operating system disk.

Step 10: The Data Volume

The server data directory, PREFECT_HOME, lives on a dedicated EBS volume mounted at /var/lib/prefect. This keeps the embedded SQLite metadata database, the server logs and the storage off the operating system disk and lets you resize or snapshot it independently. Confirm the mount with:

df -h /var/lib/prefect

To grow the data store, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device. The directory is owned by the prefect service account.

Step 11: Enable HTTPS

The Prefect server and UI are served over plain HTTP on port 80 by nginx. For production use, place them behind TLS. Obtain a certificate for your domain (for example with a managed certificate on an Application Load Balancer in front of the instance, or with Certbot installed on the instance for your-domain), then configure nginx to listen on 443 with your certificate and proxy to 127.0.0.1:4200 exactly as the bundled site does for port 80, keeping the HTTP Basic authentication and the WebSocket upgrade headers in place. Restrict the security group so ports 80 and 443 are reachable only from the networks that need the server and UI.

Step 12: Backup and Maintenance

Back up the server by snapshotting the /var/lib/prefect EBS volume, which captures the embedded SQLite metadata database, logs and storage. Apply operating system security updates with sudo apt-get update && sudo apt-get upgrade and reboot when a new kernel is installed; the Prefect server and nginx start automatically on boot. The server runs its database migrations automatically when a new Prefect version introduces a schema change.

Support

This image is published and supported by cloudimg. Support covers deployment, connecting workers and deployments, work pools, scheduling, retries and caching, TLS, authentication and performance tuning. Contact cloudimg through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.