Hatchet Distributed Task Queue and Workflow Orchestration on AWS User Guide

Overview

Hatchet is an open source distributed task queue and background job engine for durable workflow orchestration. You define workflows and tasks in code with your existing language SDK, and Hatchet runs them across a fleet of workers with automatic retries, concurrency and rate limits, fan out and scheduling, then shows you every run, worker, event and schedule in a modern web dashboard.

This cloudimg image runs Hatchet as the official all in one hatchet-lite container (engine, REST API and dashboard in one image), alongside a bundled PostgreSQL datastore and a RabbitMQ message broker, all orchestrated by docker compose under a single systemd service on a private container network. The engine, dashboard and API bind to the loopback interface and are fronted by nginx on port 443 with a per instance self-signed TLS certificate; PostgreSQL and RabbitMQ are never published to a host port. Docker Engine and its entire data root, including the pre pulled digest-pinned images and the database, broker and configuration volumes, live on a dedicated data volume mounted at /var/lib/docker, so your workflow state survives OS-disk changes and is resizable independently.

Security is enforced from first boot. The stock Hatchet compose seeds a well known default administrator and default secrets; this image never ships them. Before the port is reachable, a unique administrator, a PostgreSQL password, a RabbitMQ password, an encryption keyset and the TLS certificate are all generated for the individual instance, and open sign up and tenant creation are disabled so the instance cannot be seized. Backed by 24/7 cloudimg support.

What is included:

• Hatchet v0.94.10, the MIT licensed self hosted core, shipped as the official hatchet-lite image pinned by digest
• A bundled PostgreSQL and RabbitMQ, pinned by digest and reachable only inside a private container network
• Docker Engine with the Hatchet dashboard and API on the loopback interface behind nginx over TLS on port 443
• A unique administrator, database password, broker password and encryption keyset generated per instance on first boot
• Open sign up and tenant creation disabled, so the instance cannot be seized
• The Docker data root, workflow database and broker state on a dedicated data volume at /var/lib/docker
• An unauthenticated /api/live liveness endpoint for load-balancer health checks
• docker.service, hatchet.service and nginx.service as systemd units, enabled and active
• 24/7 cloudimg support

Prerequisites

An AWS account, an EC2 key pair in the target region, and a VPC with a public subnet. m5.large (2 vCPU / 8 GiB RAM) is the recommended instance type. Security group inbound: allow 22/tcp from your management network for SSH and 443/tcp for the web dashboard. To connect external workers over the engine gRPC port, also allow 7077/tcp from your worker network.

Step 1 - Launch the AMI

Subscribe to the listing in AWS Marketplace, then launch an instance from the AMI into your VPC with the security group described above. Choose the m5.large instance type and attach your EC2 key pair. The image keeps the pre pulled container images on the dedicated data volume, so the Hatchet stack comes up within seconds of first boot with no image re-pull.

Step 2 - Connect to your instance

Connect over SSH as the default login user for the operating system variant you launched. The login user differs per OS, so use the row for your variant:

ssh -i your-key.pem ubuntu@<public-ip>

Step 3 - Read your unique administrator credential

On first boot the instance generates a unique administrator email and password and writes them, along with the resolved dashboard URL, to a root-only file. Read it with sudo:

sudo cat /root/hatchet-credentials.txt

You will see the dashboard URL and the administrator email and password for this instance:

HATCHET_URL=https://<instance-public-ip>
HATCHET_ADMIN_EMAIL=admin@hatchet.local
HATCHET_ADMIN_PASSWORD=<unique-per-instance-password>
HATCHET_WORKER_GRPC=<instance-public-ip>:7077

No two instances share this credential, and the well known default administrator (admin@example.com / Admin123!!) is never created.

Step 4 - Confirm the stack and the security model from the command line

The three containers run under docker compose, managed by hatchet.service. Confirm they are up and healthy:

sudo docker compose -f /etc/hatchet/compose.yaml ps

SERVICE        IMAGE                                                  STATUS
hatchet-lite   ghcr.io/hatchet-dev/hatchet/hatchet-lite@sha256:...    Up 8 minutes
postgres       postgres@sha256:...                                    Up 9 minutes (healthy)
rabbitmq       rabbitmq@sha256:...                                    Up 9 minutes (healthy)

Confirm the systemd services are active:

systemctl is-active docker hatchet nginx

active
active
active

The engine, dashboard and API listen only on the loopback interface, and PostgreSQL and RabbitMQ are not published to any host port. Only nginx is exposed:

sudo ss -tlnp | grep -E ':443|:8888|:7077|:5432|:5672'

LISTEN  0.0.0.0:443          # nginx (TLS reverse proxy)
LISTEN  127.0.0.1:8888       # hatchet dashboard + API (loopback only)
LISTEN  127.0.0.1:7077       # hatchet worker gRPC (loopback only)

Note there is no 0.0.0.0:5432 (PostgreSQL) or 0.0.0.0:5672 (RabbitMQ) listener: both stay inside the private container network. The liveness endpoint answers without authentication, and the server metadata confirms open sign up and tenant creation are disabled:

curl -ks -o /dev/null -w '%{http_code}\n' https://127.0.0.1/api/live
curl -ks https://127.0.0.1/api/v1/meta | jq '{allowSignup, allowCreateTenant}'

200
{
  "allowSignup": false,
  "allowCreateTenant": false
}

Step 5 - Sign in to the dashboard

Open the dashboard in your browser at https://<instance-public-ip>/. The instance uses a per instance self-signed TLS certificate, so your browser will show a certificate warning on first visit; accept it to proceed (for production, front the instance with your own domain and certificate). Sign in with the administrator email and password from Step 3.

After signing in you land on the Runs view, which lists every workflow run with its status, workflow, timing and duration. The activity timeline at the top shows run volume over time.

Select any run to see its detail: the task steps that make up the workflow, the run status and duration, and a full activity trace showing each task being queued, assigned to a worker, started and completed.

Step 6 - Register your first worker and workflow

Hatchet runs your code on workers that connect to the engine and pull tasks. Create an API token from Settings, then point a worker at the instance using your language SDK. On a machine that can reach the instance, install the SDK and define a simple workflow:

# On a separate machine that can reach the instance (worker host):
python3 -m venv hatchet && ./hatchet/bin/pip install hatchet-sdk

# worker.py
from hatchet_sdk import Hatchet, Context, EmptyModel

hatchet = Hatchet()
pipeline = hatchet.workflow(name="sample-pipeline")

@pipeline.task()
def extract(input: EmptyModel, ctx: Context) -> dict:
    return {"records": 42}

@pipeline.task(parents=[extract])
def transform(input: EmptyModel, ctx: Context) -> dict:
    return {"transformed": 84}

if __name__ == "__main__":
    worker = hatchet.worker("sample-worker", workflows=[pipeline])
    worker.start()

Run the worker with the API token and the engine address in the environment:

export HATCHET_CLIENT_TOKEN='<your-token>'
export HATCHET_CLIENT_HOST_PORT='<public-ip>:7077'
python worker.py

Once the worker connects it appears on the Workers view as active, with its available slots and SDK version:

Trigger a run of the workflow from your code (pipeline.run()), from a schedule, or from an event, and watch each step execute in real time on the Runs view. New team members join by administrator invitation only, from Settings; open self sign up is disabled.

Step 7 - Confirm the dedicated data volume

The Docker data root, and therefore the PostgreSQL database, the RabbitMQ store and the pre pulled images, live on a dedicated EBS data volume mounted at /var/lib/docker, separate from the OS disk:

findmnt -no SOURCE,TARGET,FSTYPE /var/lib/docker
df -h /var/lib/docker

/dev/nvme1n1 /var/lib/docker ext4
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1     40G   74M   37G   1% /var/lib/docker

Back it up with EBS snapshots or AWS Backup, and resize it independently of the OS disk as your throughput grows.

Security notes

• No default or shared credentials. The well known default administrator the stock Hatchet compose seeds is never created; a unique administrator is generated on each instance's first boot and written only to /root/hatchet-credentials.txt.
• Per instance secrets. The PostgreSQL password, RabbitMQ password, encryption master keyset, JWT keysets and cookie secrets are all generated per instance on first boot.
• Closed registration. Open sign up and tenant creation are disabled, so the instance cannot be seized; new members join by administrator invitation only.
• Network isolation. The engine, dashboard and API bind exclusively to the loopback interface; PostgreSQL and RabbitMQ are never published to a host port; only the nginx TLS proxy is exposed. Restrict the security group to trusted networks and front the instance with your own domain and certificate before production.
• Rotate the administrator password from your account settings after your first sign in.

Support

cloudimg provides 24/7 technical support for this product by email and live chat.

• Contact: support@cloudimg.co.uk
• Help with deployment, reverse-proxy termination with your own domain and TLS certificate, connecting external workers, scaling the bundled PostgreSQL and RabbitMQ, and backup planning for your workflow database using EBS snapshots or AWS Backup.