Kestra Event-Driven Data and Workflow Orchestration on AWS User Guide

Overview

Kestra is an open source, event-driven orchestration platform for building and running data and business workflows. You declare pipelines as YAML flows, so the logic is readable and version-controllable, and a large plugin catalogue connects them to databases, cloud services, message queues, containers and scripts in any language. The server schedules and executes flows, persists every execution so work survives restarts, and its web interface provides a flow editor, a live executions view with a Gantt and topology breakdown, and a dashboard.

This cloudimg image runs the Apache-2.0-licensed Kestra open-source core as the official upstream kestra/kestra container, pinned by digest, in single-node server standalone mode alongside a bundled PostgreSQL, orchestrated by docker compose under a single systemd service on a private container network. Kestra publishes its web UI and API to the loopback interface only (127.0.0.1:8080) along with its management and health port (127.0.0.1:8081), and PostgreSQL is never published to a host port. A host nginx reverse proxy terminates TLS on port 443 with a per-instance self-signed certificate and enforces HTTP Basic authentication; plain HTTP on port 80 redirects to HTTPS, except the public /healthz liveness probe. Docker Engine and its entire data root, including the pre-pulled digest-pinned images and the PostgreSQL repository, live on a dedicated data volume mounted at /var/lib/docker, so your orchestration state survives OS-disk changes and is resizable independently.

Security is enforced from first boot. Kestra open-source ships no authentication of its own, and because it executes arbitrary tasks an exposed instance would be remote code execution. This image therefore never leaves it open: before Kestra starts, a per-instance Web UI credential, a per-instance PostgreSQL password and a per-instance secret encryption key are generated for the individual instance, along with the self-signed TLS certificate. The same per-instance credential gates two layers - the nginx HTTP Basic front door AND Kestra's own basic authentication. Backed by 24/7 cloudimg support.

What is included:

• The Apache-2.0-licensed Kestra open-source core, shipped as the official kestra/kestra image pinned by digest, running as a single-node standalone server
• A bundled PostgreSQL, pinned by digest, providing a durable repository AND queue reachable only inside a private container network
• Docker Engine with Kestra on the loopback interface behind nginx over TLS on port 443
• A per-instance Web UI credential, PostgreSQL password and secret encryption key generated on first boot, written to a root-only file
• Defense in depth: the same per-instance credential enforced by both nginx Basic authentication and Kestra's own basic authentication
• The Docker data root and PostgreSQL repository on a dedicated data volume at /var/lib/docker
• docker.service, kestra.service and nginx.service as systemd units, enabled and active
• 24/7 cloudimg support

Prerequisites

An AWS account, an EC2 key pair in the target region, and a VPC with a public subnet. m5.large (2 vCPU / 8 GiB RAM) is the recommended instance type - Kestra runs a JVM, so give it adequate memory. Security group inbound: allow 22/tcp from your management network for SSH and 443/tcp for the web UI.

Step 1 - Launch the AMI

Subscribe to the listing in AWS Marketplace, then launch an instance from the AMI into your VPC with the security group described above. Choose the m5.large instance type and attach your EC2 key pair. The image keeps the pre-pulled container images on the dedicated data volume, so the Kestra stack comes up within a couple of minutes of first boot with no image re-pull while it runs its database migrations.

Step 2 - Connect to your instance

Connect over SSH as the default login user for the operating system variant you launched. The login user differs per OS, so use the row for your variant:

ssh -i your-key.pem ubuntu@<public-ip>

Step 3 - Read your unique credentials

On first boot the instance generates a per-instance Web UI credential, PostgreSQL password and secret encryption key, writing them along with the resolved web URL to a root-only file. Read it with sudo:

sudo cat /root/kestra-notes.txt

You will see the Web UI login (KESTRA_UI_USER / KESTRA_UI_PASSWORD), the PostgreSQL password and the web URL for this instance. There is no default or shared login anywhere in the image; every secret is unique to this instance.

Step 4 - Verify the security model from the command line

Before opening a browser, confirm the front door is closed to anyone without the per-instance credential. The nginx TLS proxy on port 443 uses a self-signed certificate, so pass curl -k to accept it.

The public liveness probe answers with 200 and no credential (it exposes only Kestra's health status, no workflow data):

curl -sk -o /dev/null -w 'GET /healthz -> %{http_code}\n' https://127.0.0.1/healthz

An unauthenticated request to the UI or API is refused with 401:

curl -sk -o /dev/null -w 'GET /ui/ (no credential) -> %{http_code}\n' https://127.0.0.1/ui/

The same request with the per-instance credential is served with 200:

curl -sk -o /dev/null -w 'GET /ui/ (per-instance credential) -> %{http_code}\n' -u '<KESTRA_UI_USER>:<KESTRA_UI_PASSWORD>' https://127.0.0.1/ui/

Plain HTTP on port 80 redirects any non-health path to HTTPS:

curl -s -o /dev/null -w 'GET http://127.0.0.1/ui/ -> %{http_code}\n' http://127.0.0.1/ui/

Expected output:

GET /healthz -> 200
GET /ui/ (no credential) -> 401
GET /ui/ (per-instance credential) -> 200
GET http://127.0.0.1/ui/ -> 301

Step 5 - Sign in to the web UI

Browse to https://<instance-public-ip>/ and accept the self-signed certificate warning. Answer the HTTP Basic prompt with the KESTRA_UI_USER and KESTRA_UI_PASSWORD from Step 3, then sign in on Kestra's own login page with the same credentials. The Overview dashboard shows execution success ratios and run history:

The Flows page lists your declarative YAML workflows with their namespaces, labels and last execution status:

Open a flow to author or edit it in the built-in YAML editor, with inline documentation and a live topology preview:

Every execution is shown as a Gantt timeline with per-task timing, alongside logs, outputs and a topology view:

Step 6 - Author and run a flow through the API

Everything the UI does is available on the REST API, gated by the same per-instance credential. Create a small flow, execute it, and read the result back. First create the flow:

curl -sk -o /dev/null -w 'create flow -> %{http_code}\n' -u '<KESTRA_UI_USER>:<KESTRA_UI_PASSWORD>' \
  -X POST https://127.0.0.1/api/v1/main/flows -H 'Content-Type: application/x-yaml' \
  --data-binary 'id: getting_started
namespace: company.demo
tasks:
  - id: hello
    type: io.kestra.plugin.core.debug.Return
    format: "hello from cloudimg"'

Trigger an execution and capture its id:

EID=$(curl -sk -u '<KESTRA_UI_USER>:<KESTRA_UI_PASSWORD>' \
  -X POST https://127.0.0.1/api/v1/main/executions/company.demo/getting_started | jq -r '.id')
echo "execution id: $EID"

Poll the execution and confirm it reached SUCCESS:

sleep 6
curl -sk -u '<KESTRA_UI_USER>:<KESTRA_UI_PASSWORD>' \
  https://127.0.0.1/api/v1/main/executions/"$EID" | jq '{flowId, namespace, state: .state.current}'

Expected output (the scheduler and executor ran the flow and PostgreSQL persisted its state):

{
  "flowId": "getting_started",
  "namespace": "company.demo",
  "state": "SUCCESS"
}

Step 7 - Confirm the services and the data volume

The three systemd units should all be active:

systemctl is-active docker.service kestra.service nginx.service

Kestra binds only to the loopback interface (8080 UI, 8081 management/health); nginx owns the public 80 and 443; PostgreSQL appears on no host port:

sudo ss -tlnH | awk '{print $4}' | grep -E ':(80|443|8080|8081|5432)$' | sort -u

Expected: 0.0.0.0:80, 0.0.0.0:443, 127.0.0.1:8080 and 127.0.0.1:8081 - and no :5432.

The Docker data root, holding the pre-pulled images and the PostgreSQL repository, is a dedicated ext4 data volume:

findmnt -no SOURCE,TARGET,FSTYPE /var/lib/docker

Step 8 - Replace the self-signed certificate (optional)

The image ships a per-instance self-signed certificate so the UI is encrypted out of the box. For a browser-trusted certificate, point a DNS name at the instance and install a certificate from your own CA or a public issuer, updating the ssl_certificate and ssl_certificate_key paths in /etc/nginx/sites-available/kestra, then reload nginx. You can also front the instance with an Application Load Balancer terminating TLS with an ACM certificate.

Security model

• No known bootstrap credential. Kestra open-source ships no authentication; this image generates a unique Web UI credential, PostgreSQL password and secret encryption key per instance on first boot, all in a root-only file. Nothing is baked or shared across instances.
• Two authentication layers, one credential. nginx HTTP Basic authentication on port 443 is the authoritative front-door gate, and Kestra's own basic authentication validates the same credential - so even a request reaching Kestra on loopback is rejected without it.
• Least exposure. Kestra binds to 127.0.0.1 only; PostgreSQL is on the private container network with no host port; only 22, 80 and 443 are reachable from outside, and 80 redirects to 443 except the public health probe.
• Encryption at rest for secrets. Kestra's kestra.encryption.secret-key is generated per instance, so values and secrets stored in the repository are encrypted with a key unique to your instance.
• Fresh database per instance. The captured image carries no database, credential or flows; every instance re-initialises a clean PostgreSQL cluster and mints fresh secrets on first boot.

Support

This image is maintained by cloudimg with 24/7 support by email and live chat at support@cloudimg.co.uk. We help with deployment, upgrades, plugin configuration, flow authoring, performance tuning and PostgreSQL administration.

Kestra is a trademark of its respective holder. This image is not affiliated with, endorsed by or sponsored by Kestra Technologies Inc. The Kestra name is used nominatively to identify the open-source software shipped in this image.