Dagu 2.10.7 on AWS User Guide

Overview

This guide covers launching and using the Dagu 2.10.7 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg Container Platforms and PaaS listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Dagu is a self contained workflow engine. You describe a workflow as a DAG (a directed acyclic graph of steps) in a short YAML file, then run it by hand, on a cron schedule or through the REST API, and watch every run, step and log in the built in web UI. It is a single Go binary with no database or message broker, so it is a lightweight replacement for crontab sprawl and ad hoc scripts.

The cloudimg image adds:

• HTTPS from the first boot. nginx serves Dagu on port 443 with a certificate generated on your instance's first boot. Port 80 redirects to HTTPS.
• An admin password unique to your instance, checked twice. nginx asks for the user admin and a password generated on first boot, and Dagu's own basic authentication checks the same password again on its REST API. The password is written to /root/dagu-credentials.txt and to Dagu's root owned configuration file. No password is baked into the image.
• Fail closed. Dagu refuses to start until first boot has written that password, and nginx refuses to start until first boot has finished.
• Loopback only. Dagu listens on 127.0.0.1:8080 behind nginx; its distributed worker coordinator is switched off. The only open ports are 22, 80 and 443.
• Hardened defaults. Dagu runs as its own unprivileged user in a systemd sandbox. Dagu, every workflow step it runs, and nginx are blocked from the instance metadata service; the AMI requires IMDSv2; requests for a host name the instance does not serve are dropped; and cross site requests that could start a workflow are refused.
• Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Dagu 2.10.7 (/usr/local/bin/dagu, the official release binary, checksum verified), nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent. One example workflow, hello, is included; it runs only when you start it.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH), TCP 443 (the web UI and API) and TCP 80 (redirects to HTTPS) from your own networks only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Dagu 2.10.7 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (size up if your workflows do heavy work themselves), your key pair and a security group as above. Workflows, run history and logs live under /var/lib/dagu on the root volume, so give it room for the logs you want to keep.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=30,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=dagu}]'

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot takes a few seconds. Then the three units are active:

systemctl is-active dagu nginx dagu-firstboot

active
active
active

Dagu listens on loopback only; nginx serves ports 80 and 443:

sudo ss -tlnp | awk 'NR==1 || /dagu|nginx|sshd/' | awk '{print $1, $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g'

State Local Peer
LISTEN 127.0.0.1:8080 users:(("dagu"))
LISTEN 0.0.0.0:22 users:(("sshd"),("systemd"))
LISTEN 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
LISTEN 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
LISTEN [::]:22 users:(("sshd"),("systemd"))
LISTEN [::]:80 users:(("nginx"),("nginx"),("nginx"))
LISTEN [::]:443 users:(("nginx"),("nginx"),("nginx"))

Step 4: Get your URL and password

sudo cat /root/dagu-credentials.txt

The file holds the address the UI is served as (your instance's public DNS name by default), the user and the password generated on this instance:

DAGU_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/
DAGU_USERNAME=admin
DAGU_PASSWORD=<24 letters and digits>

Step 5: Sign in from your browser

Open DAGU_URL in your browser. The certificate is self signed (generated on your instance), so accept the browser warning once, or install your own certificate (Step 9). Sign in as admin with the password from Step 4. The Overview shows a timeline of today's runs:

Workflows lists every DAG file in /var/lib/dagu/dags, with its latest status, schedule and run controls:

Dagu may show a banner when a newer Dagu release exists. The image ships the version named in the listing; see Step 10 before you upgrade it.

Step 6: Run and build workflows

Open hello and press the start button. The Status tab draws the graph of steps and colours each one as it runs; the Log button shows each step's output:

The Spec tab is an editor for the workflow's YAML, with the schema reference beside it:

Create new workflows with New in the UI, or as files on the instance. Each file in /var/lib/dagu/dags is one workflow, named after the file. Add schedule: "0 * * * *" (a cron expression) to run one on a schedule; the scheduler is already running. Steps run as the unprivileged dagu user, inside Dagu's sandbox, with /var/lib/dagu as their writable area.

To add a workflow from the shell, write the file as the dagu user:

sudo -u dagu tee /var/lib/dagu/dags/nightly-report.yaml >/dev/null <<'YML'
schedule: "0 2 * * *"
steps:
  - id: collect
    run: df -h /
  - id: report
    run: echo "report done"
    depends: collect
YML

Step 7: Use the REST API

Everything in the UI is available from the REST API under /api/v1, behind the same user and password. On the instance, this lists the workflows and their latest status (the password is read from the credentials file and passed to curl on stdin, so it never appears on a command line):

H=$(sudo grep '^DAGU_URL=' /root/dagu-credentials.txt | sed -E 's#^DAGU_URL=https://([^/]+)/.*#\1#')
sudo sed -n 's/^DAGU_PASSWORD=\(.*\)$/user = "admin:\1"/p' /root/dagu-credentials.txt \
  | curl -sk -K - --resolve "$H:443:127.0.0.1" "https://$H/api/v1/dags" \
  | jq -r '.dags[] | "\(.dag.name)\t\(.latestDAGRun.statusLabel)"'

hello   succeeded

Start the example workflow and read the result of its latest run:

H=$(sudo grep '^DAGU_URL=' /root/dagu-credentials.txt | sed -E 's#^DAGU_URL=https://([^/]+)/.*#\1#')
auth() { sudo sed -n 's/^DAGU_PASSWORD=\(.*\)$/user = "admin:\1"/p' /root/dagu-credentials.txt; }
auth | curl -sk -K - --resolve "$H:443:127.0.0.1" -X POST -H 'Content-Type: application/json' -d '{}' \
  "https://$H/api/v1/dags/hello/start" | jq -r '.dagRunId | length > 0'
sleep 5
auth | curl -sk -K - --resolve "$H:443:127.0.0.1" "https://$H/api/v1/dags/hello/dag-runs/latest" \
  | jq -r '.dagRun | .statusLabel, (.nodes[] | "\(.step.id // .step.name): \(.statusLabel)")'

true
succeeded
greet: succeeded
show_date: succeeded

From your own computer, curl -u admin https://<your-dns-name>/api/v1/dags prompts for the password. The interactive API reference is the API Docs page of the UI (https://<your-dns-name>/api-docs).

Step 8: Security checks

Without the password, both nginx and Dagu itself refuse the API:

H=$(sudo grep '^DAGU_URL=' /root/dagu-credentials.txt | sed -E 's#^DAGU_URL=https://([^/]+)/.*#\1#')
curl -sk -o /dev/null -w 'through nginx: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/api/v1/dags"
curl -s -o /dev/null -w 'Dagu on loopback: %{http_code}\n' http://127.0.0.1:8080/api/v1/dags

through nginx: 401
Dagu on loopback: 401

The password lives in Dagu's configuration file, not in its environment, so workflow steps do not inherit it:

sudo grep -v 'password:' /etc/dagu/config.yaml
sudo stat -c '%a %U:%G %n' /etc/dagu/config.yaml /root/dagu-credentials.txt

host: 127.0.0.1
port: 8080
coordinator:
  enabled: false
auth:
  mode: basic
  basic:
    username: admin
640 root:dagu /etc/dagu/config.yaml
600 root:root /root/dagu-credentials.txt

The certificate was generated on this instance and names its own addresses:

H=$(sudo grep '^DAGU_URL=' /root/dagu-credentials.txt | sed -E 's#^DAGU_URL=https://([^/]+)/.*#\1#')
echo | openssl s_client -connect 127.0.0.1:443 -servername "$H" 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName

subject=CN = ec2-<public-ip-dashed>.<region>.compute.amazonaws.com, O = cloudimg
X509v3 Subject Alternative Name:
    DNS:ec2-<public-ip-dashed>.<region>.compute.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost

Dagu runs unprivileged in a sandbox, and neither Dagu (with every workflow step it runs) nor nginx can reach the instance metadata service:

systemctl show dagu -p User -p ProtectSystem -p NoNewPrivileges -p IPAddressDeny
systemctl show nginx -p IPAddressDeny

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
User=dagu
ProtectSystem=full
NoNewPrivileges=yes
IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

nginx passes Dagu a fixed host name and the real client address from the connection, never values a client sends:

grep -E 'proxy_set_header (Host|X-Forwarded-For|X-Real-IP)' /etc/nginx/sites-available/cloudimg-dagu

        proxy_set_header Host 127.0.0.1:8080;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;

nginx also refuses a state changing request (such as starting a workflow) whose Origin is another web site (HTTP 403), because browsers resend Basic Auth credentials on cross site requests.

Step 9: Your own DNS name and certificate

Point a DNS record at the instance, then set the name Dagu is served as. This mints a new self signed certificate for the name, updates nginx and the URL in /root/dagu-credentials.txt, and keeps the name across reboots:

sudo dagu-set-hostname <your-domain>

To use a certificate from your own certificate authority instead, copy it over the generated pair and reload nginx (running dagu-set-hostname again replaces it with a self signed one):

sudo install -m 0644 <your-domain>.crt /etc/ssl/dagu/dagu.crt
sudo install -m 0600 <your-domain>.key /etc/ssl/dagu/dagu.key
sudo nginx -t && sudo systemctl reload nginx

Step 10: Password, configuration and upgrades

dagu-set-password sets one new password on both gates (Dagu's configuration and the nginx password file, stored as a bcrypt hash), records it in /root/dagu-credentials.txt and restarts Dagu. Run it without arguments for a new random password, or pass your own (24 to 64 letters and digits) on stdin:

sudo dagu-set-password
sudo cat /root/dagu-credentials.txt

Dagu reads /etc/dagu/config.yaml; the image sets:

Workflow files, run history and logs are under /var/lib/dagu (DAGU_HOME). After changing the configuration, run sudo systemctl restart dagu.

The image runs Dagu 2.10.7 in basic authentication mode with a single admin user. Dagu's builtin user and role mode and its API keys are not used. If you switch to that mode, upgrade Dagu first, because Dagu's advisory GHSA-5m6c-38r8-5w76 (fixed in 2.11.0) lets a viewer role API key read YAML files through the MCP tools. To upgrade, download the release from the Dagu project, check its checksum, replace /usr/local/bin/dagu and restart the service. Snapshot the instance before you do.

Step 11: Workflows that need AWS credentials

Workflow steps run inside the Dagu service, which is blocked from the instance metadata service (169.254.169.254). This stops a workflow, or a URL a webhook passes to one, from reading the instance role's credentials. As a result, a step that relies on an instance profile (for example aws s3 cp) gets no credentials. Prefer credentials scoped to the workflow, such as a dedicated IAM user's keys or an assumed role, supplied through an environment file that only the dagu user can read.

If you accept the trade off and want steps to use the instance role, remove the block for Dagu only and restart it. Launch the instance with HttpPutResponseHopLimit=1 in either case:

sudo rm /etc/systemd/system/dagu.service.d/50-cloudimg-imds-deny.conf
sudo systemctl daemon-reload && sudo systemctl restart dagu

Step 12: Server components

dagu version
sudo journalctl -u dagu --no-pager -n 3 -o cat | cut -c1-80

Step 13: Security recommendations

• Allow ports 22, 80 and 443 in the security group from your own networks only.
• Keep the admin password secret, and rotate it if it may have been shared (Step 10).
• Use your own DNS name and a certificate your clients trust (Step 9).
• Treat anyone who can sign in as able to run commands on the instance as the dagu user: that is what a workflow engine does.
• Keep the operating system patched; unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.
• Snapshot the root volume, or copy /var/lib/dagu elsewhere, to back up your workflows and history.

Support and licensing

Dagu is open source under the GNU General Public License v3.0 (a copy is at /usr/share/doc/dagu/LICENSE); there is no per user or per CPU licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk
• Support: 24/7, with a 24 hour response SLA