Cylc 8.7.0 on AWS User Guide

Overview

This guide covers launching and using the Cylc 8.7.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg GIS, health and research applications listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Cylc ("silk") is an open source workflow engine for cycling systems: workflows that run the same graph of tasks again and again, once per cycle point (every 6 hours of model time, every day, every data delivery), with dependencies inside a cycle and between cycles. It was developed for weather and climate forecasting at NIWA and the Met Office and is used for environmental, research and data-processing pipelines. A scheduler runs each workflow; the Cylc UI is a web interface that shows every cycle point, every task and its state live, and lets you pause, trigger, hold and stop them.

The cloudimg image is a hardened, ready-to-use Cylc server:

• nginx in front, the Cylc UI Server behind it. nginx is the only public listener (ports 80 and 443). The Cylc UI Server listens on 127.0.0.1:8890, the loopback address only. The schedulers' own network ports (43001 to 43101) and the Cylc Review port (8080) are refused unless the connection comes from the instance itself.
• HTTPS from the first boot, with a certificate generated on your instance (an end-entity certificate, not a certificate authority), HTTP/1.1 with websockets for the UI's live updates, TLS 1.2 and 1.3 only. Port 80 redirects to HTTPS. The instance's name is set from its own address at first boot; requests for any other host name are dropped. You can change it to your own domain.
• A password unique to your instance. First boot generates a random password and a cookie-signing secret. The UI Server stores only an argon2id hash of the password, in a root-only file it reads through systemd credentials, and refuses to start without valid ones. Token sign-in is switched off, so no credential is ever part of a URL. Changing the password signs out every existing session.
• Sign-in protection. nginx allows 6 password attempts per minute per client (with a burst of 5), keyed on the real TCP peer address: a forged X-Forwarded-For header neither moves you out of the limit nor reaches the UI Server's log. Every Cylc page, the GraphQL API and the websocket refuse a request without a valid session; a page on another website can neither sign in nor open the websocket.
• Browser hardening. A Content-Security-Policy that allows only the UI's own scripts and refuses framing, HSTS, a same-origin referrer policy, X-Frame-Options: DENY, and an access log that never records a query string.
• An unprivileged workflow account. Workflows, their schedulers and their tasks run as the cylc account, which has no password and no sudo rights. The UI Server runs as cylc under systemd with NoNewPrivileges, a read-only operating system (ProtectSystem=strict) and core dumps off. The configuration files the UI Server reads are owned by root, so a workflow task cannot change how the UI Server authenticates.
• The metadata service is blocked for the UI Server, nginx, the first-boot service, and every scheduler and task the cylc account runs, however it was started; the AMI requires IMDSv2. If your workflows need AWS credentials, Step 9 shows how to choose that on purpose. The same accounts also cannot send to the reserved, benchmarking and multicast ranges (192.0.0.0/24, 198.18.0.0/15, 224.0.0.0/4, 240.0.0.0/4). Workflow tasks can reach other addresses in your VPC, because remote job hosts there are a normal Cylc set-up; scope that with your security groups.
• Nothing shipped that you did not ask for. The image holds no workflow and no run directory; the only job platform is this instance (localhost), and Step 8 explains how to add remote hosts. Cylc and its Python dependencies are installed from the upstream releases, every file checked against a published SHA-256 hash, on a fully patched Ubuntu 24.04 LTS with unattended security upgrades on. See Step 12.

What is on the image: Cylc 8.7.0 (cylc-flow 8.7.0, cylc-uiserver 1.10.0 with the cylc-ui 2.15.0 web interface) in a Python 3.12 virtual environment at /opt/cylc/venv, nginx and nftables from Ubuntu 24.04, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
• A security group that allows TCP 22 (SSH) and TCP 443 (the Cylc UI) and TCP 80 (redirects to HTTPS) only from your own networks: your office, your VPN, or your VPC CIDR. Do not open 8890, 8080 or 43001-43101: the UI Server listens on the loopback address only, and the scheduler and Review ports are refused from the network by the instance itself.
• For production, a DNS name you control (for example cylc.example.com) and, ideally, a certificate from a certificate authority. Until then the instance uses its public IP address with its own self-signed certificate.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Cylc 8.7.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger for big workflows), your key pair, and a security group as above. Workflow run directories live under /var/lib/cylc/cylc-run on the root volume, so size the root volume for the logs and outputs your workflows keep.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --image-id <ami-id-from-the-listing> --instance-type m5.large \
  --key-name <your-key-pair> --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=cylc}]'

First boot takes a few seconds after the instance is running. If you want the instance to use your own name from the start, put it in /etc/cloudimg/cylc-domain.preset with user data (for example a cloud-init write_files entry); otherwise first boot uses the instance's public IP address (or its private address if it has no public one), and also serves its EC2 public DNS name and its private address.

Step 2: Connect to your instance

Operating system	SSH user
Ubuntu 24.04	ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

The login message shows the address of the Cylc UI and where the password is.

Step 3: Check the services

systemctl is-active cloudimg-cylc-guard cylc-uiserver nginx

active
active
active

(cylc-firstboot runs once, on the first boot, and shows inactive on every later boot.)

Only SSH, HTTP and HTTPS are public; the UI Server listens on the loopback address:

sudo ss -tln | grep -E ':(22|80|443|8890|8080|4300[0-9]|430[1-9][0-9]|4310[01]) '

LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*
LISTEN 0      511          0.0.0.0:80        0.0.0.0:*
LISTEN 0      511          0.0.0.0:443       0.0.0.0:*
LISTEN 0      128        127.0.0.1:8890      0.0.0.0:*
LISTEN 0      4096            [::]:22           [::]:*
LISTEN 0      511             [::]:80           [::]:*
LISTEN 0      511             [::]:443          [::]:*

A built-in self-test signs in through nginx with this instance's password, checks that a request without a session and a wrong password are refused, and runs a GraphQL query and a websocket subscription:

sudo cylc-selftest

cylc-selftest: unauthenticated UI/profile/version/GraphQL/websocket and ?token= refused; wrong password 401 (no login cookie); signed in as cylc (24 permissions); GraphQL 0 workflow(s); websocket connection_ack
CYLC_SELFTEST_OK

Step 4: Read the address, the password and the certificate fingerprint

The address and certificate fingerprint are in /root/cylc-credentials.txt (root only, mode 0600). To show everything except the password:

sudo grep -v '^CYLC_PASSWORD=' /root/cylc-credentials.txt

# Cylc 8.7.0 (cloudimg) on this instance. Open CYLC_URL in a browser and sign in with CYLC_PASSWORD (password only).
# The password is unique to this instance. Change it: sudo cylc-set-password (or --stdin to choose one).
# The certificate is self-signed for this instance: compare its SHA-256 fingerprint with CYLC_TLS_SHA256.
# Workflows run as the cylc account: sudo -iu cylc   (sources ~/cylc-src, runs ~/cylc-run)
CYLC_URL=https://198.51.100.24/cylc/
CYLC_HOST=198.51.100.24
CYLC_TLS_SHA256=09:BC:CE:DE:62:BD:7C:B7:29:91:C7:B3:3F:2D:F5:12:F8:08:94:CC:09:97:76:06:AC:31:20:1F:93:F9:87:49
CYLC_ADDRESS_SOURCE=imdsv2-public-ipv4
GUIDE=https://www.cloudimg.co.uk/guides/cylc-8-7-0-on-ubuntu-24-04-aws/

The password is on the CYLC_PASSWORD line. Show it only when you are ready to copy it, and do not paste it into a ticket, a chat or a shell history:

sudo grep '^CYLC_PASSWORD=' /root/cylc-credentials.txt

The certificate is self-signed and unique to your instance, so your browser warns about it. Open the certificate details in the browser, check that its SHA-256 fingerprint matches CYLC_TLS_SHA256, and only then continue. Do not add the certificate to your operating system's trust store; for production, replace it with a certificate from a certificate authority (Step 11). Copy it to your home directory so that the commands in this guide can use it as their trust anchor:

sudo cat /etc/ssl/cylc/cylc.crt > ~/cylc-instance.crt
openssl x509 -in ~/cylc-instance.crt -noout -subject -ext basicConstraints,keyUsage,extendedKeyUsage,subjectAltName

subject=CN = 198.51.100.24, O = cloudimg
X509v3 Subject Alternative Name:
    IP Address:198.51.100.24, DNS:ec2-198-51-100-24.compute-1.amazonaws.com, IP Address:172.31.10.20, DNS:localhost, IP Address:127.0.0.1
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Key Usage: critical
    Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
    TLS Web Server Authentication

Step 5: Sign in to the Cylc UI

Browse to the CYLC_URL address. The UI Server's sign-in page asks for the password only:

Enter the password from Step 4. A wrong password is refused and gives no session; after 6 attempts in a minute from the same address, nginx answers 429 Too Many Requests until the minute has passed. The session cookie is Secure, HttpOnly and SameSite=Lax, and it stays valid until you change the password (Step 10) or for 31 days. Browsing to /logout on the instance's address removes it from that browser.

The UI opens on the workflow list. A new instance has no workflows yet; Step 6 runs one.

Step 6: Run an example cycling workflow

Workflows are run by the cylc account. Open a shell as cylc (its home is /var/lib/cylc, workflow sources go in ~/cylc-src and runs in ~/cylc-run):

sudo -iu cylc

The commands below run the same steps without an interactive shell (sudo -iu cylc starts in the cylc home, so ./cylc-src/... is the workflow source there; give Cylc a path such as ./cylc-src/<name> for a source directory, because a bare name is read as an installed workflow). First write an example workflow: three cycle points, six hours apart, each fetching observations, running a model that also depends on the previous cycle's model run, publishing a product and tidying up:

sudo -iu cylc bash -c 'mkdir -p ~/cylc-src/cloudimg-demo && cat > ~/cylc-src/cloudimg-demo/flow.cylc' <<'EOF'
[scheduler]
    UTC mode = True
    allow implicit tasks = False
[scheduling]
    initial cycle point = 20260101T0000Z
    final cycle point = 20260101T1200Z
    [[graph]]
        PT6H = """
            get_observations => run_model => publish_products => housekeep
            run_model[-PT6H] => run_model
        """
[runtime]
    [[get_observations]]
        script = sleep 10; echo "observations for ${CYLC_TASK_CYCLE_POINT}"
    [[run_model]]
        script = sleep 20; echo "model run for ${CYLC_TASK_CYCLE_POINT}"
    [[publish_products]]
        script = """
            sleep 5
            mkdir -p "${CYLC_WORKFLOW_SHARE_DIR}/products"
            echo "forecast valid ${CYLC_TASK_CYCLE_POINT}" > "${CYLC_WORKFLOW_SHARE_DIR}/products/forecast-${CYLC_TASK_CYCLE_POINT}.txt"
        """
    [[housekeep]]
        script = sleep 3
EOF
sudo -iu cylc cylc validate ./cylc-src/cloudimg-demo || { echo "FATAL: the workflow did not validate"; exit 1; }

Install it (a new run directory, ~/cylc-run/cloudimg-demo/run1) and start its scheduler:

sudo -iu cylc cylc install ./cylc-src/cloudimg-demo || { echo "FATAL: cylc install failed"; exit 1; }
sudo -iu cylc cylc play cloudimg-demo || { echo "FATAL: cylc play failed"; exit 1; }

INSTALLED cloudimg-demo/run1 from /var/lib/cylc/cylc-src/cloudimg-demo

 ▪ ■  Cylc Workflow Engine 8.7.0
 ██   Copyright (C) 2008-2026 Earth Sciences New Zealand
▝▘    & British Crown (Met Office) & Contributors

INFO - Extracting job.sh to /var/lib/cylc/cylc-run/cloudimg-demo/run1/.service/etc/job.sh
cloudimg-demo/run1: ip-172-31-10-20.ec2.internal PID=4934

Back in the browser, the dashboard counts one running workflow, and the run appears in the workflow list on the left as cloudimg-demo/run1:

Click cloudimg-demo/run1 to open it. The Tree view shows each active cycle point, its tasks and their states (waiting, preparing, submitted, running, succeeded), updated live over the UI's websocket:

A little later the run has moved on: in the first cycle point the model run has succeeded and publish_products is running, and the second cycle point's model run, which waited for the first one, is now running:

The run takes about two minutes. Wait for the scheduler to finish, then look at what it produced:

for i in $(seq 1 60); do sudo -iu cylc cylc scan --states=running --format=name | grep -q '^cloudimg-demo/' || break; sleep 5; done
sudo -iu cylc bash -c 'cat ~/cylc-run/cloudimg-demo/runN/share/products/forecast-*.txt' \
  || { echo "FATAL: the workflow published no products"; exit 1; }
sudo -iu cylc cylc cat-log cloudimg-demo//20260101T1200Z/run_model | tail -3

forecast valid 20260101T0000Z
forecast valid 20260101T0600Z
forecast valid 20260101T1200Z
2026-11-08T09:43:30Z INFO - started
model run for 20260101T1200Z
2026-11-08T09:43:50Z INFO - succeeded

When you no longer need a run, remove its run directory:

sudo -iu cylc cylc clean --yes cloudimg-demo || { echo "FATAL: cylc clean failed"; exit 1; }

Step 7: Your own workflows

Put each workflow in its own directory under ~cylc/cylc-src with a flow.cylc file, then cylc validate, cylc install and cylc play it as above. The Cylc documentation covers the workflow language, cycling, retries, events and every command. Everyday commands:

sudo -iu cylc cylc scan                        # running workflows
sudo -iu cylc cylc tui <workflow>              # the terminal UI
sudo -iu cylc cylc pause <workflow>            # stop submitting new jobs
sudo -iu cylc cylc play <workflow>             # resume, or restart a stopped run
sudo -iu cylc cylc stop <workflow>             # stop once running jobs finish
sudo -iu cylc cylc trigger <workflow>//<cycle>/<task>

Things to know on this image:

• Run workflows as cylc. The UI shows and controls the cylc account's workflows only. The account is a plain user: tasks cannot use sudo, and the operating system directories are read-only to them.
• Starting from the UI or from SSH. A workflow you start with cylc play over SSH keeps running if the UI Server restarts. A workflow you start or restart from the UI runs inside the UI Server's service, so it stops when the UI Server restarts (for example when you change the password); cylc play <workflow> over SSH resumes it where it stopped.
• The UI shows one operator. This is a single-user appliance: whoever has the password has full control of every cylc workflow. Cylc's multi-user authorisation (user_authorization) applies only behind a hub, so it is left empty here.

Step 8: Run tasks on other hosts

As shipped, the only job platform is this instance (localhost, background jobs), configured in /etc/cylc/flow/global.cylc. To run tasks on an HPC login node or a batch system, define a platform there (or in ~cylc/.cylc/flow/global.cylc). For example, a Slurm cluster reached over SSH:

[platforms]
    [[hpc]]
        hosts = login1.hpc.example.com
        job runner = slurm
        install target = hpc

Then:

1. Give the cylc account an SSH key that the remote account accepts (sudo -iu cylc ssh-keygen -t ed25519, and add ~cylc/.ssh/id_ed25519.pub to the remote account's authorized_keys), and check sudo -iu cylc ssh login1.hpc.example.com true.
2. Install Cylc 8.7.0 (cylc-flow) on the remote host, as the platform documentation describes.
3. Remote jobs report back to the scheduler on its ZMQ ports (43001-43101, set in global.cylc). The image refuses those ports from the network. To accept them from the remote hosts only, open them in your security group for those hosts and add an accept rule for their addresses before the refusal in the input chain of /etc/cloudimg/cylc-guard.nft, for example ip saddr 203.0.113.0/24 tcp dport 43001-43101 accept, then sudo systemctl restart cloudimg-cylc-guard. Alternatively set the platform's communication method = poll so that remote jobs never connect back.

Step 9: AWS credentials for workflow tasks

The image blocks the instance metadata service for the cylc account: the UI Server, every scheduler and every task. This stops a workflow task (or any code that tricks a task into making a request) from reading the instance role's temporary credentials. You can check it:

sudo -iu cylc curl -s -m 5 -o /dev/null -w 'cylc account -> metadata service: %{http_code}\n' -X PUT \
  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' http://169.254.169.254/latest/api/token || true
curl -s -m 5 -o /dev/null -w 'ubuntu account -> metadata service: %{http_code}\n' -X PUT \
  -H 'X-aws-ec2-metadata-token-ttl-seconds: 60' http://169.254.169.254/latest/api/token

cylc account -> metadata service: 000
ubuntu account -> metadata service: 200

(000 means the connection was refused.) If your tasks must call AWS services, prefer credentials scoped to the workflow, for example an ~cylc/.aws/credentials profile for a dedicated IAM user or a role assumed with aws sts assume-role. If you decide that the tasks may use the instance role, remove the block for the cylc account: delete the two meta skuid lines that name 169.254.169.254 and fd00:ec2::254 from the output chain of /etc/cloudimg/cylc-guard.nft (or keep www-data in them and remove only the cylc uid; leave the third line, the reserved ranges, in place), run sudo systemctl restart cloudimg-cylc-guard, and for workflows started from the UI also add a drop-in that replaces the UI Server's own block with the reserved ranges only:

sudo systemctl edit cylc-uiserver      # add these three lines, save, then:
[Service]
IPAddressDeny=
IPAddressDeny=192.0.0.0/24 198.18.0.0/15 224.0.0.0/4 240.0.0.0/4
sudo systemctl restart cylc-uiserver

Attach an instance role with only the permissions the workflows need.

Step 10: Change the password

Generate a new random password (written to /root/cylc-credentials.txt) or choose one of at least 12 characters, read from standard input so that it never appears on a command line:

sudo cylc-set-password
read -rs P && printf '%s\n' "$P" | sudo cylc-set-password --stdin; unset P

The UI Server restarts with the new hash, and every existing browser session is signed out. Workflows that were started from the UI stop with the restart (the command says so if there are any); resume them with sudo -iu cylc cylc play <workflow>.

Step 11: Use your own domain and certificate

Point a DNS name at the instance, then make it the instance's name. The command writes the nginx host names (requests for any other name are dropped), generates a new self-signed certificate for the names and restarts nginx:

sudo cylc-set-domain cylc.example.com

To use a certificate from a certificate authority instead, put the full chain and key in /etc/ssl/cylc/cylc.crt and /etc/ssl/cylc/cylc.key (root only, mode 0600 for the key), then sudo nginx -t && sudo systemctl reload nginx. If the instance's public address changes after a stop and start, run cylc-set-domain again with the new address. To reach the UI without opening port 443, use an SSH tunnel: ssh -L 8443:127.0.0.1:443 ubuntu@<public-ip> and browse to https://localhost:8443/cylc/.

Step 12: Versions, licences and source

sudo -iu cylc cylc version --long | head -3
cat /usr/share/doc/cylc/README.cloudimg

8.7.0 (/opt/cylc/venv/bin/cylc)

Plugins:
Cylc on this image (cloudimg)
=============================
  cylc-flow      8.7.0   GPL-3.0-or-later   https://github.com/cylc/cylc-flow
  cylc-uiserver  1.10.0  GPL-3.0-or-later   https://github.com/cylc/cylc-uiserver
  cylc-ui        2.15.0  GPL-3.0-or-later   https://github.com/cylc/cylc-ui (the web UI bundle inside cylc-uiserver)
All three are shipped UNMODIFIED, installed in /opt/cylc/venv from the hash-pinned lock in requirements.lock (every
dependency's licence: dependency-licences.txt). The licence texts are COPYING.cylc-flow, COPYING.cylc-uiserver and
COPYING.cylc-ui. The corresponding source is in source/:
  cylc_flow-8.7.0.tar.gz   sha256 743cbe7bde84784f91e3d2758dbffe1aa1afedd66ee4015b39a20221ca7c6a67   (https://files.pythonhosted.org/packages/7d/64/c0ab38c5143b648d2638992c980d1bce7f6d4e7a6f05b5af8cc35fd7c427/cylc_flow-8.7.0.tar.gz)
  cylc_uiserver-1.10.0.tar.gz    sha256 b1d5bc0ad433b7e47709784614b3ff946f210654b280c0a25cb9b240ff342810   (https://files.pythonhosted.org/packages/88/52/97dbce50928bb12c8e4863d0fc14662b52e17b46e8aacfe8118ec2250e12/cylc_uiserver-1.10.0.tar.gz)
  cylc-ui-2.15.0.tar.gz       sha256 eb1b1d980f5678ed75fde8d42074d5fa5aef92cbd219da7b8543f3f8a4d939e5   (https://codeload.github.com/cylc/cylc-ui/tar.gz/refs/tags/2.15.0)
cloudimg packages and configures Cylc; it does not modify it. Cylc is not endorsed by or affiliated with cloudimg.

Cylc (cylc-flow, cylc-uiserver and cylc-ui) is free software under the GNU General Public License version 3 or later. The image ships it unmodified; the licence texts, the full list of installed Python packages with their versions, hashes and licences, and the complete corresponding source are in /usr/share/doc/cylc. The Python packages are pinned by SHA-256 in /usr/share/doc/cylc/requirements.lock.

Ubuntu's unattended upgrades keep the operating system, nginx and Python patched. Cylc itself does not update automatically: the cloudimg listing adds a new version when Cylc releases one. To keep workflows across a move to a new instance, copy ~cylc/cylc-src (sources) and any run directories you still need.

Cylc is developed by NIWA, the Met Office and the Cylc community. cloudimg packages and configures it and is not affiliated with or endorsed by the Cylc project.

Support

cloudimg provides support for this image by email at support@cloudimg.co.uk, 24 hours a day, 7 days a week. Tell us the AMI ID and Region, and include the output of sudo cylc-selftest and systemctl status cylc-uiserver nginx. For questions about writing Cylc workflows, see the Cylc documentation and the Cylc Discourse forum.