Cylc on Ubuntu 24.04

Overview

Cylc is an open source workflow engine and meta scheduler designed for cycling workflows. Where a conventional pipeline tool runs a graph once per trigger, Cylc runs the same graph repeatedly over a sequence of cycle points, and lets a task in one cycle depend on a task in the previous cycle. Its scheduler keeps an unbounded cycling suite running continuously, handling retries, held and triggered tasks, and inter cycle dependencies without an operator having to restart anything between cycles.

That model is why Cylc is the engine behind operational numerical weather prediction and climate research at national meteorological services and academic computing centres. It suits any team running repeating, time indexed computation: forecast suites, rolling data assimilation, nightly or hourly reprocessing, long climate integrations, and research workflows that have to survive task failure and be resumed cleanly. Workflows are defined in a readable configuration file that describes the graph and the runtime for each task, validated before they run, and driven from either the command line or a live web interface that shows the state of every task in every cycle.

Common uses

• Run repeating, time indexed suites such as forecast, reprocessing or data assimilation cycles
• Express dependencies that reach back into the previous cycle, not just within a single run
• Watch and control every task in every cycle from one live web interface

See it running

Real screenshots taken while testing this image against its deployment guide.