Rucio on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and use of Rucio on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. Rucio is the open source scientific data-management system created at CERN: it keeps a single catalogue of your data files, records which storage systems hold each copy, and lets you find, upload, download and move data through one consistent REST API and command line client. Data is addressed by a logical name inside a namespace (a scope) rather than by a physical path, so a file can be located and retrieved no matter which storage element actually holds it.

The image runs the official Rucio 41.1.1 server (rucio/rucio-server, an Apache httpd + mod_wsgi application) pinned to the Rucio 41 long-term-support line, backed by a bundled PostgreSQL 16 catalogue. Rucio's core objects are all pre-created for you on first boot: the root account, a local POSIX/file storage element (RSE) called LOCAL, a scope called test, and an unlimited quota, so a real data round-trip works immediately. The whole appliance is driven by the REST API (health at GET /ping) and the bundled rucio command line client that ships pre-configured inside the server container.

Secure by default — no shipped credentials. There is no default login in the image. Both the PostgreSQL password and the Rucio administrator password are generated uniquely on the first boot of every VM and written to a file only root can read. The well-known upstream demonstration credential (ddmlab / secret) is never created, and the appliance proves it is rejected. The PostgreSQL catalogue is bound to the internal Docker network only and is never exposed to the host or the internet.

What is included:

• The official Rucio 41.1.1 server (rucio/rucio-server), run under systemd as rucio.service via Docker Compose, serving the REST API on port 80
• A bundled PostgreSQL 16 catalogue store, reachable only on the internal Docker network
• A pre-created root account with a per-VM userpass identity (username admin)
• A ready-to-use local POSIX/file RSE named LOCAL, a scope test, and unlimited root quota, so register, upload, list and locate all work on first boot
• Per-VM secrets written to /root/rucio-credentials.txt (mode 0600, root only)

Prerequisites

• Active Azure subscription, SSH public key, VNet + subnet in the target region
• Subscription to the Rucio listing on Azure Marketplace
• Network Security Group rules allowing TCP 22 (admin) and, if you want to reach the REST API remotely, TCP 80. For a single-node evaluation nothing needs to be open to the internet.

Recommended virtual machine size: Standard_B2s (2 vCPU, 4 GB RAM) is ample for the single-node server and catalogue. Size up for larger catalogues and higher request rates.

Step 1: Deploy from the Azure Portal

1. Open the Rucio offer on the Azure Marketplace and choose Get It Now, then Create.
2. Select your subscription, resource group and region, and choose the Standard_B2s size.
3. Provide your SSH public key for the azureuser account.
4. Allow inbound TCP 22 from your admin network in the Network Security Group (add TCP 80 if you want to reach the REST API from other hosts), then create the VM.

Step 2: Deploy from the Azure CLI

az vm create \
  --resource-group my-rg \
  --name rucio-01 \
  --image <rucio-marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

On the first boot the appliance generates its per-VM secrets, initialises the catalogue, creates the LOCAL storage element and the test scope, and starts the server. This takes a minute or two; the steps below wait for it.

Step 3: Retrieve the per-VM credentials

SSH in as azureuser and read the credentials file that first boot generated. It is readable only by root, so use sudo.

sudo cat /root/rucio-credentials.txt

You will see the reachable URL, the account (root), the administrator username (admin), the storage element and scope, and the unique per-VM administrator password. The ddmlab / secret demonstration credential shipped by upstream is not present and is rejected by the server.

Step 4: Confirm the server is healthy

The Rucio REST API answers an unauthenticated health check at /ping, which returns the running server version. Confirm the service and the two stack containers (the server and the PostgreSQL catalogue) are up.

systemctl is-active rucio.service
sudo docker ps --format '{{.Names}}\t{{.Status}}'
curl -s http://127.0.0.1/ping

The /ping response is the Rucio version ({"version":"41.1.1"}). Port 80 serves the REST API; PostgreSQL has no published host port and is reachable only on the internal Docker network.

Step 5: Explore the catalogue

The rucio and rucio-admin clients are pre-configured inside the rucio-server container. Run them with docker exec. Confirm who you are authenticated as, list the storage elements, list your scopes, and inspect the local RSE's POSIX/file protocol.

sudo docker exec -u root rucio-server rucio whoami
sudo docker exec -u root rucio-server rucio-admin rse list
sudo docker exec -u root rucio-server rucio-admin scope list --account root
sudo docker exec -u root rucio-server rucio-admin rse info LOCAL

whoami reports the root account. The LOCAL RSE uses the native rucio.rse.protocols.posix.Default implementation with the file scheme and prefix /data/local, so uploads write straight to local disk on this single node.

Step 6: Register, upload and list a data file

This is the core Rucio workflow. Create a data file, upload it to the LOCAL storage element under the test scope (which registers it as a data identifier and records a replica in the catalogue), list the data identifiers in the scope, and locate the physical replica.

sudo docker exec -u root rucio-server bash -lc 'echo "cloudimg Rucio demo data file" > /tmp/hello-rucio.txt'
sudo docker exec -u root rucio-server rucio --account root upload --rse LOCAL --scope test /tmp/hello-rucio.txt
sudo docker exec -u root rucio-server rucio --account root did list --filter 'type=all' 'test:*'
sudo docker exec -u root rucio-server rucio --account root replica list file test:hello-rucio.txt

The upload reports a successful replica and replication rule on LOCAL. The data identifier test:hello-rucio.txt then appears in the listing (files need --filter type=all because datasets and containers are listed by default), and the replica listing shows its size, its Adler-32 checksum, and its physical location on the RSE.

Step 7: Use the Rucio client from your workstation

The client inside the container is the quickest way to drive the server, but you can also install the rucio client on another host and point it at this VM. Install the client (for example pip install rucio-clients) and create a rucio.cfg with a [client] section:

# rucio.cfg on your workstation (adjust the host to your VM's address):
[client]
rucio_host = http://YOUR-VM-IP:80
auth_host = http://YOUR-VM-IP:80
auth_type = userpass
username = admin
password = <RUCIO_ADMIN_PASSWORD>
account = root

Use the RUCIO_ADMIN_PASSWORD from /root/rucio-credentials.txt. For anything beyond a private network, put the API behind TLS first (next step).

Step 8: Secure it for production

The appliance serves the REST API over plain HTTP on port 80, which is fine on a private network. Before exposing it more widely, terminate TLS in front of it. Put a reverse proxy such as Caddy or nginx with certbot in front of port 80 for your-domain, or enable the server's own SSL (RUCIO_ENABLE_SSL) with your own hostcert.pem. Then restrict inbound access in the Network Security Group to only the hosts that need it.

To grow beyond the single node, add more storage elements with rucio-admin rse add, give accounts quota on them, and use replication rules to place copies where you want them.

Notes and support

• The per-VM credentials live in /root/rucio-credentials.txt (mode 0600). Keep them safe; they are unique to each VM and are never shipped in the image.
• The Rucio stack is managed by rucio.service (sudo systemctl restart rucio.service to bounce it). The Compose project lives in /var/lib/rucio.
• No Rucio daemons (conveyor, reaper, judge) run on this single-node appliance; they are only needed for cross-RSE transfers, deletion and rule evaluation across multiple storage elements.
• The image is fully patched with unattended security upgrades enabled, and every deployment is backed by 24/7 cloudimg support.