Afanasy 3.4.1 on AWS User Guide

Overview

This guide covers launching and using the Afanasy 3.4.1 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg media servers listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Afanasy is the render farm manager from CGRU, the open source computer graphics tools pack. Its server, afserver, keeps the queue: artists submit jobs (a Blender, Houdini, Nuke or Maya render split into frame ranges, or any command line), render nodes running afrender connect to the server and pull tasks, and everyone watches progress in the browser based Web Monitor: jobs and their tasks, the render farm and its pools, users and connected monitors.

Afanasy itself has no login. Anyone who can reach afserver can see and control the whole farm, and a job can run commands on the server and on every render node. The cloudimg image is built around that fact:

• afserver listens on loopback only. Upstream afserver always listens on every address; cloudimg patches it to listen on the address in /etc/default/afserver, which ships as 127.0.0.1. Nothing outside the instance reaches it until you deliberately admit your render nodes from your VPC (Step 8).
• HTTPS from the first boot. nginx serves the Web Monitor and its API on port 443 with a certificate generated on your instance's first boot. Port 80 redirects to HTTPS, and requests for host names the instance does not serve are dropped.
• A login unique to your instance. The Web Monitor sits behind a username and password generated on first boot and written to /etc/cloudimg-credentials.txt. No password is baked into the image, and the stored copy is a bcrypt hash.
• Fail closed. afserver refuses to start until first boot has created the login and the certificate, and nginx refuses to start before first boot has finished.
• Cross site requests refused. A web page on another site cannot drive your farm through a signed in browser: state changing requests from a foreign origin are rejected, and sign in attempts are rate limited.
• Hardened defaults. afserver, and every job pre command it runs, works as an unprivileged afanasy user in a systemd sandbox where only the farm data is writable. afserver, nginx and the first boot service are all blocked from the instance metadata service, the AMI requires IMDSv2, and no compiler ships on the image.
• Farm data on its own volume. Jobs, users, pools and render node records live on a dedicated EBS volume mounted at /var/lib/cgru, which you can resize or snapshot on its own.
• Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: afserver 3.4.1 built from the official CGRU 3.4.1 source (checksum verified, server only, with the cloudimg listen address patch), the Afanasy Web Monitor, nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent. The CGRU source, including the patch, is in /opt/cgru.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet.
• A security group that allows:
• TCP 22 (SSH) from your own networks only
• TCP 443 (the Web Monitor) from the networks your artists and administrators work from
• TCP 80 (redirects to HTTPS) from the same sources as 443

Do not open TCP 51000 here. Render nodes reach afserver on that port only after Step 8, and only from their own security group.

afserver is a light C++ service: the queue for a farm of dozens of nodes fits easily on an m5.large. The rendering itself happens on your render nodes, so size those for your renderer.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Afanasy 3.4.1 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above. The AMI includes the farm data volume, so leave its block device mapping in place.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=afanasy}]'

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot generates the certificate and the login, then afserver and nginx start:

systemctl is-active cgru-firstboot afserver nginx
sudo test -f /var/lib/cloudimg/cgru-firstboot.done && echo FIRSTBOOT_DONE

All three units are active and the first boot marker is present:

active
active
active
FIRSTBOOT_DONE

nginx is the only service on public addresses (80 and 443); afserver listens on loopback:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | sed -E 's/pid=[0-9]+,fd=[0-9]+/.../g' | sort

0.0.0.0:22 users:(("sshd",...),("systemd",...))
0.0.0.0:443 users:(("nginx",...),("nginx",...),("nginx",...))
0.0.0.0:80 users:(("nginx",...),("nginx",...),("nginx",...))
127.0.0.1:51000 users:(("afserver",...))
127.0.0.53%lo:53 users:(("systemd-resolve",...))
127.0.0.54:53 users:(("systemd-resolve",...))
[::]:22 users:(("sshd",...),("systemd",...))
[::]:443 users:(("nginx",...),("nginx",...),("nginx",...))
[::]:80 users:(("nginx",...),("nginx",...),("nginx",...))

Step 4: Get your URL and login

sudo cat /etc/cloudimg-credentials.txt

The file is readable by root only:

# Afanasy 3.4.1 - generated on first boot by cgru-firstboot.service
# The Web Monitor login is unique to this instance; none was baked into the image. Keep this file secret.
AFANASY_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/
AFANASY_WEB_USER=afadmin
AFANASY_WEB_PASSWORD=<AFANASY_WEB_PASSWORD>
# afserver listens on 127.0.0.1:51000 only (render nodes: see the guide, "Connect render nodes").
# Change the name the Web Monitor is served as:  sudo afanasy-set-hostname <your-dns-name>
# User guide: https://www.cloudimg.co.uk/guides/afanasy-3-4-1-on-ubuntu-24-04-aws/

The URL uses the instance's public DNS name from the EC2 instance metadata. The certificate also names the public and private IP addresses, so https://<public-ip>/ works too. To use your own DNS name, see Step 9. When you need to share the file, for example in a support request, print it with the password masked:

sudo sed -E 's/^(AFANASY_WEB_PASSWORD=).*/\1********/' /etc/cloudimg-credentials.txt | grep '^AFANASY_'

Step 5: Check the server from the command line

Port 80 redirects to HTTPS, the health check is open for load balancers, and a host name the instance does not serve gets no answer:

H=$(sudo grep '^AFANASY_URL=' /etc/cloudimg-credentials.txt | sed -E 's#^AFANASY_URL=https://([^/]+)/.*#\1#')
curl -s -o /dev/null -w 'http://      -> %{http_code} %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/"
curl -sk -o /dev/null -w 'healthz      -> HTTP %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/healthz"
curl -sk -o /dev/null -w 'unknown host -> HTTP %{http_code}\n' -H 'Host: other.example' https://127.0.0.1/ || true

http://      -> 301 https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/
healthz      -> HTTP 200
unknown host -> HTTP 000

The Web Monitor refuses a request with no password or a wrong one, and accepts this instance's login. The password is passed to curl on standard input, so it never appears in the process list or your shell history:

sudo bash -c '
  F=/etc/cloudimg-credentials.txt
  P=$(grep "^AFANASY_WEB_PASSWORD=" $F | cut -d= -f2-)
  H=$(grep "^AFANASY_URL=" $F | sed -E "s#^AFANASY_URL=https://([^/]+)/.*#\1#")
  R="--resolve $H:443:127.0.0.1"
  curl -sk $R -o /dev/null -w "no password         -> HTTP %{http_code}\n" "https://$H/"
  printf "user = \"afadmin:wrong\"\n" | curl -K - -sk $R -o /dev/null -w "wrong password      -> HTTP %{http_code}\n" "https://$H/"
  printf "user = \"afadmin:%s\"\n" "$P" | curl -K - -sk $R -o /dev/null -w "per-instance login  -> HTTP %{http_code}\n" "https://$H/"
'

no password         -> HTTP 401
wrong password      -> HTTP 401
per-instance login  -> HTTP 200

The same login protects afserver's JSON API, which the Web Monitor and your scripts use. This lists the jobs on the farm; a new farm holds only afserver's own system job, afanasy:

sudo bash -c '
  F=/etc/cloudimg-credentials.txt
  P=$(grep "^AFANASY_WEB_PASSWORD=" $F | cut -d= -f2-)
  H=$(grep "^AFANASY_URL=" $F | sed -E "s#^AFANASY_URL=https://([^/]+)/.*#\1#")
  Q="{\"get\":{\"type\":\"jobs\"}}"
  printf "user = \"afadmin:%s\"\n" "$P" | curl -K - -sk --resolve "$H:443:127.0.0.1" -X POST \
    -H "AFANASY: ${#Q}" -H "Origin: https://$H" --data-binary "$Q" "https://$H/" | jq -r ".jobs[].name"
'

afanasy

Step 6: Use the Web Monitor

Open AFANASY_URL in your browser and sign in as afadmin with the password from Step 4. The certificate is self signed, so your browser asks you to confirm it the first time (or install your own, Step 9).

Jobs lists every job with its state, owner, priority and frame range. The afanasy job is afserver's own system job, which runs post commands, wake on LAN and events; demo-render here is a 24 frame job waiting for render nodes. The buttons on the left start, pause, stop, restart and delete the selected job:

Double click a job to open its tasks. Each frame is a task with its state, error count and starts; SKIP, RESTART and CONTINUE act on the selected tasks:

Farm shows the render pools and the render nodes connected to them, with controls to add pools, pause nodes, mark them busy (NIMBY), eject tasks and run services. A new farm has only the root pool / until render nodes connect (Step 8):

Users lists everyone who has submitted jobs, with their priority, error handling and how their jobs are ordered:

Monitors lists the connected Web Monitor sessions. Every browser session arrives from 127.0.0.1 because nginx is the only path to afserver; the real client addresses are in /var/log/nginx/access.log:

A note on Afanasy's own permissions: the Web Monitor's VISOR and GOD modes, which let a user manage other users' jobs and the farm, are switches in the browser, not passwords checked by the server. Treat everyone who has the Web Monitor login as a farm administrator.

Step 7: Submit a job

Artists normally submit from their applications with CGRU's submitters or the af Python module on a workstation that can reach afserver (Step 8). You can also submit through the authenticated API. This sends a small two frame test job, lists it, and deletes it again; with no render nodes connected its tasks wait in the queue:

sudo bash -c '
  F=/etc/cloudimg-credentials.txt
  P=$(grep "^AFANASY_WEB_PASSWORD=" $F | cut -d= -f2-)
  H=$(grep "^AFANASY_URL=" $F | sed -E "s#^AFANASY_URL=https://([^/]+)/.*#\1#")
  api() { printf "user = \"afadmin:%s\"\n" "$P" | curl -K - -sk --resolve "$H:443:127.0.0.1" -X POST \
            -H "AFANASY: ${#1}" -H "Origin: https://$H" --data-binary "$1" "https://$H/"; }
  J="{\"job\":{\"name\":\"hello-farm\",\"user_name\":\"afadmin\",\"host_name\":\"guide\",\"blocks\":[{\"name\":\"frames\",\"service\":\"generic\",\"capacity\":1000,\"flags\":1,\"frame_first\":1,\"frame_last\":2,\"frames_per_task\":1,\"frames_inc\":1,\"command\":\"echo frame @#@\"}]}}"
  ID=$(api "$J" | jq -r ".id")
  echo "submitted job id $ID"
  api "{\"get\":{\"type\":\"jobs\"}}" | jq -r ".jobs[] | select(.name==\"hello-farm\") | \"\(.name): \(.blocks[0].tasks_num) tasks\""
  api "{\"action\":{\"type\":\"jobs\",\"ids\":[$ID],\"user_name\":\"afadmin\",\"host_name\":\"guide\",\"operation\":{\"type\":\"delete\"}}}" | jq -r ".info.text"
'

submitted job id 2
hello-farm: 2 tasks
Action processed.

Step 8: Connect render nodes

Render nodes, and workstations that submit with CGRU's own tools, connect straight to afserver on TCP 51000. That protocol has no authentication, so admit only machines you control and only from inside your VPC.

1. Find the instance's private address:

jq -r '.ds["meta-data"]["local-ipv4"]' /run/cloud-init/instance-data.json

1. Make afserver listen on that address. Edit /etc/default/afserver and set AF_SERVER_BIND_ADDRESS to the private address, then restart afserver:

sudo sed -i 's/^AF_SERVER_BIND_ADDRESS=.*/AF_SERVER_BIND_ADDRESS=<private-ip>/' /etc/default/afserver
sudo systemctl restart afserver
sudo ss -lntp | grep afserver

1. Allow TCP 51000 into this instance's security group only from the render nodes' security group (never from 0.0.0.0/0):

aws ec2 authorize-security-group-ingress --group-id <afanasy-server-sg> \
  --protocol tcp --port 51000 --source-group <render-nodes-sg>

1. On each render node, install CGRU 3.4.1 (the same release as the server), point it at the server in <cgru-location>/config.json and start afrender:

{"cgru_config": {"af_servername": "<private-ip>", "af_serverport": 51000}}

<cgru-location>/start/AFANASY/_afrender.sh

The node appears in Farm within a few seconds and starts taking tasks. Render nodes run job commands as the user that runs afrender, so run it as a dedicated unprivileged user on each node.

To close the server again, set AF_SERVER_BIND_ADDRESS=127.0.0.1, restart afserver and remove the security group rule. afserver refuses to start if the address is empty or not an IPv4 address.

Step 9: Use your own DNS name and certificate

Point a DNS record at the instance (an Elastic IP keeps the address stable), then tell the instance its name. This regenerates the self signed certificate for that name, updates the credentials file and reloads nginx:

sudo afanasy-set-hostname farm.your-domain.example

To use a certificate from a public CA instead, replace /etc/nginx/ssl/afanasy.crt (full chain) and /etc/nginx/ssl/afanasy.key (mode 0600) and run sudo systemctl reload nginx.

Step 10: Security checks

afserver listens on loopback and cannot start without the first boot login; afserver, nginx and the first boot service are blocked from the instance metadata service:

grep '^AF_SERVER_BIND_ADDRESS=' /etc/default/afserver
systemctl show afserver -p Requires --value
systemctl show afserver nginx cgru-firstboot -p IPAddressDeny --value

AF_SERVER_BIND_ADDRESS=127.0.0.1
-.mount var-lib-cgru.mount cgru-firstboot.service sysinit.target system.slice
169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

afserver runs unprivileged in a read only system with only the farm data writable, and there is no compiler on the image:

systemctl show afserver -p User -p ProtectSystem -p NoNewPrivileges -p ReadWritePaths
command -v gcc g++ make cmake || echo "no compiler toolchain on the image"

User=afanasy
ReadWritePaths=/var/lib/cgru
ProtectSystem=strict
NoNewPrivileges=yes
no compiler toolchain on the image

Maintenance

• Configuration: /opt/cgru/config.json (port, store folder) and /etc/default/afserver (listen address). Apply changes with sudo systemctl restart afserver. CGRU's full list of settings is in /opt/cgru/afanasy/config_default.json; put overrides in /opt/cgru/config.json rather than editing the defaults.
• Changing the Web Monitor password: printf '%s\n' '<new-password>' | sudo htpasswd -i -B -C 12 /etc/nginx/.htpasswd afadmin, then update /etc/cloudimg-credentials.txt. First boot runs only once, so it never overwrites your change. Add more people with the same command and their own user names.
• Farm data: jobs, users, pools and render node records are in /var/lib/cgru/store on the data volume. Snapshot that volume to back up the farm; stop afserver first for a consistent copy.
• Logs: afserver logs to the journal (journalctl -u afserver); nginx, with the real client addresses, logs to /var/log/nginx/.
• Security updates: unattended security upgrades are on. Reboot when a new kernel is installed.

Support

cloudimg provides 24/7/365 expert technical support for this image. Contact support@cloudimg.co.uk. Afanasy is part of CGRU, developed by the CGRU team and licensed under the GNU Lesser General Public License version 3; the corresponding source, including the cloudimg listen address patch in afanasy/src/server/threadacceptclient.cpp, is on the image in /opt/cgru and upstream at github.com/CGRU/cgru. This image is provided by cloudimg and is not affiliated with or endorsed by the CGRU project.