Bazel Buildfarm on Ubuntu 24.04 on Azure User Guide

Overview

Bazel Buildfarm is an open source remote build cache and remote execution service that implements Bazel's Remote Execution API (REAPI). When your Bazel builds point at the farm, build actions are cached centrally and executed on the farm's worker instead of on every developer machine and CI runner, so a shared cache and offloaded compilation make incremental builds dramatically faster and CI cheaper and more consistent.

The cloudimg image delivers a complete single node farm, fully installed and wired together, so a working Remote Execution endpoint is reachable within minutes of launch. It runs three managed services on one node: a Redis SHARD backplane bound to the loopback interface, a Buildfarm server that exposes the Remote Execution API on gRPC port 8980 (the endpoint your clients dial), and a Buildfarm worker on gRPC port 8981 that executes actions.

Secure by default, no default secret: Buildfarm 2.x is a sharded architecture, so it needs a Redis backplane that holds the content addressable storage, the action cache and the execution queue. This image generates a unique backplane secret on each VM's first boot and writes it to a root only file, and Redis is bound to the loopback interface and protected by that secret. No two VMs share a credential and none is baked into the image.

Fail secure: the server and worker both carry a bootstrap gate that only clears after first boot has minted the per VM secret and rendered the configuration, so the farm cannot start until a secret exists. If first boot were interrupted the farm simply would not serve, rather than coming up with a build time or default secret. A firewall guard keeps the internal worker port and the backplane off the public network while leaving the server endpoint your build fleet uses reachable.

Note on the interface: Buildfarm is a gRPC API service with no web UI. You drive it from Bazel by pointing --remote_executor and --remote_cache at it. This guide uses the bazel client (through bazelisk) to demonstrate a real build offloaded to the farm.

What is included:

• Bazel Buildfarm (verified at 2.17.1) as the official buildfarm-server and buildfarm-worker container images, run under systemd
• A Redis SHARD backplane, bound to loopback and protected by a per VM secret
• buildfarm-server.service exposing the Remote Execution API on :8980 and buildfarm-worker.service executing actions on :8981
• buildfarm-firstboot.service that mints the per VM backplane secret, renders the config and writes the fail secure bootstrap marker
• A fail secure gate so the server and worker never start without the per VM secret
• A firewall guard that drops off box access to the internal worker port and the backplane
• Ubuntu 24.04 LTS base, fully patched, with a swapfile sized for the two JVMs and Redis
• 24/7 cloudimg support, 24h response SLA

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet with a subnet. Recommended VM size: Standard_B2s (2 vCPU, 4 GB RAM) for evaluation and small teams, or a Standard_D4s_v5 or larger for busy CI fleets that run many parallel actions. The worker executes actions on the node, so more vCPU and RAM directly raise how much work the farm can run at once.

Step 1: Deploy from the Azure Portal

Search the Marketplace for Bazel Buildfarm on Ubuntu 24.04, choose your VM size, and attach an NSG that allows TCP 22 (SSH) from your management network and TCP 8980 (the Remote Execution endpoint) from the build and CI machines that will use the farm. Only the server port 8980 needs to be reachable by clients; the worker port 8981 and the Redis backplane are internal and are kept off the public network by the built in firewall guard.

Step 2: Deploy from the Azure CLI

RG="buildfarm-prod"; LOCATION="eastus"; VM_NAME="buildfarm-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/buildfarm-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name bf-vnet --address-prefix 10.90.0.0/16 --subnet-name bf-subnet --subnet-prefix 10.90.1.0/24
az network nsg create -g "$RG" --name bf-nsg
az network nsg rule create -g "$RG" --nsg-name bf-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name bf-nsg --name allow-farm --priority 110 \
  --source-address-prefixes "<your-build-cidr>" --destination-port-ranges 8980 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_B2s --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name bf-vnet --subnet bf-subnet --nsg bf-nsg --public-ip-sku Standard

Step 3: Connect via SSH

ssh azureuser@<vm-ip>

Step 4: Verify the services and listeners

The farm is three services: the Redis backplane, the Buildfarm server on :8980, and the Buildfarm worker on :8981. Confirm all three are active, the server and worker are listening, and the loopback backplane requires the per VM secret.

sudo systemctl is-active redis-server buildfarm-server buildfarm-worker
ss -tln | grep -E ':8980|:8981'
echo "redis (no credential) -> $(redis-cli -h 127.0.0.1 ping 2>&1)"

You should see all three services active, listeners on 8980 and 8981, and NOAUTH Authentication required from the unauthenticated Redis ping, which confirms the backplane is protected.

Step 5: Run a real build on the farm (remote execution)

Create a tiny Bazel workspace and build it, pointing Bazel at the local farm. The build action is executed on the Buildfarm worker rather than on this machine, which Bazel reports as a remote process.

mkdir -p /tmp/buildfarm-demo && cd /tmp/buildfarm-demo
printf 'module(name = "demo")\n' > MODULE.bazel
printf 'genrule(name = "demo", outs = ["demo.txt"], cmd = "echo built-on-the-farm > $@")\n' > BUILD.bazel
echo 7.4.1 > .bazelversion
bazelisk build //:demo \
  --remote_executor=grpc://127.0.0.1:8980 \
  --remote_cache=grpc://127.0.0.1:8980 \
  --remote_instance_name=shard
cat bazel-bin/demo.txt

Bazel reports the genrule as 1 remote in its process summary (INFO: ... processes: ... 1 remote.), the build completes successfully, and demo.txt holds the artifact the worker produced. The first time you run bazelisk it downloads the pinned Bazel release, so allow a few extra seconds. If the worker has only just started it can take a few seconds to register with the backplane; re run the build and the action is dispatched remotely.

Step 6: Rebuild from the remote cache

Because the action result is now stored on the farm, a fresh build of the same target is served from the remote cache instead of being re executed. Clean the local state and build again.

cd /tmp/buildfarm-demo
bazelisk clean
bazelisk build //:demo \
  --remote_executor=grpc://127.0.0.1:8980 \
  --remote_cache=grpc://127.0.0.1:8980 \
  --remote_instance_name=shard

Bazel now reports 1 remote cache hit in its process summary, proving the result came back from the farm rather than being rebuilt. This is exactly how a shared cache speeds up your team: once one machine builds a target, every other machine and CI runner pulls the result from the farm.

Step 7: Point your own project at the farm

Add the farm to your project's .bazelrc, or pass the flags on the command line. The endpoint and instance name for this VM are in the root only info note.

sudo grep -E '^BUILDFARM_ENDPOINT|^BUILDFARM_INSTANCE_NAME' /root/buildfarm-info.txt

Then, in your project's .bazelrc, point Bazel at the server (use the VM's reachable address for <vm-ip>):

# .bazelrc — offload build actions to the farm and share the cache
build --remote_executor=grpc://<vm-ip>:8980
build --remote_cache=grpc://<vm-ip>:8980
build --remote_instance_name=shard
# optional: keep building locally if the farm is briefly unreachable
build --remote_local_fallback

Run any build twice across your fleet: the first populates the cache and offloads execution, and subsequent builds on any machine reuse the cached results. For traffic across untrusted networks, restrict the NSG to networks you control and terminate TLS in front of the endpoint, or enable Buildfarm's own gRPC TLS (see the Buildfarm documentation).

Step 8: The per VM backplane secret and security

The backplane secret is generated on this VM's first boot, written to a root only file, and used to protect the loopback Redis backplane. The server and worker are gated on a bootstrap marker that first boot writes only after the secret exists, so the farm is fail secure.

sudo stat -c '%n perms=%a owner=%U:%G' /root/buildfarm-info.txt
PW=$(sudo grep '^BUILDFARM_REDIS_PASSWORD=' /root/buildfarm-info.txt | cut -d= -f2-)
echo "redis (no credential) -> $(redis-cli -h 127.0.0.1 ping 2>&1)"
echo "redis (per-VM secret) -> $(redis-cli -h 127.0.0.1 -a "$PW" ping 2>/dev/null)"

The info note is 0600 root:root, the unauthenticated ping is rejected with NOAUTH, and the per VM secret returns PONG. The internal worker port 8981 and Redis 6379 are dropped off box by an nftables guard, while the server endpoint 8980 stays reachable for your clients.

Sizing, persistence and first boot

The worker's content addressable storage lives on the OS disk under /var/lib/buildfarm/worker with a size cap; raise the cap or attach a larger disk for bigger build graphs. Scale the farm by choosing a larger VM size so the worker can execute more actions in parallel. The first boot service generates the per VM secret, renders the configuration and writes the fail secure marker, then disables itself so subsequent reboots are unaffected. The OS ships fully patched with unattended security upgrades enabled.

sudo systemctl is-enabled buildfarm-firstboot.service buildfarm-server.service buildfarm-worker.service
sudo docker ps --format '  {{.Names}}  {{.Status}}'
export DEBIAN_FRONTEND=noninteractive; sudo apt-get update -y >/dev/null 2>&1
echo "held packages: $(apt-mark showhold | tr '\n' ' ')[none]"

Support

Every cloudimg deployment includes 24/7 support with a 24 hour response SLA. Contact support@cloudimg.co.uk for help with deployment, configuration or scaling.

This is a repackaged open source software product with additional charges for cloudimg support services. Bazel and Bazel Buildfarm are trademarks of their respective owners. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.