bazel-remote on Ubuntu 24.04 on Azure User Guide

Overview

bazel-remote is a fast, self hosted remote build cache that speaks the HTTP and gRPC Remote Cache (REAPI) protocols used by Bazel, Buck and Goma. It stores Content Addressable Storage (CAS) blobs and Action Cache (AC) entries on disk with a configurable size cap and least recently used eviction, so your build and CI machines share compiled artifacts and action results instead of rebuilding them from scratch.

The cloudimg image installs the official bazel-remote single static Go binary at /usr/local/bin/bazel-remote, running as the bazel-remote service. It serves the HTTP cache on port 8080 and the gRPC cache on port 9092, with the disk cache at /var/lib/bazel-remote/cache on the OS disk.

Secure by default, no default login: bazel-remote normally starts with no authentication and binds every interface, which is unacceptable for a network cache. This image never ships that. HTTP Basic Auth is enforced on both the HTTP and the gRPC interface with --htpasswd_file. A bazel-remote-firstboot.service oneshot generates a unique username and password on each VM's first boot, writes them to a root only file, resolves the instance URL and writes a root only info note, then disables itself. No two VMs share a credential and none is baked into the image.

Fail secure: the cache server carries a bootstrap gate (ConditionPathExists) that only clears after first boot has written the per VM credential, so the cache cannot start at all until a credential exists. If first boot were interrupted the cache simply would not serve, rather than coming up open.

Always current: on first boot the appliance downloads the newest bazel-remote release binary (falling back to the baked version if the network is unavailable), so every launch runs current software rather than the version frozen at build time.

Note on the interface: bazel-remote is an API service with no web UI. You drive it through Bazel (or another REAPI client) by pointing --remote_cache at it, or through its HTTP REST API with curl. This guide uses curl to demonstrate the cache and shows the .bazelrc configuration.

What is included:

• bazel-remote (verified at 2.6.1) as the official single static Go binary
• bazel-remote.service serving HTTP on :8080 and gRPC on :9092
• HTTP Basic Auth enforced on both interfaces via --htpasswd_file
• bazel-remote-firstboot.service for the first boot credential, binary refresh and info note
• A unique per VM username and password generated on first boot, in a root only 0600 file
• A fail secure bootstrap gate so the cache never starts without a credential
• Ubuntu 24.04 LTS base, fully patched
• 24/7 cloudimg support, 24h response SLA

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet with a subnet. The cache is CPU light; performance is dominated by disk and network throughput and cache capacity. Recommended VM size: Standard_B2s (2 vCPU, 4 GB RAM) for evaluation and small teams, or a Standard_D4s_v5 or larger with a bigger data disk for busy CI fleets. The disk cache lives on the OS disk by default, so expand the disk (and raise the size cap, see the sizing section) before caching large build graphs.

Step 1: Deploy from the Azure Portal

Search the Marketplace for bazel-remote on Ubuntu 24.04, choose your VM size, and attach an NSG that allows TCP 22 (SSH) from your management network and TCP 8080 (HTTP cache) and TCP 9092 (gRPC cache) from the build and CI machines that will use the cache. Because Basic Auth is enforced, the cache is safe to expose to your build fleet, but you should still restrict the source ranges to networks you control.

Step 2: Deploy from the Azure CLI

RG="bazel-remote-prod"; LOCATION="eastus"; VM_NAME="bazel-remote-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/bazel-remote-cache-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name br-vnet --address-prefix 10.90.0.0/16 --subnet-name br-subnet --subnet-prefix 10.90.1.0/24
az network nsg create -g "$RG" --name br-nsg
az network nsg rule create -g "$RG" --nsg-name br-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name br-nsg --name allow-cache --priority 110 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 8080 9092 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_B2s --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name br-vnet --subnet br-subnet --nsg br-nsg --public-ip-sku Standard

Step 3: Connect via SSH

ssh azureuser@<vm-ip>

Step 4: Verify the service and listeners

bazel-remote serves the HTTP cache on :8080 and the gRPC cache on :9092. Every request requires authentication, so an unauthenticated call to the status endpoint returns 401.

sudo systemctl start bazel-remote 2>/dev/null || true
sudo systemctl is-active bazel-remote
ss -tln | grep -E ':8080|:9092'
echo "unauthenticated /status -> HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/status)"

You should see the service active, both listening sockets, and HTTP 401 from the unauthenticated status call, which confirms authentication is enforced.

Step 5: Read your per VM cache credentials

The username and password were generated on this VM's first boot and stored in a root only file. Read them, keep them secret, and confirm that the per VM credential authenticates while an unauthenticated request does not.

sudo cat /root/bazel-remote-info.txt
U=$(sudo grep '^BAZEL_REMOTE_USER=' /root/bazel-remote-info.txt | cut -d= -f2-)
P=$(sudo grep '^BAZEL_REMOTE_PASSWORD=' /root/bazel-remote-info.txt | cut -d= -f2-)
echo "unauthenticated -> HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/status)"
echo "authenticated   -> HTTP $(curl -s -o /dev/null -w '%{http_code}' -u "$U:$P" http://127.0.0.1:8080/status)"

The info note holds BAZEL_REMOTE_USER, BAZEL_REMOTE_PASSWORD and the cache URL, is owned root:root with mode 0600, and the status endpoint returns 401 without the credential and 200 with it. Basic Auth is enforced identically on the gRPC interface.

Step 6: Store and fetch an artifact (CAS round trip)

The Content Addressable Storage keys every blob by the SHA-256 of its content. Store a blob with an authenticated PUT, then fetch it straight back and confirm it matches byte for byte. This is exactly what Bazel does under the hood for every cached artifact.

U=$(sudo grep '^BAZEL_REMOTE_USER=' /root/bazel-remote-info.txt | cut -d= -f2-)
P=$(sudo grep '^BAZEL_REMOTE_PASSWORD=' /root/bazel-remote-info.txt | cut -d= -f2-)
BLOB="hello from bazel-remote at $(date -u +%FT%TZ)"
HASH=$(printf '%s' "$BLOB" | sha256sum | awk '{print $1}')
printf '%s' "$BLOB" | curl -s -u "$U:$P" -X PUT --data-binary @- "http://127.0.0.1:8080/cas/$HASH" -o /dev/null -w 'PUT /cas -> HTTP %{http_code}\n'
GOT=$(curl -s -u "$U:$P" "http://127.0.0.1:8080/cas/$HASH")
[ "$GOT" = "$BLOB" ] && echo "GET /cas -> byte for byte match" || echo "GET /cas -> MISMATCH"
echo "unauthenticated GET /cas -> HTTP $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8080/cas/$HASH)"

The PUT returns 200, the authenticated GET returns the identical blob, and an unauthenticated GET of the same key returns 401, so cached content is never served without the credential.

Step 7: Point Bazel at the cache

Add the cache to your project's .bazelrc, or pass it on the command line. Use the username and password from Step 5. Over HTTP, the credentials go in the URL:

# .bazelrc — HTTP remote cache (username:password from /root/bazel-remote-info.txt)
build --remote_cache=http://<cache-username>:<cache-password>@<vm-ip>:8080
# optional: keep going if the cache is briefly unreachable
build --remote_local_fallback

To use the gRPC interface instead, point Bazel at grpc://<vm-ip>:9092 and pass the same credentials as a Basic Auth header:

# .bazelrc — gRPC remote cache. Compute the header once:
#   echo -n "<cache-username>:<cache-password>" | base64
build --remote_cache=grpc://<vm-ip>:9092
build --remote_header=authorization="Basic <base64-of-username:password>"

Run any build twice: the first populates the cache, and a clean second build (bazel clean then bazel build //...) pulls the results back from the cache instead of recompiling. Share the same cache across your CI fleet so every machine benefits.

Step 8: Sizing the cache

The cache size is capped by BAZEL_REMOTE_MAX_SIZE_GIB in /etc/bazel-remote/bazel-remote.env (default 10 GiB), with least recently used eviction once the cap is reached. Inspect the current setting and disk usage:

grep -E 'MAX_SIZE|DIR' /etc/bazel-remote/bazel-remote.env
du -sh /var/lib/bazel-remote/cache 2>/dev/null
df -h /var/lib/bazel-remote

To grow the cache, raise BAZEL_REMOTE_MAX_SIZE_GIB and sudo systemctl restart bazel-remote. For large caches, attach a dedicated data disk, mount it at /var/lib/bazel-remote/cache (preserving bazel-remote ownership), and raise the cap to match the disk. Keep the cap comfortably below the disk size to leave room for in flight evictions.

Step 9: Optional TLS

The cache enforces Basic Auth, which protects the credential over the wire only if the transport is encrypted, so restrict the NSG source ranges to networks you control and, for traffic across untrusted networks, enable TLS. bazel-remote serves HTTPS and gRPCS natively when you supply a certificate and key:

# add to the service and restart (see the bazel-remote docs for --tls_cert_file / --tls_key_file)
#   --tls_cert_file /etc/bazel-remote/tls/cert.pem --tls_key_file /etc/bazel-remote/tls/key.pem
# clients then use https://... and grpcs://... ; supply the CA to Bazel with --tls_certificate
echo "See https://github.com/buchgr/bazel-remote for the TLS and mTLS flags."

With TLS enabled, point Bazel at https://<vm-ip>:8080 or grpcs://<vm-ip>:9092 and pass the certificate authority to Bazel with --tls_certificate.

Persistence, first boot and updates

The disk cache lives on the OS disk under /var/lib/bazel-remote/cache. The first boot service generates the per VM username and password, refreshes the bazel-remote binary to the latest release, writes the fail secure bootstrap marker, then disables itself so subsequent reboots are unaffected. The OS ships fully patched with unattended security upgrades enabled.

sudo /usr/local/bin/bazel-remote --help 2>/dev/null | head -1
systemctl is-enabled bazel-remote-firstboot.service bazel-remote.service
ls -ld /var/lib/bazel-remote/cache
export DEBIAN_FRONTEND=noninteractive; sudo apt-get update -y >/dev/null 2>&1
echo "held packages: $(apt-mark showhold | tr '\n' ' ')[none]"

Support

Every cloudimg deployment includes 24/7 support with a 24 hour response SLA. Contact support@cloudimg.co.uk for help with deployment, configuration or scaling.

This is a repackaged open source software product with additional charges for cloudimg support services. bazel-remote and Bazel are trademarks of their respective owners. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.