bazel-remote 2.6.3 on AWS User Guide

Overview

This guide covers launching and using the bazel-remote 2.6.3 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg artifact and container registries listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

bazel-remote is a remote build cache for Bazel and other clients of the Remote Execution API cache protocol. Your developers' machines and CI runners upload the outputs of the build actions they run, and every later build that runs the same action, anywhere, downloads the result instead of rebuilding it. Teams typically see clean and CI builds go from minutes to seconds once the cache is warm.

Upstream bazel-remote listens on plain HTTP and gRPC on every interface with no authentication, so anyone who can reach it can read your build outputs or, worse, write poisoned ones that your builds will trust. The cloudimg image makes it a production cache:

• TLS from the first boot, for both protocols. nginx serves the cache on port 443 with a certificate generated on your instance's first boot, naming the instance's DNS name and IP addresses: the gRPC cache at grpcs://<host> and the HTTP cache at https://<host>/. bazel-remote itself listens on 127.0.0.1 only, and nothing listens on port 80.
• A password unique to your instance, for every request. Reads and writes, over gRPC and HTTP, need HTTP Basic authentication as the user bazel, with a password generated on the first boot and written to /root/bazel-remote-info.txt (root only). bazel-remote stores only a bcrypt hash of it. No password is baked into the image, and unauthenticated reads are never allowed.
• Fail closed. bazel-remote and nginx refuse to start until first boot has generated the certificate and the password, and a guard refuses settings that would turn on the unauthenticated metrics endpoint, profiling or a listener off loopback.
• Locked down serving. Requests for host names the instance does not serve are dropped; bazel-remote, nginx and the first boot service are blocked from the instance metadata service, and the AMI requires IMDSv2.
• The cache on its own disk. The cache lives on a separate 20 GiB EBS volume mounted at /var/lib/bazel-remote without device files, set-uid programs or execution. First boot grows it to the size you choose at launch and caps the cache at 80% of it, evicting the least recently used entries above that.
• The official 2.6.3 release, verified against its published SHA-256 at build time, and fully patched Ubuntu 24.04 LTS, with unattended security upgrades switched on.

What is on the image: bazel-remote 2.6.3 (the official release binary in /opt/bazel-remote, licence in /usr/share/doc/bazel-remote), nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent. Bazel is not on the image: you run it on your workstations and CI runners.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet your build machines can reach.
• A security group that allows TCP 22 (SSH) from your own networks only, and TCP 443 (the cache) from the networks of your developers and CI runners. Nothing else needs to be open.
• Bazel on the machines that build (any current release; the examples use Bazel 8.8.1).

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick bazel-remote 2.6.3 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair, and a security group as above. The AMI carries the 20 GiB cache volume; make it as large as your cache should be on the launch page and first boot uses all of it (you can also grow it later, Step 9).

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=bazel-remote}]'

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot generates the certificate, sizes the cache and generates the password, then bazel-remote and nginx start:

systemctl is-active bazel-remote nginx
sudo test -f /var/lib/cloudimg/bazel-remote-firstboot.done && echo FIRSTBOOT_DONE

active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on the network. bazel-remote's HTTP (8080) and gRPC (9092) listeners are on loopback behind nginx, and the DNS resolver listens on loopback only:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | sed -E 's/pid=[0-9]+,fd=[0-9]+/.../g' | sort

0.0.0.0:22 users:(("sshd",...),("systemd",...))
0.0.0.0:443 users:(("nginx",...),("nginx",...),("nginx",...))
127.0.0.1:8080 users:(("bazel-remote",...))
127.0.0.1:9092 users:(("bazel-remote",...))
127.0.0.53%lo:53 users:(("systemd-resolve",...))
127.0.0.54:53 users:(("systemd-resolve",...))
[::]:22 users:(("sshd",...),("systemd",...))
[::]:443 users:(("nginx",...),("nginx",...),("nginx",...))

Step 4: Get your URLs and password

The credentials file is readable by root only. This prints it with the password hidden; sudo cat /root/bazel-remote-info.txt shows the password itself:

sudo sed -E 's/^(BAZEL_REMOTE_PASSWORD=).*/\1<hidden>/' /root/bazel-remote-info.txt

# bazel-remote 2.6.3 - generated on first boot by bazel-remote-firstboot.service
# This password is unique to this instance; none was baked into the image. bazel-remote keeps only a bcrypt hash
# of it in /etc/bazel-remote/htpasswd. Every request (reads too) needs it, over HTTPS and gRPC. Keep this file secret.

BAZEL_REMOTE_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/
BAZEL_REMOTE_GRPC_URL=grpcs://ec2-<public-ip-dashed>.compute-1.amazonaws.com
BAZEL_REMOTE_USER=bazel
BAZEL_REMOTE_PASSWORD=<hidden>
BAZEL_REMOTE_CA_CERT=/etc/nginx/tls/bazel-remote.crt
BAZEL_REMOTE_CERT_SHA256=BB:33:29:65:3B:D3:AB:36:57:4C:5D:DF:C1:9D:0F:19:85:D5:F6:32:1E:9C:D9:D8:58:93:47:32:C2:14:AF:B9
BAZEL_REMOTE_MAX_SIZE_GIB=15
#
# On a Bazel client, with a copy of the certificate above (bazel-remote.crt), in a .bazelrc you keep private:
#   build --remote_cache=grpcs://ec2-<public-ip-dashed>.compute-1.amazonaws.com
#   build "--remote_header=Authorization=Basic <base64 of bazel:<BAZEL_REMOTE_PASSWORD>>"
#   build --tls_certificate=/path/to/bazel-remote.crt
# Change the password:          sudo bazel-remote-set-password
# Add a user:                   sudo bazel-remote-set-password --user <name>
# Change the name it serves as: sudo bazel-remote-set-hostname <your-dns-name>
# User guide: https://www.cloudimg.co.uk/guides/bazel-remote-2-6-3-on-ubuntu-24-04-aws/

The URLs use the instance's public DNS name from EC2 instance metadata, which resolves to the private address inside your VPC. The certificate also names the public and private IP addresses, so grpcs://<private-ip> works too. BAZEL_REMOTE_CERT_SHA256 is the certificate's fingerprint, to compare against the copy you give your build machines. To serve under your own DNS name, see Step 8.

Step 5: Try the cache from the instance

The HTTP cache stores a file under its SHA-256 at /cas/<sha256>. Upload a small file and read it back, with the password handed to curl on standard input so it never appears in the process list:

sudo bash -c '
  P=$(grep "^BAZEL_REMOTE_PASSWORD=" /root/bazel-remote-info.txt | cut -d= -f2-)
  auth() { printf "user = \"bazel:%s\"\n" "$P"; }
  C=/etc/nginx/tls/bazel-remote.crt
  echo "hello from the cloudimg bazel-remote guide" > /tmp/blob.txt
  H=$(sha256sum /tmp/blob.txt | cut -d" " -f1)
  auth | curl -K - -s --cacert $C -o /dev/null -w "upload   PUT /cas/<sha256> -> HTTP %{http_code}\n" -X PUT --data-binary @/tmp/blob.txt https://127.0.0.1/cas/$H
  auth | curl -K - -s --cacert $C https://127.0.0.1/cas/$H
  rm -f /tmp/blob.txt'

upload   PUT /cas/<sha256> -> HTTP 200
hello from the cloudimg bazel-remote guide

The /status endpoint (it needs the password too) reports the cache's size and the running release:

sudo bash -c 'P=$(grep "^BAZEL_REMOTE_PASSWORD=" /root/bazel-remote-info.txt | cut -d= -f2-); printf "user = \"bazel:%s\"\n" "$P" | curl -K - -s --cacert /etc/nginx/tls/bazel-remote.crt https://127.0.0.1/status' | python3 -m json.tool

{
    "CurrSize": 110592,
    "UncompressedSize": 106496,
    "ReservedSize": 0,
    "MaxSize": 16106127360,
    "NumFiles": 11,
    "ServerTime": 1792449164,
    "GitCommit": "985cc081b1d5299e9d651ab01a605362f6bb170c",
    "GitTags": "v2.6.3",
    "NumGoroutines": 11
}

MaxSize is the cache cap in bytes (15 GiB on the default 20 GiB volume).

Step 6: Point Bazel at the cache

On each workstation and CI runner, copy the server's certificate from the instance and compare its fingerprint with BAZEL_REMOTE_CERT_SHA256 from Step 4:

# on your workstation
scp -i <your-key.pem> ubuntu@<public-ip>:/etc/nginx/tls/bazel-remote.crt ~/bazel-remote.crt
openssl x509 -noout -fingerprint -sha256 -in ~/bazel-remote.crt

Bazel sends the password as an Authorization header. Build the header value from the password in Step 4 and put the cache settings in a .bazelrc that only you can read (your home directory's ~/.bazelrc, or a CI secret file passed with --bazelrc), never in the repository's own .bazelrc. Use the instance's address your build machines reach: the public IP as here, the private IP, or the DNS name from Step 4 (all are in the certificate). Then build as usual. The first build runs the actions and uploads their results; after bazel clean (or on another machine) the same actions come from the cache:

# on your workstation
B64=$(printf 'bazel:%s' '<cache-password>' | base64 | tr -d '\n')
cat >> ~/.bazelrc <<EOF
build --remote_cache=grpcs://<public-ip>
build "--remote_header=Authorization=Basic ${B64}"
build --tls_certificate=${HOME}/bazel-remote.crt
EOF
chmod 600 ~/.bazelrc
bazel build //hello
bazel clean
bazel build //hello

Keep the quotes around the --remote_header line in the file: the header value contains a space. This is real output from a small genrule target, built against the instance:

INFO: Analyzed target //hello:hello (6 packages loaded, 8 targets configured).
INFO: Found 1 target...
Target //hello:hello up-to-date:
  bazel-bin/hello/hello.txt
INFO: Elapsed time: 11.910s, Critical Path: 0.16s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions

INFO: Starting clean (this may take a while). Use --async if the clean takes more than several minutes.

INFO: Analyzed target //hello:hello (6 packages loaded, 8 targets configured).
INFO: Found 1 target...
Target //hello:hello up-to-date:
  bazel-bin/hello/hello.txt
INFO: Elapsed time: 1.340s, Critical Path: 0.19s
INFO: 2 processes: 1 remote cache hit, 1 internal.
INFO: Build completed successfully, 2 total actions

1 remote cache hit is the cache at work. If the password is wrong, Bazel prints a Remote Cache: UNAUTHENTICATED warning and builds everything locally, so a typo never breaks a build, it only makes it slower.

The HTTP cache works too, with the user and password in the URL instead of a header: build --remote_cache=https://bazel:<cache-password>@<public-dns-name> with the same --tls_certificate line. Prefer the gRPC form: Bazel uses fewer connections with it, and the password is not part of a URL that tools may print.

Step 7: Check the security posture

Requests without the password, or with a wrong one, are refused; the health check stays open for load balancers; a request for a host name the instance does not serve is dropped without an answer, and nothing answers on port 80:

C=/etc/nginx/tls/bazel-remote.crt
curl -s --cacert $C -o /dev/null -w 'no login         -> HTTP %{http_code}\n' https://127.0.0.1/status
curl -s --cacert $C -K <(printf 'user = "bazel:not-the-password"\n') -o /dev/null -w 'wrong password   -> HTTP %{http_code}\n' https://127.0.0.1/status
curl -s --cacert $C -o /dev/null -w 'health check     -> HTTP %{http_code}\n' https://127.0.0.1/healthz
curl -s --cacert $C -H 'Host: evil.example' -o /dev/null -w 'unknown host     -> %{http_code}\n' https://127.0.0.1/healthz || true
curl -s -o /dev/null -w 'plain http :80   -> %{http_code}\n' --connect-timeout 3 http://127.0.0.1/ || true

no login         -> HTTP 401
wrong password   -> HTTP 401
health check     -> HTTP 200
unknown host     -> 000
plain http :80   -> 000

000 means no answer: nginx closes the connection for a host name it does not serve, and nothing listens on port 80.

The gRPC cache refuses a call without the password with the gRPC status UNAUTHENTICATED (16):

printf '\0\0\0\0\0' > /tmp/caps.req
curl -s --http2 --cacert /etc/nginx/tls/bazel-remote.crt -o /dev/null -D - -X POST -H 'content-type: application/grpc' -H 'te: trailers' \
  --data-binary @/tmp/caps.req https://127.0.0.1/build.bazel.remote.execution.v2.Capabilities/GetCapabilities | grep -i '^grpc-'
rm -f /tmp/caps.req

grpc-status: 16
grpc-message: no authentication metadata found

bazel-remote runs as the unprivileged bazel-remote user with Basic authentication on and its listeners on loopback, and bazel-remote, nginx and the first boot service cannot reach the instance metadata service:

ps -o user:14=,args= -C bazel-remote
systemctl show bazel-remote nginx bazel-remote-firstboot -p Id -p IPAddressDeny

bazel-remote   /opt/bazel-remote/bazel-remote --dir /var/lib/bazel-remote/cache --http_address 127.0.0.1:8080 --grpc_address 127.0.0.1:9092 --htpasswd_file /etc/bazel-remote/htpasswd
IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128
Id=bazel-remote.service

IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128
Id=nginx.service

IPAddressDeny=168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128
Id=bazel-remote-firstboot.service

bazel-remote logs at start that authentication is required for every request and that the metrics endpoint is off:

sudo journalctl -u bazel-remote -b --no-pager -o cat | grep -E 'built with|Authentication|Access mode|Endpoint metrics|Starting (HTTP|gRPC)' | sed -E 's/^[0-9/]+ [0-9:]+ //' | tail -6

bazel-remote built with go1.27.2 from git commit 985cc081b1d5299e9d651ab01a605362f6bb170c v2.6.3.
Authentication: basic
Access mode: authentication required
Endpoint metrics: disabled
Starting gRPC server on address 127.0.0.1:9092
Starting HTTP server on address 127.0.0.1:8080

bazel-remote keeps only a bcrypt hash of the password; the plain text is in the root only credentials file, and the TLS key is root only. The certificate was generated on this instance and names its own addresses:

sudo sed -E 's/^(bazel:\$2y\$10\$).*/\1<bcrypt-hash>/' /etc/bazel-remote/htpasswd
sudo stat -c '%a %U:%G %n' /etc/bazel-remote/htpasswd /root/bazel-remote-info.txt /etc/nginx/tls/bazel-remote.key
openssl x509 -noout -subject -ext subjectAltName -in /etc/nginx/tls/bazel-remote.crt

bazel:$2y$10$<bcrypt-hash>
640 root:bazel-remote /etc/bazel-remote/htpasswd
600 root:root /root/bazel-remote-info.txt
600 root:root /etc/nginx/tls/bazel-remote.key
subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = cloudimg
X509v3 Subject Alternative Name: 
    DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost

Step 8: Passwords, users, your own DNS name

bazel-remote-set-password manages the login file. Run it with no arguments to generate a new random password for bazel; it is written to /root/bazel-remote-info.txt. To choose the password yourself, hand it over on standard input (printf is a shell builtin, so the password never appears in the process list; start the line with a space so most shells keep it out of your history). Give each team or CI system its own user with --user <name>, and remove one with --delete <name>. Every user can read and write the whole cache. Passwords are 12 to 128 printable characters without spaces or :. bazel-remote reads the login file at start, so the helper restarts it:

sudo bazel-remote-set-password
 printf '%s\n' '<paste-a-password-for-ci>' | sudo bazel-remote-set-password --user ci --stdin
sudo bazel-remote-set-password --delete ci

Point a DNS record at the instance, then set the name the cache is served as. This mints a new self signed certificate for the name (and the instance's IP addresses), updates nginx and the URLs in the credentials file, and keeps the name across reboots. Give your build machines the new certificate afterwards:

sudo bazel-remote-set-hostname <your-domain>

To use a certificate from your own certificate authority instead, copy it over the generated pair and reload nginx (running bazel-remote-set-hostname again replaces it with a self signed one). Build machines that trust your authority no longer need --tls_certificate:

sudo install -m 0644 <your-domain>.crt /etc/nginx/tls/bazel-remote.crt
sudo install -m 0600 <your-domain>.key /etc/nginx/tls/bazel-remote.key
sudo nginx -t && sudo systemctl reload nginx

Step 9: The cache volume and its size

The cache lives on the separate EBS volume, owned by the bazel-remote service account and mounted without device files, set-uid programs or execution. The size cap, in GiB, is in /etc/bazel-remote/bazel-remote.env:

findmnt -no SOURCE,FSTYPE,OPTIONS /var/lib/bazel-remote
df -h --output=size,used,avail,target /var/lib/bazel-remote
sudo stat -c '%a %U:%G %n' /var/lib/bazel-remote
sudo grep -v '^#' /etc/bazel-remote/bazel-remote.env

/dev/nvme1n1 ext4   rw,nosuid,nodev,noexec,relatime
 Size  Used Avail Mounted on
  20G  3.2M   19G /var/lib/bazel-remote
750 bazel-remote:bazel-remote /var/lib/bazel-remote
BAZEL_REMOTE_MAX_SIZE=15

To grow the cache later, modify the volume's size in the EC2 console, then grow the filesystem, raise the cap to about 80% of the new size (80 for a 100 GiB volume) and restart bazel-remote:

sudo resize2fs "$(findmnt -no SOURCE /var/lib/bazel-remote)"
sudo sed -i -E 's/^BAZEL_REMOTE_MAX_SIZE=.*/BAZEL_REMOTE_MAX_SIZE=80/' /etc/bazel-remote/bazel-remote.env
sudo systemctl restart bazel-remote

A build cache can always be rebuilt, so it needs no backup; if it is ever corrupted, stop bazel-remote, empty /var/lib/bazel-remote/cache and start it again. Other documented BAZEL_REMOTE_* settings can be added to the same file (for example BAZEL_REMOTE_MAX_SIZE_HARD_LIMIT); settings that would turn off authentication, move a listener or enable profiling or the metrics endpoint are refused, and bazel-remote does not start with them.

Step 10: Server components

cat /opt/bazel-remote/VERSION
sha256sum /opt/bazel-remote/bazel-remote
nginx -v 2>&1

bazel-remote 2.6.3 (official release binary, Apache-2.0, sha256 65427624c288e29b17c7b0e189cf1d03ae1d8745354d9effc432f75c4361159a)
65427624c288e29b17c7b0e189cf1d03ae1d8745354d9effc432f75c4361159a  /opt/bazel-remote/bazel-remote
nginx version: nginx/1.24.0 (Ubuntu)

Logs: sudo journalctl -u bazel-remote for bazel-remote, /var/log/nginx/access.log for requests (with the real client address).

To move to a newer bazel-remote release, download the linux-amd64 binary from the bazel-remote releases page, check it against the SHA-256 the release page publishes, replace /opt/bazel-remote/bazel-remote and run sudo systemctl restart bazel-remote. The cache in /var/lib/bazel-remote is kept.

Step 11: Security recommendations

• Allow port 22 in the security group from your own networks only, and port 443 only from your developers' and CI runners' networks.
• Give each team or CI system its own user (Step 8) and rotate a password if it may have been shared: anyone with a password can write to the cache, and builds trust what the cache returns.
• Keep the cache settings in a private .bazelrc or a CI secret, never in the repository.
• Use your own DNS name and certificate (Step 8).
• Keep the operating system patched; unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.

Support and licensing

bazel-remote is open source under the Apache License 2.0; there is no per user or per CPU licence fee. The licence is in /usr/share/doc/bazel-remote/. Bazel is a trademark of Google LLC. cloudimg is not affiliated with or endorsed by the bazel-remote project or Google. The AMI price covers cloudimg's hardening, first boot setup and support.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk
• Support: 24/7, with a 24 hour response SLA