bazel-remote Build Cache on AWS User Guide

Overview

This image runs bazel-remote, a fast, self-hosted remote build cache that speaks the HTTP and gRPC Remote Cache (REAPI) protocols used by Bazel, Buck and Goma. When your build and CI machines share a remote cache, they download previously compiled artifacts and action results instead of rebuilding them, so local and CI builds get dramatically faster. bazel-remote is a single static Go binary; this image installs it, hardens it and wires it up as a systemd service, so a production ready cache is answering within minutes of launch.

This is a headless image. There is no web interface; you operate it over SSH and point your build tools at it. The cache serves the HTTP Remote Cache on TCP 8080 and the gRPC Remote Cache on TCP 9092.

Secure by default. bazel-remote normally starts unauthenticated on every interface, which is unacceptable for a network cache. This image does the opposite: HTTP Basic Auth is enforced on both the HTTP and the gRPC interface, and there is no baked or shared credential. On the first boot of your instance a one-shot service generates a unique username and a long random password for that specific instance and writes them to a root-owned file, so every request is authenticated from the first cache hit. A fail secure bootstrap gate ensures the cache cannot start at all until that per-instance credential exists, so the image can never come up as an open cache. On first boot the appliance also refreshes itself to the newest bazel-remote release.

The cache store (Content Addressable Storage blobs and Action Cache entries) lives on a dedicated EBS data volume mounted at /var/lib/bazel-remote, so it sits on its own independently resizable and snapshottable disk, separate from the operating system disk, and its size cap can be raised without resizing root.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 (SSH) from your management network and inbound ports 8080 and 9092 from your own CI and build hosts
• Bazel (or Buck, or Goma) on the build machines you want to share the cache

Step 1: Launch the Instance from the AWS Marketplace

1. Open the product page in the AWS Marketplace and choose Continue to Subscribe, then Continue to Configuration.
2. Select the software version and the Region you want to launch in, then choose Continue to Launch.
3. Under Choose Action select Launch through EC2, review the instance type (the listing recommends m5.large), your key pair, and a security group that allows inbound TCP 22, 8080 and 9092 from your own hosts, then launch.

Step 2: Launch the Instance from the AWS CLI

Replace the AMI ID, key name, subnet and security group with your own values.

aws ec2 run-instances \
  --image-id ami-xxxxxxxxxxxxxxxxx \
  --instance-type m5.large \
  --key-name your-key \
  --security-group-ids sg-xxxxxxxx \
  --subnet-id subnet-xxxxxxxx \
  --associate-public-ip-address \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=bazel-remote-cache}]'

Your security group should allow inbound TCP 22 from your management network and inbound TCP 8080 and 9092 only from the CI and build hosts that will use the cache.

Step 3: Connect to Your Instance

SSH in as the default login user for your operating system variant:

ssh -i your-key.pem ubuntu@<public-ip>

Step 4: Confirm the Cache Is Running

The cache starts automatically after the first-boot service has minted this instance's credential and written the bootstrap marker. Confirm the service is active and that it is listening on TCP 8080 (HTTP) and 9092 (gRPC):

sudo systemctl is-active bazel-remote
sudo ss -ltn | grep -E ':8080|:9092'
grep -m1 -i 'remote build cache' <<< "$(/usr/local/bin/bazel-remote --help 2>&1)"

Step 5: Retrieve This Instance's Cache Credentials

On first boot a unique username and a long random password were generated for this instance and written to a root-owned file. Read them with:

sudo cat /root/bazel-remote-info.txt

The file records BAZEL_REMOTE_USER, BAZEL_REMOTE_PASSWORD, a ready-to-use BAZEL_REMOTE_URL (HTTP) and BAZEL_REMOTE_GRPC_URL (gRPC), all unique to your instance. Keep this credential safe: every request to the cache, over HTTP or gRPC, must present it.

Step 6: Point Bazel at the Cache

Add the cache to your project's .bazelrc using the per-instance username and password and your instance's public IP. Bazel accepts either the HTTP or the gRPC endpoint:

# HTTP Remote Cache
build --remote_cache=http://your-user:your-password@<public-ip>:8080

# or gRPC Remote Cache
build --remote_cache=grpc://your-user:your-password@<public-ip>:9092

You can verify the cache end to end on the instance itself. This reads the per-instance credential and performs an authenticated content addressed PUT then GET round-trip against the local HTTP endpoint, confirming the store lives on the dedicated data volume:

U=$(sudo grep '^BAZEL_REMOTE_USER=' /root/bazel-remote-info.txt | cut -d= -f2-)
P=$(sudo grep '^BAZEL_REMOTE_PASSWORD=' /root/bazel-remote-info.txt | cut -d= -f2-)
BLOB=$(head -c 512 /dev/urandom | base64 -w0)
H=$(printf '%s' "$BLOB" | sha256sum | awk '{print $1}')
printf '%s' "$BLOB" | curl -s -o /dev/null -w 'PUT -> HTTP %{http_code}\n' -u "$U:$P" -X PUT --data-binary @- "http://127.0.0.1:8080/cas/$H"
GOT=$(curl -s -u "$U:$P" "http://127.0.0.1:8080/cas/$H")
[ "$GOT" = "$BLOB" ] && echo "GET -> 200 OK, retrieved bytes match stored bytes (cache hit)"
df -h /var/lib/bazel-remote | tail -1

Step 7: Authentication Is Enforced

The cache has no anonymous access. An unauthenticated request to either the status endpoint or a CAS object is rejected with HTTP 401, and the same Basic Auth is enforced on the gRPC interface:

curl -s -o /dev/null -w 'unauthenticated /status -> HTTP %{http_code}\n' http://127.0.0.1:8080/status
curl -s -o /dev/null -w 'unauthenticated /cas    -> HTTP %{http_code}\n' http://127.0.0.1:8080/cas/deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef
sudo ss -Htln 'sport = :9092' && echo "gRPC listener live on :9092"

Step 8: Size the Cache and Its Data Volume

The cache store lives on a dedicated EBS data volume at /var/lib/bazel-remote/cache. The maximum on-disk cache size is capped by BAZEL_REMOTE_MAX_SIZE_GIB (default 10 GiB) in the environment file. bazel-remote evicts least-recently-used entries when the cap is reached. To grow the cache, raise the cap and restart the service:

sudo sed -i 's/^BAZEL_REMOTE_MAX_SIZE_GIB=.*/BAZEL_REMOTE_MAX_SIZE_GIB=40/' /etc/bazel-remote/bazel-remote.env
sudo systemctl restart bazel-remote

If you need more than the data volume's capacity, resize the EBS volume in the AWS console and grow the filesystem (sudo resize2fs /dev/nvme1n1), then raise the cap to match.

Step 9: Production Notes

• Restrict access. Scope the instance security group so only your own CI and build hosts can reach TCP 8080 and 9092.
• TLS. For encrypted transport, terminate TLS in front of the cache (for example with a reverse proxy such as nginx or Caddy) or run bazel-remote with its own --tls_cert_file and --tls_key_file. cloudimg support can help you wire this up.
• Metrics. bazel-remote exposes Prometheus metrics and a status endpoint (authenticated) for monitoring cache hit rates and size.
• Updates. Each instance refreshes itself to the newest bazel-remote release on first boot; for a running instance, replace /usr/local/bin/bazel-remote with a newer release binary and restart the service.

Support

cloudimg provides 24/7 technical support for this product by email (support@cloudimg.co.uk) and live chat. We help with .bazelrc integration over HTTP or gRPC, sizing the cache and its dedicated data volume, restricting access, adding TLS termination, monitoring, and bazel-remote version upgrades. Critical issues receive a one hour average response time.

bazel-remote is distributed under the Apache License 2.0. Bazel is a trademark of its respective owner. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.