GPT-Load on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and configuration of GPT-Load on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. GPT-Load is a high performance LLM API key pool manager. Its unit of work is a group: one upstream provider plus a pool of many API keys for that provider.

That is the problem it solves. If you hold twenty Gemini keys, or a handful of OpenAI keys across several accounts, an application can only ever use one of them at a time and has no idea which ones are still alive. GPT-Load puts the whole pool behind one endpoint, rotates through it, checks every key in the background, blacklists keys that start failing and restores them when they recover.

GPT-Load is a transparent relay, not a translation layer. It forwards your request to the provider's own native API and streams the native response straight back, so nothing about the request or response shape changes — you point your existing SDK at a different base URL and everything else stays the same.

GPT-Load hosts no model. It relays to providers using your API keys, so it needs no GPU and runs comfortably on a small CPU only VM. It is a single Go executable with the web dashboard compiled into it, which is why this appliance needs no container runtime, no language runtime, no database server and no cache server.

The cloudimg image ships the free and open source, MIT licensed GPT-Load as the official upstream release binary, pinned by content hash and tied to an exact upstream commit. No provider key is ever baked into the image — you add your own, and they are encrypted at rest. A unique auth key, encryption key and gateway password are generated for each VM on first boot, and the gateway in front of GPT-Load will not start until first boot has proved that an unauthenticated request is answered with 401. Backed by 24/7 cloudimg support.

GPT-Load is an independent open source project. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by the GPT-Load project or its contributors. It ships the free and open source MIT licensed software, unmodified.

What is included:

• GPT-Load v1.4.9 — the official upstream linux/amd64 release binary, pinned by sha256 content hash, not by a moving tag
• Verified provenance: Go stamps the source commit into the binary, and this appliance asserts it is 3028b1ff9d455e83340b1b8b2aa7123d5cb6a8ac (upstream tag v1.4.9) built from an unmodified tree
• The GPT-Load web dashboard, compiled into the same binary
• An nginx access gateway on port 80, installed as a host package so unattended security upgrades keep the internet facing component patched
• An embedded SQLite store and an in process cache — no database server and no cache server of any kind on the instance
• A unique sk-cloudimg-… auth key, a unique encryption key and a unique gateway password generated per VM on first boot, never baked into the image
• Your pooled provider keys encrypted at rest with this VM's own encryption key
• A shipped dependency licence inventory at /usr/share/gpt-load/dependency-licences.txt covering all 352 dependencies compiled into the binary

Prerequisites

• An Azure subscription with permission to create VMs
• At least one API key for a provider you want to pool (OpenAI, Google Gemini, Anthropic, or any OpenAI compatible endpoint) — you add these yourself after launch
• A Standard_B2s VM or larger. GPT-Load is a single Go executable whose work per request is a key selection, a header rewrite and a streaming copy, so 2 vCPU and 4 GiB is genuinely sufficient for most deployments
• Inbound TCP 80 open to the networks your applications and administrators connect from
• An SSH key pair for administrative access

Step 1: Deploy from the Azure Portal

1. In the Azure Portal, search the Marketplace for GPT-Load on Ubuntu 24.04 LTS by cloudimg and select Create.
2. Choose your subscription, resource group and region.
3. Set the VM size to Standard_B2s or larger.
4. Set the authentication type to SSH public key and supply your key.
5. Under Inbound port rules, allow SSH (22) and HTTP (80).
6. On the Disks tab, StandardSSD_LRS is a good balance of cost and performance.
7. Select Review + create, then Create.

First boot takes about a minute. During that minute the appliance generates this VM's credentials and proves its gateway rejects unauthenticated requests before opening port 80.

Step 2: Deploy from the Azure CLI

az vm create \
  --resource-group my-resource-group \
  --name gpt-load-relay \
  --image cloudimg:gpt-load-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --storage-sku StandardSSD_LRS \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

# then open the ports your applications need
az vm open-port \
  --resource-group my-resource-group \
  --name gpt-load-relay \
  --port 80 --priority 1001

Restrict that rule to your own address ranges in production. GPT-Load spends money on your behalf, so the smaller the set of networks that can reach port 80, the better.

Step 3: Connect to your VM

ssh azureuser@<vm-ip>

Step 4: Confirm the services are running

sudo systemctl is-active gpt-load.service nginx.service

Expected output:

active
active

The relay itself binds loopback only. nginx on port 80 is the sole ingress and the security gate:

sudo ss -lntp | grep -E ':(80|3001)\b'

Expected output:

LISTEN 0  511      0.0.0.0:80    0.0.0.0:*  users:(("nginx",pid=13533,fd=5),("nginx",pid=13532,fd=5))
LISTEN 0  4096   127.0.0.1:3001  0.0.0.0:*  users:(("gpt-load",pid=13130,fd=6))
LISTEN 0  511         [::]:80       [::]:*  users:(("nginx",pid=13533,fd=6),("nginx",pid=13532,fd=6))

127.0.0.1:3001 is the important line. GPT-Load is not reachable from the network at all — only through the gate.

nginx answers its own liveness literal without a credential, and it never touches the relay:

curl -s http://127.0.0.1/nginx-health

Expected output:

OK

Step 5: Read the per instance credentials

Every credential on this appliance was generated for this VM on its first boot. Nothing is baked into the image.

sudo cat /root/gpt-load-credentials.txt

You get four values:

Value	What it is for
GPTLOAD_GATE_USERNAME	The cloudimg gateway's HTTP Basic username, for a browser
GPTLOAD_GATE_PASSWORD	The cloudimg gateway's HTTP Basic password, for a browser
GPTLOAD_AUTH_KEY	The one credential your applications use, and the dashboard's Auth Key
GPTLOAD_ENCRYPTION_KEY	Encrypts your pooled provider keys at rest. Keep it

Keep GPTLOAD_ENCRYPTION_KEY. Your pooled provider keys are encrypted with it inside the SQLite database. Changing it without running upstream's key migration makes the stored pool unreadable.

Step 6: Open the dashboard

Browse to http://<vm-ip>/.

Your browser is first challenged by the cloudimg gateway. Sign in with GPTLOAD_GATE_USERNAME and GPTLOAD_GATE_PASSWORD. The gateway then issues an HttpOnly, SameSite=Lax cookie that the dashboard's own requests ride on.

The GPT-Load dashboard then asks for its Auth Key. Paste GPTLOAD_AUTH_KEY.

The dashboard opens with nothing configured — no groups, no keys, no providers. That is deliberate and it is exactly the state you receive: this image never ships a provider key.

Step 7: Understand the access model

This appliance is an API key pool relay. That makes an unauthenticated instance materially worse than an unauthenticated web app: it is a free proxy spending your provider credit, and it is machine callable by design.

Measured against the stock upstream build during this image's own build, these routes answer a completely anonymous request. The evidence ships on the instance:

sudo cat /usr/share/gpt-load/stock-exposure.txt

So the cloudimg image puts nginx in front, default deny at server scope. Every route inherits the gate — the dashboard, the admin API, /health and every /proxy/* relay path. Exactly one location is unauthenticated, and it is nginx's own static /nginx-health literal, which never touches the relay.

Prove it. Every one of these returns 401 with no credential:

for p in / /health /api/settings /api/groups /proxy/default/v1/models; do
  printf '%s -> %s\n' "$p" "$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1$p")"
done

Expected output:

/ -> 401
/health -> 401
/api/settings -> 401
/api/groups -> 401
/proxy/default/v1/models -> 401

One credential, five ways to send it

A plain password prompt would be wrong for this product twice over. A browser's stored HTTP Basic credential travels in the same Authorization header the dashboard's own JavaScript overwrites, and an OpenAI, Anthropic or Google SDK needs that header for its API key.

So the gate accepts your one GPTLOAD_AUTH_KEY in every idiom GPT-Load itself accepts, plus the browser cookie and HTTP Basic:

KEY=$(sudo grep -m1 'GPTLOAD_AUTH_KEY=' /root/gpt-load-credentials.txt | cut -d= -f2-)
for h in "Authorization: Bearer $KEY" "X-Api-Key: $KEY" "X-Goog-Api-Key: $KEY"; do
  printf '%-34s -> %s\n' "${h%%:*}" "$(curl -s -o /dev/null -w '%{http_code}' -H "$h" http://127.0.0.1/api/settings)"
done
printf '%-34s -> %s\n' "?key=" "$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1/api/settings?key=$KEY")"

Expected output:

Authorization                      -> 200
X-Api-Key                          -> 200
X-Goog-Api-Key                     -> 200
?key=                              -> 200

Because those are the same idioms GPT-Load's own authentication reads, your application sends one credential that satisfies both the cloudimg gate and GPT-Load's own check. There is no double authentication and no SDK change.

Every comparison is an exact string match against a value generated for this VM. A one character mutation is refused:

curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer ${KEY}x" http://127.0.0.1/api/settings

($KEY is still set from the previous block.)

Expected output:

401

The gate cannot be started unconfigured

nginx.service carries three ConditionPathExists guards and an ExecStartPre guard. The first marker is created by first boot only after it has proved, against a throwaway loopback only listener running this VM's real gate map, that an unauthenticated request is answered with exactly 401 and that a one character mutation of every credential is refused.

sudo systemctl cat nginx.service | grep -E 'ConditionPathExists|ExecStartPre'

Expected output:

ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ConditionPathExists=/var/lib/cloudimg/gpt-load-gate.ready
ConditionPathExists=/etc/nginx/gpt-load.htpasswd
ConditionPathExists=/etc/nginx/conf.d/gpt-load-gate-map.conf
ExecStartPre=/usr/local/sbin/gpt-load-preflight.sh --nginx

Take the credential away and nginx refuses to start rather than serving an open relay. gpt-load.service is guarded the same way, and additionally refuses to start if AUTH_KEY is empty, looks like a default, or if the relay is configured to bind anything other than loopback.

Step 8: Build your first key pool

A group is one provider plus a pool of keys for it.

1. In the dashboard, open Keys and select Create Group.
2. Give it a name — this becomes part of your endpoint URL, so keep it short and URL safe, for example gemini.
3. Choose the Channel Type that matches the provider: openai, gemini or anthropic.
4. Set the Upstream Address to the provider's own API base, for example https://generativelanguage.googleapis.com for Gemini.
5. Save the group, then paste your API keys into it — one per line, or comma separated. This is the pool.

Your keys are written to the SQLite store encrypted with this VM's GPTLOAD_ENCRYPTION_KEY. Confirm the running relay agrees its key matches its own store:

KEY=$(sudo grep -m1 'GPTLOAD_AUTH_KEY=' /root/gpt-load-credentials.txt | cut -d= -f2-)
curl -s -H "Authorization: Bearer $KEY" -H 'Accept-Language: en' \
  http://127.0.0.1/api/dashboard/encryption-status | python3 -m json.tool

Expected output — has_mismatch: false means the key the relay is running with is the key its stored data was encrypted with:

{
    "code": 0,
    "message": "Success",
    "data": {
        "has_mismatch": false,
        "message": "",
        "scenario_type": "",
        "suggestion": ""
    }
}

The Accept-Language: en header is worth keeping on any curl against this API: without it the relay answers in its default language.

Step 9: Point your application at the relay

Your endpoint is http://<vm-ip>/proxy/<your-group-name> and your API key is GPTLOAD_AUTH_KEY.

OpenAI Python SDK:

from openai import OpenAI

client = OpenAI(
    base_url="http://<vm-ip>/proxy/<your-group-name>/v1",
    api_key="<GPTLOAD_AUTH_KEY>",
)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Anthropic Python SDK:

from anthropic import Anthropic

client = Anthropic(
    base_url="http://<vm-ip>/proxy/<your-group-name>",
    api_key="<GPTLOAD_AUTH_KEY>",
)

curl, against a Gemini group:

curl -s "http://<vm-ip>/proxy/<your-group-name>/v1beta/models/gemini-2.0-flash:generateContent" \
  -H "X-Goog-Api-Key: <GPTLOAD_AUTH_KEY>" \
  -H 'Content-Type: application/json' \
  -d '{"contents":[{"parts":[{"text":"Hello"}]}]}'

The OpenAI SDK sends Authorization: Bearer, the Anthropic SDK sends x-api-key and the Google SDK sends x-goog-api-key. All three are accepted by the cloudimg gate and are GPT-Load's native idiom, so no SDK needs modifying.

Note the path shape: everything after the group name is passed through to the provider unchanged. /proxy/gemini/v1beta/models/... becomes https://generativelanguage.googleapis.com/v1beta/models/... with one of your pooled keys attached.

Step 10: Key health, rotation and blacklisting

This is what you are actually paying GPT-Load to do.

• Rotation. Each request takes the next key from the pool, so load is spread across every key you own rather than hammering one.
• Validation. Select Validate on a group and GPT-Load tests every key in the pool against the upstream and reports which are alive.
• Blacklisting. A key that fails repeatedly is moved out of rotation automatically, so a dead or rate limited key stops poisoning your traffic.
• Restore. Restore all invalid puts blacklisted keys back into rotation — useful after a quota window resets.
• Aggregate groups. A group can contain sub groups with weights, so you can blend several providers behind one endpoint.

Every request is logged with its group, the key used, the status and the latency. Logs and Dashboard show the request volume, success rate and per key behaviour over time.

Step 11: Where your data lives

Everything GPT-Load stores is in one directory, on the OS disk, readable only by root:

sudo ls -la /var/lib/gpt-load/data/
sudo stat -c '%a %U:%G %n' /var/lib/gpt-load/data

Expected output:

700 root:root /var/lib/gpt-load/data

gpt-load.db is the whole application state: your groups, your encrypted key pool, the settings table and the request log. There is no database server and no cache server on this instance, so there is nothing else listening and nothing else to patch.

Back it up by stopping the relay and copying the file — and back up /root/gpt-load-credentials.txt with it, because the database is useless without the encryption key:

sudo systemctl stop gpt-load.service
sudo tar czf /root/gpt-load-backup.tar.gz /var/lib/gpt-load/data /root/gpt-load-credentials.txt
sudo systemctl start gpt-load.service

Step 12: What is in the image, and under what licence

GPT-Load is MIT licensed. The upstream licence ships on the instance:

head -3 /usr/share/gpt-load/LICENSE.upstream

Expected output:

MIT License

Copyright (c) 2024 GPT-Load Contributors

The binary proves its own source tree. Go stamps the source commit into the executable, and this appliance ships the exact commit the upstream v1.4.9 tag points at, built from an unmodified working tree:

LC_ALL=C grep -a -o 'vcs\.revision=[0-9a-f]\{40\}' /usr/local/bin/gpt-load | head -1
LC_ALL=C grep -a -o 'vcs\.modified=[a-z]*' /usr/local/bin/gpt-load | head -1

Expected output:

vcs.revision=3028b1ff9d455e83340b1b8b2aa7123d5cb6a8ac
vcs.modified=false

Every dependency compiled into that binary was licence checked at build time and the inventory ships with the image:

awk -F'\t' 'NR>2 {print $4}' /usr/share/gpt-load/dependency-licences.txt | sort | uniq -c | sort -rn

Expected output:

    288 MIT
     20 Apache-2.0
     15 BSD
     13 ISC
     10 BSD-2-Clause
      3 BSD-3-Clause
      1 Python-2.0
      1 MPL-2.0
      1 (MIT OR CC0-1.0)

The build fails if any dependency declares AGPL, Affero, SSPL, BUSL, Business Source, Elastic (ELv2), Commons Clause, non commercial, PolyForm, Prosperity or RSAL terms — and it fails closed on any dependency whose licence cannot be resolved. Everything in the image is MIT, Apache-2.0, BSD, ISC, MPL-2.0 or Python-2.0.

Note that the binary links database client drivers (SQLite, MySQL, PostgreSQL, Redis) but this appliance installs and runs no database or cache server. That is what keeps the image clear of MongoDB's SSPL, Elasticsearch's ELv2 and the RSALv2 relicensing of Redis 7.4 and later:

sudo ss -lntH | awk '{print $4}' | sort -u

Expected output — the gate on 80, sshd on 22, GPT-Load on loopback 3001, and the platform's own DNS stub on loopback. Nothing else is listening at all:

0.0.0.0:22
0.0.0.0:80
127.0.0.1:3001
127.0.0.53%lo:53
127.0.0.54:53
[::]:22
[::]:80

Day to day operation

# service state
sudo systemctl status gpt-load.service --no-pager
sudo systemctl restart gpt-load.service

# logs
sudo journalctl -u gpt-load.service -f
sudo journalctl -u gpt-load-firstboot.service --no-pager

# the gateway
sudo systemctl restart nginx.service
sudo nginx -t

To rotate the gateway password, run this on the VM and enter the new password when prompted (it is asked for twice, and never echoed):

sudo htpasswd -B /etc/nginx/gpt-load.htpasswd gptload
sudo systemctl reload nginx

Record the new password alongside /root/gpt-load-credentials.txt, which still shows the original one — htpasswd stores only a bcrypt hash and cannot be read back.

Security hardening

• Restrict port 80 to your own networks. This relay spends your provider credit; the network ACL is your first control, not your last.
• Put TLS in front. Terminate HTTPS on an Azure Application Gateway, or install a certificate on this nginx with certbot for a real hostname. The gate cookie is deliberately not marked Secure so it works over plain http on a bare VM IP; once you are on HTTPS, add Secure to the add_header Set-Cookie line in /etc/nginx/sites-available/gpt-load.conf.
• Treat GPTLOAD_AUTH_KEY as a production secret. It is the only credential between the internet and your entire key pool.
• Set per group rate limits in the dashboard, so a compromised client cannot drain a quota in minutes.
• Keep unattended upgrades on. They are enabled in this image and they patch nginx, which is the only internet facing component.

Troubleshooting

Port 80 answers nothing at all. That is the appliance being closed rather than broken. nginx will not start until first boot has proved the gate. Check:

sudo systemctl status gpt-load-firstboot.service --no-pager
sudo journalctl -u gpt-load-firstboot.service --no-pager | tail -30

Everything returns 401, including with the right key. Confirm you are sending the key in a recognised idiom (Authorization: Bearer, X-Api-Key, X-Goog-Api-Key or ?key=) and that it matches /root/gpt-load-credentials.txt exactly. The comparison is an exact string match, so a trailing newline from a copy and paste will fail.

The dashboard loads but every panel is empty. The dashboard needs its own Auth Key in addition to the gateway sign in. Paste GPTLOAD_AUTH_KEY when it asks.

A group returns errors from the provider. Use Validate on the group. GPT-Load will test each key and mark the dead ones; the pool then routes around them.

The relay will not start. The preflight guard prints exactly why:

sudo /usr/local/sbin/gpt-load-preflight.sh --app; echo "exit=$?"

Support

This image is published and supported by cloudimg. For assistance with the image, deployment or configuration, contact cloudimg support 24/7 through the Azure Marketplace listing.