llama.cpp Server b9937 on AWS User Guide

Overview

This guide covers launching and verifying llama.cpp Server b9937 on Ubuntu 24.04 from cloudimg on Amazon EC2. It is one version of the cloudimg Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace; you pick it as the software version when you subscribe and launch. The other LLM servers, gateways and chat interfaces in the listing are described on the listing's guide hub.

llama.cpp is the reference engine for running GGUF-quantized large language models on ordinary CPUs. Its llama-server binary serves two things from one process: a browser chat UI and an OpenAI-compatible REST API (/v1/chat/completions, /v1/completions, /v1/models), so existing OpenAI SDK code usually needs only a new base URL. No GPU is needed: the model runs in the instance's memory on its vCPUs.

What is on the image:

• llama.cpp build b9937 (upstream commit 2021515a1, ggml-org/llama.cpp, MIT licence), the official prebuilt CPU-only Linux release pinned by sha256, run by llama-cpp.service as the unprivileged llama-cpp user
• llama-server bound to loopback only (127.0.0.1:8080), with nginx as the only network-facing service
• HTTPS on port 443 with a self-signed certificate created on your instance's first boot; port 80 redirects to HTTPS
• A per-instance password for the chat UI and the API (user admin) and a per-instance llama-server API key, both generated on first boot into a root-only file
• A demo model, Qwen2.5-1.5B-Instruct (Q4_K_M GGUF, about 1.1 GB, Apache License 2.0), on its own 20 GiB gp3 volume at /var/lib/llama-cpp, so the chat UI and API work offline a few minutes after launch, with no download
• llama-cpp-set-hostname, a helper that serves llama.cpp Server under your own DNS name and re-issues the certificate for it
• The llama.cpp MIT licence, the model's Apache-2.0 licence and a notice in /usr/share/doc/cloudimg/llama-cpp/

Default user: ubuntu · Architecture: x86_64 · Root volume: 8 GiB gp3 · Model volume: 20 GiB gp3

Security posture

• No default credential. The chat UI and API password, the llama-server API key and the TLS key are generated on each instance's first boot. None of them exists in the image, so no two instances share one.
• Two gates, both per instance. nginx asks for the password (HTTP Basic, bcrypt-hashed) on the chat UI and on every API path. Behind it, llama-server itself requires its own API key, which nginx adds upstream after checking your password. A local user on the instance who calls 127.0.0.1:8080 directly is refused by llama-server with 401.
• Fails closed. nginx will not start until first boot has created the certificate and password, and first boot stops nginx again unless it proves that an anonymous request and a wrong password get 401 and the right password completes a chat request. llama-cpp.service refuses to start with an empty key file.
• Only nginx faces the network. It answers only for the configured hostname (any other Host header gets its connection closed), uses TLS 1.2 or 1.3, refuses cross-origin POST requests and WebSocket upgrades (403), strips llama-server's CORS headers, and replaces any client-supplied X-Forwarded-For, X-Real-IP or Forwarded header with the real peer address.
• The hostname comes from configuration, never from a request. First boot reads it from EC2 instance metadata (IMDSv2) or from /etc/cloudimg/llama-cpp-hostname.
• A small server surface. The slot monitor (/slots, which shows other users' prompts), runtime changes to server properties (POST /props), slot save and restore, sleep mode, the experimental agent tools and MCP proxy, and downloading models by URL are all off. The image does not ship llama.cpp's RPC server.
• No instance metadata access for llama-server. llama-cpp.service is blocked from the instance metadata service (169.254.169.254 and fd00:ec2::254) by systemd IPAddressDeny, so nothing a request asks the server to do can read the instance's role credentials. Amazon Time Sync (169.254.169.123) is unaffected.
• IMDSv2 required. The AMI is registered with ImdsSupport v2.0, so instances launched from it refuse token-less (IMDSv1) metadata requests.
• Sandboxed service and per-instance identity. llama-cpp.service runs with NoNewPrivileges, ProtectSystem=full, ProtectHome, PrivateTmp and PrivateDevices. SSH host keys and the machine ID are generated fresh on every instance.

Prerequisites

1. An AWS account and permission to launch EC2 instances
2. An EC2 key pair in the Region you launch in

Recommended instance type: m5.large (2 vCPUs, 8 GiB) runs the bundled 1.5B demo model at roughly 8 to 10 tokens per second. Inference is CPU-bound and runs in memory, so choose m5.xlarge (4 vCPUs, 16 GiB) or larger for faster replies, several users at once, or models above about 3B parameters, and avoid burstable instance types for sustained use. First boot sets llama-server's thread count to the instance's vCPU count.

Connecting to your instance

Step 1: Launch the instance

Option A: AWS Marketplace console

1. Open the cloudimg Hardened Ollama, LiteLLM & Self Hosted LLM Server AMIs listing on AWS Marketplace and choose View purchase options, then Subscribe. One subscription covers every version in the listing.
2. Once the subscription is active, choose Launch your software.
3. Choose One-click launch from AWS Marketplace (or Launch from EC2 Console for the full launch wizard). In Version, choose llama.cpp Server b9937 on Ubuntu 24.04, then choose your Region.
4. Pick an instance type (m5.large or larger; the software charge is $0.04 per vCPU per hour) and your key pair.
5. Security group: the listing's default group opens SSH (22), HTTP (80) and HTTPS (443). Restrict SSH to your own IP address, and restrict 80 and 443 to the networks your users and applications use.
6. Choose your VPC and subnet, leave the metadata defaults (the image already requires IMDSv2), then choose Launch. The 20 GiB model volume is created and attached automatically.

Option B: AWS CLI

Run this on your own workstation. It creates a security group that allows SSH from your management address and HTTPS from your own network, then launches the instance. Replace the AMI ID with the one shown for your Region on the Marketplace launch page, and the VPC, subnet, key pair and CIDR ranges with your own:

SG=$(aws ec2 create-security-group --group-name llama-cpp-server \
  --description "llama.cpp Server" --vpc-id vpc-0123456789abcdef0 \
  --query GroupId --output text)
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 22 --cidr <your-mgmt-cidr>
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 443 --cidr 10.0.0.0/16
aws ec2 authorize-security-group-ingress --group-id "$SG" --protocol tcp --port 80 --cidr 10.0.0.0/16
aws ec2 run-instances \
  --image-id ami-0123456789abcdef0 \
  --instance-type m5.large \
  --key-name my-key-pair \
  --security-group-ids "$SG" \
  --subnet-id subnet-0123456789abcdef0 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=llama-cpp-server}]'

You do not need --metadata-options: the AMI already makes IMDSv2 required.

Step 2: Connect and find your credentials

Connect as ubuntu with the key pair you chose:

ssh -i ~/.ssh/my-key-pair.pem ubuntu@<public-ip>

First boot finishes a few minutes after launch: a new instance restores the model volume from its snapshot as it is first read, so first boot reads the model through once before llama-server finishes loading it. It then sets the hostname llama.cpp Server is served as (the instance's public DNS name when it has one), creates the TLS certificate for it, generates the password, proves the gate refuses anonymous and wrong credentials, starts nginx and writes /root/llama-cpp-credentials.txt (mode 0600, root). These are the entries in that file (the secret values are not shown here):

sudo grep -E '^LLAMA_CPP_(URL|USERNAME|API_URL)=' /root/llama-cpp-credentials.txt
sudo grep -oE '^[A-Z_]+=' /root/llama-cpp-credentials.txt
sudo stat -c '%a %U:%G' /root/llama-cpp-credentials.txt

LLAMA_CPP_URL=https://ec2-54-166-74-253.compute-1.amazonaws.com/
LLAMA_CPP_USERNAME=admin
LLAMA_CPP_API_URL=https://ec2-54-166-74-253.compute-1.amazonaws.com/v1/chat/completions
LLAMA_CPP_HOST=
LLAMA_CPP_URL=
LLAMA_CPP_USERNAME=
LLAMA_CPP_PASSWORD=
LLAMA_CPP_API_URL=
LLAMA_CPP_LOOPBACK_API_KEY=
600 root:root

To read the password, run sudo cat /root/llama-cpp-credentials.txt in your own session and store it in your secrets manager.

Step 3: Confirm the server is running

The key unit, llama-server, nginx and the first-boot unit are active, and only sshd and nginx listen beyond loopback. llama-server itself is on 127.0.0.1:8080, so nginx is the only way in from the network.

systemctl is-active llama-cpp-apikey.service llama-cpp.service nginx.service llama-cpp-firstboot.service
sudo ss -Htlnp | awk '{print $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u
LD_LIBRARY_PATH=/opt/llama-cpp/llama-b9937 /opt/llama-cpp/llama-b9937/llama-server --version 2>&1 | head -2

active
active
active
active
0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:8080 users:(("llama-server"))
127.0.0.53%lo:53 users:(("systemd-resolve"))
127.0.0.54:53 users:(("systemd-resolve"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))
version: 9937 (2021515a1)
built with GNU 11.4.0 for Linux x86_64

/healthz is answered by nginx itself without a password and reveals nothing about your configuration, so it is safe to use as a load balancer health check:

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt "https://$H/healthz"

ok

Step 4: Prove the gates

Without the password, the chat UI and every API path are refused, and so is a password that is wrong by one character. This instance's password is accepted:

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
for R in / /v1/models /v1/chat/completions /props; do
  printf '%s -> %s\n' "$R" "$(curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w '%{http_code}' "https://$H$R")"
done
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w 'wrong password -> %{http_code}\n' -K <(printf 'user = "admin:%sx"\n' "$P") "https://$H/v1/models"
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w 'this instance password -> %{http_code}\n' -K <(printf 'user = "admin:%s"\n' "$P") "https://$H/v1/models"

/ -> 401
/v1/models -> 401
/v1/chat/completions -> 401
/props -> 401
wrong password -> 401
this instance password -> 200

The second gate is llama-server's own API key. A call straight to the loopback port, without going through nginx, is refused by llama-server itself:

curl -s -w ' HTTP_%{http_code}\n' -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"hi"}]}' http://127.0.0.1:8080/v1/chat/completions

{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}} HTTP_401

The commands above pass the password to curl through a configuration read from a pipe (-K), so it never appears in the process list or your shell history.

Step 5: Chat in the browser

Browse to LLAMA_CPP_URL (https://<public-dns-name>/). The certificate is self-signed, so your browser warns once; accept it for this host (or install your own certificate, see Step 8). The browser then asks for a username and password: use admin and LLAMA_CPP_PASSWORD.

The chat UI opens with the bundled model selected:

Type a message and press Enter. The reply streams in token by token, with the prompt and generation speed shown under each message:

Conversations are stored in your browser, not on the server. Under Settings you can set a system message and the sampling parameters. Leave API Key empty: nginx already supplies llama-server's key once you have signed in.

The first time you send a message the UI may offer to add MCP servers. That feature runs in your browser and connects from there; the server-side MCP proxy is turned off on this image.

Step 6: Call the OpenAI-compatible API

The model is listed under the alias qwen2.5-1.5b-instruct:

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -K <(printf 'user = "admin:%s"\n' "$P") "https://$H/v1/models" | jq -r '.data[] | [.id, .owned_by] | @tsv'

qwen2.5-1.5b-instruct   llamacpp

A chat completion, generated on the instance's CPU:

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -K <(printf 'user = "admin:%s"\n' "$P") \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is the capital of France? Answer in one word."}],"max_tokens":16,"temperature":0}' \
  "https://$H/v1/chat/completions" | jq -r '.choices[0].message.content, .usage.total_tokens'

Paris
43

From your own workstation, copy the instance's certificate (/etc/ssl/llama-cpp/llama-cpp.crt) and trust it, or put a certificate your clients already trust in front (Step 8). With the OpenAI Python SDK, pass the password as HTTP Basic credentials through the HTTP client; the api_key value is not used by the gate:

import httpx
from openai import OpenAI

client = OpenAI(
    base_url="https://<LLAMA_CPP_HOST>/v1",
    api_key="unused",
    http_client=httpx.Client(auth=("admin", "<LLAMA_CPP_PASSWORD>"), verify="llama-cpp.crt"),
)
reply = client.chat.completions.create(
    model="qwen2.5-1.5b-instruct",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(reply.choices[0].message.content)

Add "stream": true to the request body (or stream=True in the SDK) to receive the reply token by token.

Step 7: Use a different model

The model lives on its own gp3 volume at /var/lib/llama-cpp, so you can grow it independently of the root volume (modify the volume in the EC2 console, then run sudo resize2fs on its device). The volume and the running service:

df -h /var/lib/llama-cpp | tail -1
sudo ls -l /var/lib/llama-cpp/*.gguf
systemctl show -p ExecStart --value llama-cpp.service | grep -oE -- '--(model|alias|ctx-size|threads) [^ ]+'

/dev/nvme1n1     20G  1.1G   18G   6% /var/lib/llama-cpp
-rw-r----- 1 llama-cpp llama-cpp 1117320736 Oct 20 01:47 /var/lib/llama-cpp/qwen2.5-1.5b-instruct-q4_k_m.gguf
--model /var/lib/llama-cpp/qwen2.5-1.5b-instruct-q4_k_m.gguf
--alias qwen2.5-1.5b-instruct
--ctx-size 4096
--threads 2

To serve another GGUF model, download it onto the volume from a source you trust (check its licence), point the unit at it and restart. For example:

sudo -u llama-cpp curl -fL -o /var/lib/llama-cpp/<model>.gguf https://huggingface.co/<org>/<repo>/resolve/main/<model>.gguf
sudo sed -i 's#--model [^ ]*#--model /var/lib/llama-cpp/<model>.gguf#; s#--alias [^ ]*#--alias <model-alias>#' /etc/systemd/system/llama-cpp.service
sudo systemctl daemon-reload && sudo systemctl restart llama-cpp

A model needs roughly its file size in memory plus room for the context, so size the instance to match. Keep the --api-key-file and --no-slots options in the unit.

Step 8: Serve llama.cpp Server under your own DNS name

Port 80 redirects to HTTPS on the configured hostname, and the certificate covers that name, the instance's public and private addresses, localhost and 127.0.0.1:

H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://$H/v1/models"
openssl x509 -in /etc/ssl/llama-cpp/llama-cpp.crt -noout -subject -enddate -ext subjectAltName

301 https://ec2-54-166-74-253.compute-1.amazonaws.com/v1/models
subject=CN = ec2-54-166-74-253.compute-1.amazonaws.com, O = cloudimg
notAfter=Jan 22 01:48:08 2029 GMT
X509v3 Subject Alternative Name: 
    DNS:ec2-54-166-74-253.compute-1.amazonaws.com, IP Address:54.166.74.253, IP Address:172.31.87.137, DNS:ip-172-31-87-137.ec2.internal, IP Address:127.0.0.1, DNS:localhost

To use your own name, point a DNS record at the instance (an Elastic IP keeps the address stable across stops), then run the helper. It records the name in /etc/cloudimg/llama-cpp-hostname, issues a new self-signed certificate for it, updates the credentials file and reloads nginx:

sudo llama-cpp-set-hostname <your-dns-name>

For a publicly trusted certificate, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate, or replace /etc/ssl/llama-cpp/llama-cpp.crt and llama-cpp.key with your own and run sudo systemctl reload nginx.

Step 9: Check the security controls

llama-server carries the metadata block and runs with the slot monitor and property changes off; nginx refuses cross-origin writes and requests for any hostname it was not configured with:

systemctl show -p IPAddressDeny llama-cpp.service
K=$(sudo grep '^LLAMA_CPP_LOOPBACK_API_KEY=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") -w ' HTTP_%{http_code}\n' http://127.0.0.1:8080/slots
curl -s -K <(printf 'header = "Authorization: Bearer %s"\n' "$K") -w ' HTTP_%{http_code}\n' -H 'Content-Type: application/json' -d '{}' http://127.0.0.1:8080/props
H=$(sudo grep '^LLAMA_CPP_HOST=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
P=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s --cacert /etc/ssl/llama-cpp/llama-cpp.crt -o /dev/null -w 'cross-origin POST: %{http_code}\n' -K <(printf 'user = "admin:%s"\n' "$P") \
  -X POST -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{}' "https://$H/v1/chat/completions"
code=$(curl -s -o /dev/null -w '%{http_code}' -H 'Host: evil.example' http://127.0.0.1/ || true); echo "unknown Host: ${code} (connection closed by nginx)"

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
{"error":{"code":501,"message":"This server does not support slots endpoint. Start it with `--slots`","type":"not_supported_error"}} HTTP_501
{"error":{"code":501,"message":"This server does not support changing global properties. Start it with `--props`","type":"not_supported_error"}} HTTP_501
cross-origin POST: 403
unknown Host: 000 (connection closed by nginx)

The cloudimg build also proves, before the image is captured, that a process inside llama-cpp.service cannot reach the metadata service while the same process still reaches loopback, that a forged X-Forwarded-For never reaches llama-server, and that no password or key appears in the journal, /var/log or any process command line.

Maintenance

• Logs: sudo journalctl -u llama-cpp.service -f for the inference server, /var/log/nginx/ for HTTPS access.
• Restart: sudo systemctl restart llama-cpp.service (the model reloads in a few seconds).
• Operating system updates: Ubuntu's unattended security upgrades stay on, so nginx and the rest of the OS keep receiving patches. Apply the rest with sudo apt update && sudo apt upgrade in a maintenance window.
• Upgrading llama.cpp: llama.cpp is a pinned release in /opt/llama-cpp/llama-b9937. Upgrade deliberately: unpack a newer official llama-<build>-bin-ubuntu-x64.tar.gz release next to it, point ExecStart and LD_LIBRARY_PATH in /etc/systemd/system/llama-cpp.service at the new directory, keep the existing options, then sudo systemctl daemon-reload && sudo systemctl restart llama-cpp.
• Rotating the password: printf '%s\n' '<new-password>' | sudo htpasswd -iB /etc/nginx/.llama-cpp.htpasswd admin, update /root/llama-cpp-credentials.txt, then sudo systemctl reload nginx.
• Rotating llama-server's key: remove /etc/llama-cpp/api-key and /etc/nginx/llama-cpp-upstream-auth.conf, run sudo systemctl restart llama-cpp-apikey llama-cpp nginx (a new key is minted), and update LLAMA_CPP_LOOPBACK_API_KEY in the credentials file if you use it.

Troubleshooting

The page does not answer right after launch. nginx starts only when first boot has finished, which takes a few minutes while the model volume is read for the first time. Follow it with sudo journalctl -u llama-cpp-firstboot.service -f and wait for First boot complete.

The browser keeps asking for a password. Use admin and LLAMA_CPP_PASSWORD from /root/llama-cpp-credentials.txt exactly, over https://.

Replies are slow. CPU inference speed scales with vCPUs and memory bandwidth. Use a larger, non-burstable instance (m5.xlarge or above) or a smaller model; check the thread count matches nproc.

nginx will not start. It waits for first boot to create the certificate and password by design. Read the first-boot log with sudo journalctl -u llama-cpp-firstboot.service --no-pager | tail -20.

llama-cpp.service fails after a model change. Read sudo journalctl -u llama-cpp.service -n 50 --no-pager: the usual causes are a wrong --model path, a file the llama-cpp user cannot read, or a model too large for the instance's memory.

Support

cloudimg provides 24/7 support for this image by email and live chat: launching and connecting, models and sizing, TLS and hostnames, and upgrades.

Trademarks

llama.cpp is developed by the ggml-org community. Qwen is a trademark of Alibaba Cloud. Ubuntu is a registered trademark of Canonical Ltd. All other product and company names are trademarks or registered trademarks of their respective holders. cloudimg is not affiliated with or endorsed by them.