Developer Tools Azure

llama.cpp Server on Ubuntu 24.04 on Azure User Guide

Last updated: 2026-07-09 | Product: llama.cpp Server on Ubuntu 24.04 LTS on Azure

Overview

llama.cpp is the reference CPU-first inference engine for GGUF-quantized large language models - no GPU required. This appliance installs upstream's own llama-server binary (the official prebuilt CPU release, not a third-party wrapper) which serves two things from a single process: a built-in browser chat UI and an OpenAI-compatible REST API (/v1/chat/completions, /v1/completions, /v1/models). llama-server has no login system of its own, so the cloudimg image locks it down for a marketplace appliance the same way as the committed GoAccess/Zipkin builds - llama-server is bound to loopback only, and nginx does all customer-facing work on port 80: a per-VM HTTP Basic Auth gate protects both the chat UI and the API, and an unauthenticated /healthz endpoint serves load balancer probes. A small, genuinely redistributable demo model - Qwen2.5-1.5B-Instruct, Apache License 2.0 - is pre-downloaded onto a dedicated data disk so you can start chatting or calling the API within a minute of first boot, with no internet access required at runtime. Backed by 24/7 cloudimg support.

What is included:

llama.cpp build b9937 (ggml-org/llama.cpp, MIT license) installed from the official prebuilt CPU-only Linux release - no compiler toolchain needed
The built-in llama.cpp chat UI and the full OpenAI-compatible API on :80, fronted by nginx with llama-server bound to loopback only
Per-VM HTTP Basic Auth (user admin) protecting both the chat UI and the API, with a unique password generated on first boot
A bundled demo model - Qwen2.5-1.5B-Instruct, Q4_K_M GGUF quantization (~1.04 GiB), published under the Apache License 2.0 by the Qwen team - pre-loaded and ready to chat with immediately
A dedicated Azure data disk at /var/lib/llama-cpp holding the model file(s), independently resizable and re-provisioned on every deploy
llama-cpp.service + nginx.service as systemd units, enabled and active; thread count auto-tuned to your VM's actual vCPU count on first boot
An unauthenticated /healthz endpoint on :80 for Azure Load Balancer health probes
24/7 cloudimg support

About the bundled demo model - license and attribution

This image ships Qwen2.5-1.5B-Instruct (quantized to GGUF Q4_K_M) so the chat UI and API work immediately after first boot with no model download or internet access required. This model is published by the Qwen team (Alibaba Cloud) and, per its Hugging Face model card metadata, is licensed under the Apache License 2.0 - separate from and in addition to llama.cpp's own MIT license. Apache 2.0 permits commercial use and redistribution, which is why this specific model (rather than the 3B or 72B Qwen2.5 variants, which use a different, more restrictive Qwen License) was chosen for bundling. If you replace it with a different model, check that model's own license before redistributing or relying on it in production.

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet + subnet in the target region. Standard_D4s_v3 (4 vCPU / 16 GiB RAM, dedicated, non-burstable) is the recommended starting point - CPU inference is compute-bound, so a burstable B-series size will throttle under sustained chat/API use once its credits are exhausted. Size up for a larger model or higher concurrent request volume. NSG inbound: allow 22/tcp from your management network, 80/tcp for the chat UI and API, and 443/tcp if you add TLS. llama-server serves plain HTTP; for production use, terminate TLS in front of it with your own domain.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for llama.cpp by cloudimg, and select Create. On Basics pick your subscription, resource group, region and a 4+ vCPU size (Standard_D4s_v3 recommended); under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name llama-cpp \
  --image <marketplace-image-urn> \
  --size Standard_D4s_v3 \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name llama-cpp --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active llama-cpp.service nginx.service

Both report active. llama-server serves the chat UI and the full OpenAI-compatible API on the loopback address 127.0.0.1:8080 only; nginx fronts port 80 with the per-VM HTTP Basic Auth gate, and that's the only public path in. The bundled demo model lives on the dedicated data disk mounted at /var/lib/llama-cpp.

The llama-cpp and nginx services active, llama-server listening on loopback 127.0.0.1:8080 only, and the dedicated data disk mounted at /var/lib/llama-cpp

Step 5 - Retrieve your password

nginx protects the chat UI and API with HTTP Basic Auth. The username is admin and a unique password is generated on the first boot of your VM and written to a root-only file:

sudo cat /root/llama-cpp-credentials.txt

This file contains LLAMA_CPP_USERNAME, LLAMA_CPP_PASSWORD, the LLAMA_CPP_URL to open in a browser, an example curl command against the API, and a note on the bundled model's license. The password is stored on disk only as a bcrypt hash in /etc/nginx/.llama-cpp.htpasswd, so no plaintext password ships in the image. Store the password somewhere safe.

The per-VM credentials file with the generated admin password, an example API curl command, and the bundled demo model's license note; and confirmation that the nginx Basic Auth file is a bcrypt hash with no plaintext password shipped in the image

Step 6 - Confirm the health endpoint and authentication

nginx serves an unauthenticated health endpoint for load balancers and probes, while the chat UI and API require the per-VM password:

PW=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
echo "healthz : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/healthz)"
echo "unauth  : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/)"
echo "wrongpw : $(curl -s -o /dev/null -w '%{http_code}' -u admin:wrong-pw http://127.0.0.1/)"
echo "authed  : $(curl -s -o /dev/null -w '%{http_code}' --compressed -u admin:$PW http://127.0.0.1/)"

It prints healthz : 200, unauth : 401, wrongpw : 401, authed : 200. Only the per-VM password reaches the chat UI and API, because llama-server itself is bound to loopback and nginx port 80 is the only way in. (The --compressed flag is needed for the chat UI specifically - its static assets are served gzip-only.)

The HTTP Basic Auth round-trip on port 80: an unauthenticated healthz probe returning 200, an unauthenticated UI request returning 401, a wrong password rejected with 401, and the per-VM password authenticating with 200

Step 7 - Call the OpenAI-compatible chat completions API

PW=$(sudo grep '^LLAMA_CPP_PASSWORD=' /root/llama-cpp-credentials.txt | cut -d= -f2-)
curl -s -u admin:$PW -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"What is the capital of France? Answer in one short sentence."}],"max_tokens":60}' \
  http://127.0.0.1/v1/chat/completions | python3 -m json.tool

The response is a standard OpenAI-shaped chat completion - choices[0].message.content holds the generated answer, model reports the loaded model's alias (qwen2.5-1.5b-instruct), and timings reports live generation speed in tokens/second, useful for judging whether your VM size is a good fit for your workload. Point any OpenAI-API-compatible client or SDK at http://<vm-ip>/v1/ with HTTP Basic Auth credentials to integrate llama.cpp Server into your own applications.

An authenticated POST to /v1/chat/completions returning a real generated answer, the loaded model alias, and live tokens/second generation speed

Step 8 - Sign in and chat in the browser

Browse to http://<vm-public-ip>/. Your browser prompts for a username and password: enter admin and the password from Step 5. llama.cpp's built-in chat UI opens, showing the loaded model (qwen2.5 1.5B instruct) in the message composer.

The llama.cpp built-in chat UI landing view, behind the Basic Auth gate, with the loaded demo model shown in the message composer

Type a question and press Enter. The UI streams the response back token by token and reports generation stats (tokens, time, tokens/second) under each message.

A chat conversation in progress in the llama.cpp UI - a user question and the generated answer, with per-message token count, latency and tokens/second stats

Step 9 - Explore the model/sampling settings

Click the gear icon to open Settings, then Sampling to see and adjust the inference parameters the UI sends with every request - temperature, top-K, top-P and more - each with an inline explanation of what it controls. These settings are stored in your browser and applied to the next request; they do not change the systemd service's own defaults.

The llama.cpp UI Settings > Sampling panel, showing inference parameters such as Temperature, Dynamic temperature range/exponent and Top K with inline explanations

Step 10 - List the loaded model via the API

GET /v1/models (also reachable directly in a browser once authenticated) returns the OpenAI-compatible model list, confirming the alias, context size and quantization of the model llama-server currently has loaded:

A real authenticated GET /v1/models API response viewed in-browser, showing the qwen2.5-1.5b-instruct model's id, context size and Q4_K_M quantization

Swapping in a different or larger model

The bundled Qwen2.5-1.5B-Instruct model is intentionally small so first boot is fast and inference stays responsive on a modest VM. For more capability, download a different GGUF model (for example a larger Qwen2.5, Llama, or Mistral instruct model in GGUF format) onto the data disk and point llama-server at it:

sudo curl -fL -o /var/lib/llama-cpp/<your-model>.gguf "<model-download-url>"
sudo chown llama-cpp:llama-cpp /var/lib/llama-cpp/<your-model>.gguf
sudo sed -i 's|--model [^ ]*|--model /var/lib/llama-cpp/<your-model>.gguf|' /etc/systemd/system/llama-cpp.service
sudo systemctl daemon-reload
sudo systemctl restart llama-cpp

A larger model needs more vCPU and RAM than this VM's current size - resize the VM (or redeploy onto a larger size) before switching to a 7B+ parameter model. Always check the license of any model you download before using it commercially.

Maintenance

Password: the UI/API password is set on first boot and stored as a bcrypt entry in /etc/nginx/.llama-cpp.htpasswd. To change it, run sudo htpasswd -B /etc/nginx/.llama-cpp.htpasswd admin and then sudo systemctl reload nginx.
Thread count: llama-server's --threads flag is set from nproc at first boot, so it always matches your VM's actual vCPU count - even if you resize the VM after deployment, a re-run of first boot (or a manual edit of /etc/systemd/system/llama-cpp.service) keeps it correctly tuned.
Context size: the default --ctx-size 4096 balances memory use against conversation length. Increase it in /etc/systemd/system/llama-cpp.service for longer conversations/documents (needs more RAM), then sudo systemctl daemon-reload && sudo systemctl restart llama-cpp.
Restrict access: llama-server serves plain HTTP on port 80. For production, restrict access to trusted IP ranges in your Network Security Group, and front it with TLS (for example certbot with your own domain) terminating on :443.
Loopback binding: llama-server is bound to 127.0.0.1:8080 in /etc/systemd/system/llama-cpp.service (--host 127.0.0.1), so nginx is the only path in. Keep it that way - do not change the bind address to a public interface.
Data disk: the model file(s) live on the dedicated data disk at /var/lib/llama-cpp, independently resizable from the OS disk.
Logs: journalctl -u llama-cpp shows recent model load and inference activity; journalctl -u nginx shows the HTTP access/error logs.
Security patches: unattended-upgrades remains enabled so the OS continues to receive security updates automatically.

Support

cloudimg provides 24/7 expert support for this image. Contact support@cloudimg.co.uk.