vLLM on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and configuration of vLLM on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. vLLM is a high-throughput, memory-efficient inference and serving engine for large language models. Its PagedAttention scheduler delivers leading serving throughput, and it exposes an OpenAI-compatible REST API (/v1/chat/completions, /v1/completions, /v1/models) so existing OpenAI SDK code works unchanged.

The image installs the pinned vLLM release into a dedicated Python virtual environment at /opt/vllm and runs it under systemd. The vLLM server binds to loopback only (127.0.0.1:8000) with its native API-key authentication enabled; nginx is the sole network-facing surface and terminates TLS on port 443 while independently enforcing the same per-VM bearer API key.

GPU required. vLLM serves models on an NVIDIA GPU. Launch this image on an Azure NC or ND-series (Ampere or newer) GPU VM and add the NVIDIA GPU Driver Extension; vllm.service then starts automatically and serves the model on the GPU. On a VM with no GPU the service intentionally does not start — it logs a clear diagnostic rather than falling back to a broken state. See Step 3 to enable GPU acceleration.

Security by design — no anonymous access, no default login. vLLM's OpenAI-compatible API ships open, so an exposed port is an open model API for anyone who can reach it. On the very first boot, this image generates a per-VM bearer API key and writes it into both the vLLM service environment and the nginx auth map, and it regenerates a fresh self-signed TLS certificate for this specific VM (the certificate Subject Alternative Names include the VM public IP and hostname). Anonymous requests to the inference API return HTTP 401 with a WWW-Authenticate: Bearer challenge; only the per-VM key is accepted. The open GET /health endpoint stays available for load-balancer probes. The key is written to /root/vllm-credentials.txt (mode 0600, root only). No shared or default key ships in the image.

What is included:

• vLLM installed into a dedicated Python 3.12 virtual environment at /opt/vllm, run under systemd as the unprivileged vllm system user

• The vLLM server bound to loopback only (127.0.0.1:8000) — never directly network exposed — with its native API-key authentication enabled

• nginx terminating TLS on :443 and enforcing a per-VM bearer API key on the inference API; GET /health stays open for health probes

• Port :80 returns a 301 redirect to HTTPS for every path except an unauthenticated /health endpoint (HTTP 200)

• A per-VM bearer API key generated on first boot and stored in /root/vllm-credentials.txt (0600) — no anonymous access, no shared default key

• A self-signed TLS certificate regenerated per VM on first boot (SAN includes the VM public IP + hostname) — replace it with your own CA-signed certificate for production

• A small open-weights starter model (Qwen/Qwen2.5-1.5B-Instruct, Apache-2.0) pre-downloaded into the HuggingFace cache at /var/lib/vllm/hf so the API serves a model on a GPU VM out of the box

• A best-effort update to the latest vLLM release on first boot (offline-safe: ships a working pinned build regardless)

• Ubuntu 24.04 LTS base with latest security patches applied at build time

• Azure Linux Agent for seamless cloud integration and SSH key injection

• 24/7 cloudimg support with guaranteed 24 hour response SLA

Prerequisites

• Active Azure subscription, SSH public key, VNet + subnet in target region

• Subscription to the vLLM listing on Azure Marketplace

Recommended virtual machine size: vLLM requires an NVIDIA GPU for inference, and its default attention backend requires Ampere or newer (it does not support the older Turing/T4 generation). A good starting point is Standard_NC24ads_A100_v4 (24 vCPU, 220 GB RAM, one NVIDIA A100 80 GB); the A10-based NVadsA10_v5 sizes also work well. Larger models and higher concurrency need more GPU memory. The image installs and configures correctly on any VM size, but real inference only runs once the VM has an NVIDIA GPU and driver.

Step 1: Deploy from the Azure Portal

Search vLLM in Marketplace, select the cloudimg publisher, click Create, and choose an NC or ND-series GPU VM size. NSG rules: TCP 22 (admin), TCP 443 (inference API over HTTPS), and TCP 80 (HTTP to HTTPS redirect and health probe) from your client networks.

Step 2: Deploy from the Azure CLI

# Deploy on a GPU VM size (subscription <sub-id>). Replace the gallery image version.
RG="vllm-prod"; LOCATION="eastus"; VM_NAME="vllm-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/vllm-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name vllm-vnet --address-prefix 10.100.0.0/16 --subnet-name vllm-subnet --subnet-prefix 10.100.1.0/24
az network nsg create -g "$RG" --name vllm-nsg
az network nsg rule create -g "$RG" --nsg-name vllm-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name vllm-nsg --name allow-https --priority 110 \
  --destination-port-ranges 443 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name vllm-nsg --name allow-http --priority 120 \
  --destination-port-ranges 80 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_NC24ads_A100_v4 --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name vllm-vnet --subnet vllm-subnet --nsg vllm-nsg --public-ip-sku Standard

Step 3: Enable GPU acceleration (NVIDIA GPU Driver Extension)

vLLM needs the NVIDIA driver present on the VM. On an NC/ND-series GPU VM, add the NVIDIA GPU Driver Extension, which installs the datacenter driver:

# Add the NVIDIA driver to the GPU VM you created in Step 2 (subscription <sub-id>).
az vm extension set -g "$RG" --vm-name "$VM_NAME" \
  --name NvidiaGpuDriverLinux --publisher Microsoft.HpcCompute --version 1.9 \
  --settings '{}'

Once the driver is installed, vllm.service starts automatically (it is enabled) and serves the model on the GPU. Confirm the GPU is visible with nvidia-smi and the service is active with sudo systemctl status vllm.service.

Step 4: Confirm the service and retrieve the API key

SSH in as azureuser. vLLM is installed in the /opt/vllm venv; nginx fronts it on :443, and the vLLM server binds loopback :8000 only:

/opt/vllm/bin/python -c 'import vllm; print("vLLM", vllm.__version__)'
systemctl is-active nginx.service
systemctl is-enabled vllm.service
ss -tln | grep -E ':443 |:80 '

Retrieve the per-VM bearer API key (root-only file):

sudo grep '^VLLM_API_KEY=' /root/vllm-credentials.txt

Step 5: The security model — no default login

vLLM ships with no authentication, so this image never exposes it directly. nginx enforces the per-VM bearer key: the open health endpoint answers without auth, an anonymous inference request is rejected with 401, and only the per-VM key is accepted. Verify it on the VM (the -k flag skips the self-signed cert check; in production use --cacert /etc/nginx/tls/vllm.crt or your own CA cert):

KEY="$(sudo grep '^VLLM_API_KEY=' /root/vllm-credentials.txt | cut -d= -f2-)"
echo -n 'open   /health          -> '; curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/health
echo -n 'anon   /v1/models       -> '; curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/v1/models
echo -n 'keyed  /v1/models       -> '; curl -sk -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $KEY" https://127.0.0.1/v1/models

The open endpoint returns 200, the anonymous inference request returns 401, and the keyed request is accepted by nginx (it returns the model list as 200 once the GPU-backed vLLM upstream is serving; before then nginx forwards the accepted request and reports the upstream as unavailable).

Step 6: GPU-required behaviour

vLLM is GPU-required. On a VM with no NVIDIA GPU, vllm.service does not start — it logs a clear diagnostic so the state is never ambiguous. Once the NVIDIA driver is present (Step 3), the service starts and serves on the GPU.

Step 7: Call the OpenAI-compatible API

From a client that can reach the VM, list models and run a chat completion. Replace <vm-ip> with your VM public IP and <key> with the value from Step 4 (add --cacert with the VM certificate, or -k for the self-signed cert during testing):

# List the served models
curl -k -H "Authorization: Bearer <key>" https://<vm-ip>/v1/models

# Chat completion
curl -k -H "Authorization: Bearer <key>" https://<vm-ip>/v1/chat/completions \
  -d '{"model":"Qwen/Qwen2.5-1.5B-Instruct","messages":[{"role":"user","content":"Say hello in one sentence."}]}'

A successful /v1/chat/completions call returns an OpenAI-style JSON response with a choices[0].message.content field containing the model's reply.

Step 8: Configuration and serving a different model

The service reads its model name and API key from /etc/vllm/vllm.env; the starter model is pre-downloaded into the HuggingFace cache:

sudo sed 's/^VLLM_API_KEY=.*/VLLM_API_KEY=REDACTED/' /etc/vllm/vllm.env
find -L /var/lib/vllm/hf -type f -name '*.safetensors' | sed 's#.*/hub/##'

To serve a different model, edit MODEL in /etc/vllm/vllm.env and restart the service (larger models need more GPU memory):

# On the VM, as root — set MODEL to any HuggingFace model id (subscription <sub-id> example):
sudo sed -i 's#^MODEL=.*#MODEL=meta-llama/Llama-3.1-8B-Instruct#' /etc/vllm/vllm.env
sudo systemctl restart vllm.service

Step 9: Use with the OpenAI SDK

Point any OpenAI SDK, LangChain, or LlamaIndex client at the VM by setting the base URL and the per-VM key:

from openai import OpenAI
client = OpenAI(base_url="https://<vm-ip>/v1", api_key="<key>")
resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Explain PagedAttention in one sentence."}],
)
print(resp.choices[0].message.content)

For production, replace the self-signed certificate at /etc/nginx/tls/vllm.crt and /etc/nginx/tls/vllm.key with your own CA-signed certificate (keep the same paths) and reload nginx, so clients can verify the endpoint without -k.

Step 10: Model storage and backups

The model cache lives at /var/lib/vllm/hf on the OS disk. For large model libraries, attach a separate Azure managed data disk, format and mount it, and point HF_HOME at it in /etc/vllm/vllm.env, so model weights sit on an independently sized and snapshot-able volume. Snapshot the disk with the Azure CLI on your own schedule.

Support

24/7 technical support is included with every cloudimg image. Contact support@cloudimg.co.uk for help with vLLM deployment, model selection, GPU sizing, throughput tuning, quantization, the OpenAI-compatible API, TLS termination, and scaling. Critical issues receive a one-hour average response.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.