Ollama on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and configuration of Ollama on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. Ollama is the easiest way to run open large language models on your own infrastructure. It downloads, quantizes and serves models such as Llama, Mistral, Gemma, Phi, Qwen and DeepSeek with a single command, exposing a REST API that is also OpenAI chat-completions compatible (/v1/chat/completions).

The image installs the official pinned Ollama release to /usr/local/bin/ollama and runs it under systemd. The Ollama server binds to loopback only (127.0.0.1:11434) and runs with no authentication of its own; nginx is the sole network facing surface and terminates TLS on port 443 while enforcing HTTP Basic authentication against a bcrypt htpasswd file.

Security by design — no anonymous access. Ollama ships with no built-in authentication, so an open port is an open model API for anyone who can reach it. On the very first boot, this image generates a per-VM admin credential and writes it into the nginx htpasswd file, and it regenerates a fresh self-signed TLS certificate for this specific VM (the certificate Subject Alternative Names include the VM public IP and hostname). Anonymous requests to the model API return HTTP 401 with a WWW-Authenticate: Basic challenge; only the per-VM credential is accepted. The open GET /api/version health endpoint stays available for load balancer probes. The credential is written to /root/ollama-credentials.txt (mode 0600, root only).

CPU capable, GPU ready. The image runs models on CPU on any VM size and ships Ollama's bundled CUDA runtime, so it auto-offloads inference to the GPU when launched on an NC or ND-series GPU VM with the NVIDIA driver present. A small starter model (llama3.2:1b) is pre-pulled so the API responds immediately.

What is included:

• Ollama installed to /usr/local/bin/ollama, run under systemd as the unprivileged ollama system user, with the model store at /var/lib/ollama/models

• The Ollama server bound to loopback only (127.0.0.1:11434) — never directly network exposed

• nginx terminating TLS on :443 and enforcing htpasswd Basic auth on the model API; GET /api/version and /healthz stay open for health probes

• Port :80 returns a 301 redirect to HTTPS for every path except an unauthenticated /healthz endpoint (HTTP 200)

• A per-VM admin credential generated on first boot and stored in /root/ollama-credentials.txt (0600) — no anonymous access, no shared default password

• A self-signed TLS certificate regenerated per VM on first boot (SAN includes the VM public IP + hostname) — replace it with your own CA-signed certificate for production

• A pre-pulled llama3.2:1b starter model so the API answers on launch; pull more models with one command

• A best-effort update to the latest Ollama release on first boot (offline-safe: ships a working pinned build regardless)

• Ubuntu 24.04 LTS base with latest security patches applied at build time

• Azure Linux Agent for seamless cloud integration and SSH key injection

• 24/7 cloudimg support with guaranteed 24 hour response SLA

Prerequisites

• Active Azure subscription, SSH public key, VNet + subnet in target region

• Subscription to the Ollama listing on Azure Marketplace

Recommended virtual machine size: for real inference workloads, launch on a GPU VM — Standard_NC4as_T4_v3 (4 vCPU, 28 GB RAM, one NVIDIA T4) is a good entry point, and larger NC/ND-series VMs serve bigger models and more concurrency. The image also runs on CPU-only sizes such as Standard_B4ms or Standard_D4s_v5 for light use and testing, at lower throughput. See Step 9 to enable GPU acceleration.

Step 1: Deploy from the Azure Portal

Search Ollama in Marketplace, select the cloudimg publisher, click Create. NSG rules: TCP 22 (admin), TCP 443 (model API over HTTPS), and TCP 80 (HTTP to HTTPS redirect and health probe) from your client networks.

Step 2: Deploy from the Azure CLI

RG="ollama-prod"; LOCATION="eastus"; VM_NAME="ollama-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/ollama-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name ollama-vnet --address-prefix 10.100.0.0/16 --subnet-name ollama-subnet --subnet-prefix 10.100.1.0/24
az network nsg create -g "$RG" --name ollama-nsg
az network nsg rule create -g "$RG" --nsg-name ollama-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name ollama-nsg --name allow-https --priority 110 \
  --destination-port-ranges 443 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name ollama-nsg --name allow-http --priority 120 \
  --destination-port-ranges 80 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_NC4as_T4_v3 --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name ollama-vnet --subnet ollama-subnet --nsg ollama-nsg --public-ip-sku Standard

Step 3: Connect via SSH and retrieve your credential

The per-VM admin credential is generated on the first boot and stored in a root only file. Connect over SSH and read it:

ssh azureuser@<public-ip>
sudo cat /root/ollama-credentials.txt

The file contains the API URL, the username (admin) and the per-VM password, plus quick start commands. Store the password somewhere safe — it is unique to this VM.

Step 4: Verify the Ollama and nginx services

Confirm both services are active and that Ollama is bound to loopback only. In the listener output, 127.0.0.1:11434 (Ollama) is loopback only, while nginx listens publicly on :443 and :80:

systemctl status ollama nginx --no-pager
ss -tln | grep -E ':11434|:443|:80 '

Step 5: Check the health endpoint

GET /api/version is served by nginx without authentication (ideal for load balancer and Azure health probes) and returns the Ollama version as JSON:

curl -s --cacert /etc/nginx/tls/ollama.crt https://127.0.0.1/api/version; echo
curl -k -o /dev/null -w 'healthz: HTTP %{http_code}\n' https://127.0.0.1/healthz

Step 6: List models and confirm anonymous access is rejected

Use the per-VM credential to list models through the API, then confirm that an anonymous request (no credential) is rejected with HTTP 401. The self-signed certificate lives at /etc/nginx/tls/ollama.crt, so we pass it with --cacert:

PASS=$(sudo grep '^OLLAMA_PASSWORD=' /root/ollama-credentials.txt | cut -d= -f2-)
echo "# Authenticated model list:"
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/ollama.crt https://127.0.0.1/api/tags | jq '.models[].name'
echo "# Anonymous request (no credential) must be rejected:"
curl -s -o /dev/null -w 'anonymous /api/tags -> HTTP %{http_code}\n' --cacert /etc/nginx/tls/ollama.crt https://127.0.0.1/api/tags

The authenticated request returns the model list as JSON; the anonymous request returns HTTP 401. There is no anonymous access to this API.

Step 7: Run a real completion (generate and chat)

Send a prompt to the pre-pulled model over the authenticated API. The /api/generate endpoint returns a completion; the OpenAI-compatible /v1/chat/completions endpoint accepts the same requests any OpenAI SDK sends.

PASS=$(sudo grep '^OLLAMA_PASSWORD=' /root/ollama-credentials.txt | cut -d= -f2-)
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/ollama.crt https://127.0.0.1/api/generate \
  -d '{"model":"llama3.2:1b","prompt":"In one sentence, what is Ollama?","stream":false}' | jq -r '.response'

The model returns generated text. On a CPU-only VM this takes a few seconds; on a GPU VM it is near instant.

Step 8: Pull more models and inspect them

Ollama serves any model from the library. From a client, pull a larger model into your VM over the authenticated API (replace <public-ip> and <password> from /root/ollama-credentials.txt):

curl -u admin:<password> --cacert ollama.crt https://<public-ip>/api/pull -d '{"name":"llama3.2:3b"}'

Inspect the models already in the local store on the VM:

sudo -u ollama OLLAMA_HOST=127.0.0.1:11434 ollama show llama3.2:1b
sudo -u ollama OLLAMA_HOST=127.0.0.1:11434 ollama list

Step 9: Enable GPU acceleration (NC/ND-series VMs)

The image runs on CPU by default and auto-offloads to the GPU once the NVIDIA driver is present. On a GPU VM:

• Add the NVIDIA GPU Driver Extension to the VM (Portal: VM → Extensions → add NVIDIA GPU Driver Extension; or az vm extension set --name NvidiaGpuDriverLinux --publisher Microsoft.HpcCompute ...). Reboot when it completes.

• Confirm the driver: nvidia-smi lists the GPU.

• Restart Ollama and confirm offload: sudo systemctl restart ollama, then run a completion and check sudo -u ollama OLLAMA_HOST=127.0.0.1:11434 ollama ps — PROCESSOR shows the GPU and size_vram is greater than zero.

No image changes are needed — the same Ollama binary detects and uses the GPU.

Step 10: Use the OpenAI-compatible endpoint from your apps

Point any OpenAI SDK at https://<public-ip>/v1 with basic auth. Example with curl:

PASS=$(sudo grep '^OLLAMA_PASSWORD=' /root/ollama-credentials.txt | cut -d= -f2-)
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/ollama.crt https://127.0.0.1/v1/chat/completions \
  -d '{"model":"llama3.2:1b","messages":[{"role":"user","content":"Say hello in French"}]}' | jq -r '.choices[0].message.content'

From Python (LangChain, LlamaIndex, or the OpenAI SDK), set base_url="https://<public-ip>/v1", api_key="ollama", and pass the admin / password basic-auth pair (and your CA-trusted or replaced certificate).

Step 11: Use your own domain and a trusted certificate (production)

The self-signed certificate is a convenience for getting started. For production, front Ollama with your own DNS name and a CA-signed certificate:

• Point a DNS record (for example ollama.example.com) at the VM public IP.

• Replace /etc/nginx/tls/ollama.crt and /etc/nginx/tls/ollama.key with your CA-signed certificate and key (for example from Let's Encrypt or your internal CA), then run sudo nginx -t && sudo systemctl reload nginx.

• Clients then call https://ollama.example.com/ with no certificate trust step.

Step 12: Server components

Step 13: Managing the service

sudo systemctl status ollama --no-pager
sudo journalctl -u ollama --no-pager -n 20
sudo systemctl reload nginx

To restart the runtime after a config change: sudo systemctl restart ollama.

Model weights live under /var/lib/ollama/models and persist across service restarts and VM reboots. For large model libraries, attach and mount an Azure data disk at /var/lib/ollama/models (stop ollama, move the data, update the mount, restart).

Step 14: Security recommendations

• Restrict the NSG. Allow TCP 443 (and 22 for admin) only from the client networks that need the API. Do not expose the model API to the public internet unless you intend to.

• Rotate the credential. To change the admin password: sudo htpasswd -B /etc/nginx/.ollama.htpasswd admin then sudo systemctl reload nginx. Update /root/ollama-credentials.txt accordingly.

• Add more users. Grant additional accounts with sudo htpasswd -B /etc/nginx/.ollama.htpasswd <username> and reload nginx.

• Use a trusted certificate (Step 11) for production so clients validate the endpoint without a manual trust step.

• Keep the OS patched. Unattended security upgrades remain enabled on the running VM.

Step 15: Support and Licensing

Ollama is distributed under the MIT License. This cloudimg image bundles the unmodified upstream Ollama release. cloudimg provides the packaging, hardening, per-VM credential and TLS automation, and 24/7 support with a guaranteed 24 hour response SLA.

Deploy on Azure

Find Ollama on Ubuntu 24.04 on the Azure Marketplace, published by cloudimg. Deploy from the Portal or the Azure CLI as shown above.

Need Help?

Email support@cloudimg.co.uk for deployment help, configuration questions, or licensing enquiries.