Infinity on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and configuration of Infinity on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. Infinity is a high-throughput, low-latency REST server for text embeddings, reranking and CLIP models. It serves models such as BGE, GTE, E5, Sentence Transformers and mixedbread with dynamic batching and an OpenAI-compatible embeddings API (/embeddings), so existing OpenAI SDK code works unchanged.

The image installs the pinned Infinity release (infinity-emb) into a dedicated Python virtual environment at /opt/infinity/venv and runs it under systemd. The Infinity server binds to loopback only (127.0.0.1:7997) and runs with no authentication of its own; nginx is the sole network facing surface and terminates TLS on port 443 while enforcing HTTP Basic authentication against a bcrypt htpasswd file.

Security by design — no anonymous access. Infinity ships with no built-in authentication, so an open port is an open embeddings API for anyone who can reach it. On the very first boot, this image generates a per-VM admin credential and writes it into the nginx htpasswd file, and it regenerates a fresh self-signed TLS certificate for this specific VM (the certificate Subject Alternative Names include the VM public IP and hostname). Anonymous requests to the API return HTTP 401 with a WWW-Authenticate: Basic challenge; only the per-VM credential is accepted. The open GET /health endpoint stays available for load balancer probes. The credential is written to /root/infinity-credentials.txt (mode 0600, root only).

CPU capable, GPU ready. The image generates embeddings on CPU on any VM size and ships PyTorch's bundled CUDA runtime, so it can offload to the GPU when launched on an NC or ND-series GPU VM with the NVIDIA driver present (see Step 9). A small open-weights model (BAAI/bge-small-en-v1.5, 384 dimensions) is pre-downloaded so the API responds immediately.

What is included:

• Infinity installed into a Python virtual environment at /opt/infinity/venv, run under systemd as the unprivileged infinity system user, with the model cache at /var/lib/infinity/hf

• The Infinity server bound to loopback only (127.0.0.1:7997) — never directly network exposed

• nginx terminating TLS on :443 and enforcing htpasswd Basic auth on the API; GET /health and /healthz stay open for health probes

• Port :80 returns a 301 redirect to HTTPS for every path except an unauthenticated /healthz endpoint (HTTP 200)

• A per-VM admin credential generated on first boot and stored in /root/infinity-credentials.txt (0600) — no anonymous access, no shared default password

• A self-signed TLS certificate regenerated per VM on first boot (SAN includes the VM public IP + hostname) — replace it with your own CA-signed certificate for production

• A pre-downloaded BAAI/bge-small-en-v1.5 model so the API answers on launch; serve a different model by editing one line of configuration

• A best-effort update to the latest Infinity release on first boot (offline-safe: ships a working pinned build regardless)

• Ubuntu 24.04 LTS base with latest security patches applied at build time

• Azure Linux Agent for seamless cloud integration and SSH key injection

• 24/7 cloudimg support with guaranteed 24 hour response SLA

Prerequisites

• Active Azure subscription, SSH public key, VNet + subnet in target region

• Subscription to the Infinity listing on Azure Marketplace

Recommended virtual machine size: for high-throughput embedding workloads, launch on a GPU VM — Standard_NC4as_T4_v3 (4 vCPU, 28 GB RAM, one NVIDIA T4) is a good entry point, and larger NC/ND-series VMs serve bigger models and more concurrency. The image also runs on CPU-only sizes such as Standard_B4ms or Standard_D4s_v5 for light use and testing, at lower throughput. See Step 9 to enable GPU acceleration.

Step 1: Deploy from the Azure Portal

Search Infinity in Marketplace, select the cloudimg publisher, click Create. NSG rules: TCP 22 (admin), TCP 443 (API over HTTPS), and TCP 80 (HTTP to HTTPS redirect and health probe) from your client networks.

Step 2: Deploy from the Azure CLI

RG="infinity-prod"; LOCATION="eastus"; VM_NAME="infinity-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/infinity-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name infinity-vnet --address-prefix 10.100.0.0/16 --subnet-name infinity-subnet --subnet-prefix 10.100.1.0/24
az network nsg create -g "$RG" --name infinity-nsg
az network nsg rule create -g "$RG" --nsg-name infinity-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name infinity-nsg --name allow-https --priority 110 \
  --destination-port-ranges 443 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name infinity-nsg --name allow-http --priority 120 \
  --destination-port-ranges 80 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_NC4as_T4_v3 --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name infinity-vnet --subnet infinity-subnet --nsg infinity-nsg --public-ip-sku Standard

Step 3: Connect via SSH and retrieve your credential

The per-VM admin credential is generated on the first boot and stored in a root only file. Connect over SSH and read it:

ssh azureuser@<public-ip>
sudo cat /root/infinity-credentials.txt

The file contains the API URL, the served model, the username (admin) and the per-VM password, plus quick start commands. Store the password somewhere safe — it is unique to this VM.

Step 4: Verify the Infinity and nginx services

Confirm both services are active and that Infinity is bound to loopback only. In the listener output, 127.0.0.1:7997 (Infinity) is loopback only, while nginx listens publicly on :443 and :80:

systemctl status infinity nginx --no-pager
ss -tln | grep -E ':7997|:443|:80 '

Step 5: Check the health endpoint

GET /health is served by nginx without authentication (ideal for load balancer and Azure health probes):

curl -s -o /dev/null -w 'health: HTTP %{http_code}\n' --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/health
curl -k -o /dev/null -w 'healthz: HTTP %{http_code}\n' https://127.0.0.1/healthz

Both return HTTP 200.

Step 6: Generate an embedding and confirm anonymous access is rejected

Use the per-VM credential to generate an embedding through the API, then confirm that an anonymous request (no credential) is rejected with HTTP 401. The self-signed certificate lives at /etc/nginx/tls/infinity.crt, so we pass it with --cacert:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
MODEL=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
echo "# Authenticated embedding (vector dimension):"
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/embeddings \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"cloudimg makes self-hosted AI easy\"]}" | jq '.data[0].embedding | length'
echo "# Anonymous request (no credential) must be rejected:"
curl -s -o /dev/null -w 'anonymous /embeddings -> HTTP %{http_code}\n' --cacert /etc/nginx/tls/infinity.crt \
  -X POST https://127.0.0.1/embeddings -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"x\"]}"

The authenticated request returns the embedding length (384 for the default model); the anonymous request returns HTTP 401. There is no anonymous access to this API.

Step 7: Batch embeddings for semantic search

Infinity embeds a whole batch in a single call with dynamic batching, so you can vectorise a query and its candidate documents together and then rank the documents by cosine similarity to the query. Send several inputs at once and confirm you get one vector per input:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
MODEL=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/embeddings \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"What is the capital of France?\",\"Paris is the capital of France.\",\"Berlin is a city in Germany.\"]}" \
  | jq '{vectors: (.data | length), dim: (.data[0].embedding | length), tokens: .usage.total_tokens}'

The response contains one embedding per input (three vectors of 384 dimensions here). Compute cosine similarity between the query vector and each document vector in your application to rank documents — the query matches the France sentence far more closely than the Germany one.

Reranking and classification. Infinity also serves /rerank (cross-encoder reranking) and /classify (sequence classification). Those endpoints require a model with the matching capability — the shipped BAAI/bge-small-en-v1.5 is embedding-only. Serve a reranker (for example mixedbread-ai/mxbai-rerank-base-v1 or BAAI/bge-reranker-base) or a classifier by editing MODEL in /etc/infinity/infinity.env and restarting the service (Step 13), and the corresponding endpoint becomes available.

Step 8: List the served models

The /models endpoint lists the models Infinity is serving (the same call the OpenAI SDK makes for client.models.list()):

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/models | jq '.'

The response is an OpenAI-style {"object":"list","data":[...]} document listing the served model id, its backend (torch) and capabilities (embed).

Step 9: Enable GPU acceleration (NC/ND-series VMs)

The image generates embeddings on CPU by default. On a GPU VM you can offload embedding to the GPU for much higher throughput:

• Add the NVIDIA GPU Driver Extension to the VM (Portal: VM → Extensions → add NVIDIA GPU Driver Extension; or az vm extension set --name NvidiaGpuDriverLinux --publisher Microsoft.HpcCompute ...). Reboot when it completes.

• Confirm the driver: nvidia-smi lists the GPU.

• Switch Infinity to the GPU: edit the ExecStart line in /etc/systemd/system/infinity.service, changing --device cpu to --device cuda, then sudo systemctl daemon-reload && sudo systemctl restart infinity.

• The service logs (sudo journalctl -u infinity -n 30) show the model loading on the CUDA backend.

The same image and model cache are used — only the device flag changes.

Step 10: Use the OpenAI-compatible endpoint from your apps

Infinity exposes an OpenAI-compatible embeddings API. Point any OpenAI SDK at https://<public-ip> (Infinity mounts /embeddings and /models at the root, with no /v1 prefix) using HTTP Basic auth. Example with curl:

PASS=$(sudo grep '^INFINITY_PASSWORD=' /root/infinity-credentials.txt | cut -d= -f2-)
MODEL=$(sudo grep '^INFINITY_MODEL=' /root/infinity-credentials.txt | cut -d= -f2-)
sudo curl -s -u "admin:$PASS" --cacert /etc/nginx/tls/infinity.crt https://127.0.0.1/embeddings \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"input\":[\"first document\",\"second document\"]}" | jq '.data | length'

From Python (LangChain, LlamaIndex, or the OpenAI SDK), set base_url="https://<public-ip>", pass the admin / password basic-auth pair, and feed the returned vectors into a vector database such as Weaviate or Chroma for retrieval augmented generation.

Step 11: Use your own domain and a trusted certificate (production)

The self-signed certificate is a convenience for getting started. For production, front Infinity with your own DNS name and a CA-signed certificate:

• Point a DNS record (for example infinity.example.com) at the VM public IP.

• Replace /etc/nginx/tls/infinity.crt and /etc/nginx/tls/infinity.key with your CA-signed certificate and key (for example from Let's Encrypt or your internal CA), then run sudo nginx -t && sudo systemctl reload nginx.

• Clients then call https://infinity.example.com/ with no certificate trust step.

Step 12: Server components

Step 13: Managing the service and changing the model

sudo systemctl status infinity --no-pager
sudo journalctl -u infinity --no-pager -n 20
sudo systemctl reload nginx

To serve a different embedding or reranking model, edit MODEL in /etc/infinity/infinity.env (any HuggingFace embedding or reranking model id), then sudo systemctl restart infinity. The model downloads into /var/lib/infinity/hf on first use.

The model cache lives under /var/lib/infinity/hf and persists across service restarts and VM reboots. For large model libraries, attach and mount an Azure data disk at /var/lib/infinity/hf (stop infinity, move the data, update the mount, restart).

Step 14: Security recommendations

• Restrict the NSG. Allow TCP 443 (and 22 for admin) only from the client networks that need the API. Do not expose the API to the public internet unless you intend to.

• Rotate the credential. To change the admin password: sudo htpasswd -B /etc/nginx/.infinity.htpasswd admin then sudo systemctl reload nginx. Update /root/infinity-credentials.txt accordingly.

• Add more users. Grant additional accounts with sudo htpasswd -B /etc/nginx/.infinity.htpasswd <username> and reload nginx.

• Use a trusted certificate (Step 11) for production so clients validate the endpoint without a manual trust step.

• Keep the OS patched. Unattended security upgrades remain enabled on the running VM.

Step 15: Support and Licensing

Infinity is distributed under the MIT License. This cloudimg image bundles the unmodified upstream Infinity release. cloudimg provides the packaging, hardening, per-VM credential and TLS automation, and 24/7 support with a guaranteed 24 hour response SLA.

Deploy on Azure

Find Infinity on Ubuntu 24.04 on the Azure Marketplace, published by cloudimg. Deploy from the Portal or the Azure CLI as shown above.

Need Help?

Email support@cloudimg.co.uk for deployment help, configuration questions, or licensing enquiries.