Ollama + AnythingLLM on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and operation of Ollama + AnythingLLM on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. The image is a private AI workspace in a single VM, built from two open source, MIT licensed projects:

• AnythingLLM is the web application you use: chat, retrieval augmented generation (RAG) over your own documents, workspaces, AI agents and multi user accounts with roles
• Ollama is the local model server. It runs open weight models on the VM itself and serves both the chat model AnythingLLM answers with and the embedding model it uses to index your documents

The two are already wired together, and two openly licensed models are baked into the image, so your first chat works without downloading anything and without any API key. Prompts, documents, embeddings and chat history stay on the VM, inside your own Azure subscription.

Both baked models are released under the Apache 2.0 licence, which permits commercial use and redistribution. Models you pull later carry their own licences; check each one before you use it commercially.

No password, key or certificate is baked into the image. AnythingLLM normally starts in single user mode with no password. This image switches on multi user mode and first boot creates an admin account with a password generated uniquely for each VM, together with the app's own signing secrets and a TLS certificate. The web app does not start until first boot has done this, and first boot proves that an anonymous caller and a wrong password are refused before it marks the VM ready. Backed by 24/7 cloudimg support.

AnythingLLM is developed by Mintplex Labs and Ollama by Ollama, Inc.; both are licensed under the MIT licence. Granite is a model family from IBM and Nomic Embed is from Nomic AI. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by Mintplex Labs, Ollama, IBM or Nomic AI. The names are used here solely to identify the software and models this image contains.

What is included:

• AnythingLLM 1.18.0, built from the upstream v1.18.0 release on Node.js 22 LTS, with its server and document collector running as a dedicated anythingllm service account
• Ollama 0.40.4 from the official release archive (SHA-256 verified), running as a dedicated ollama service account on 127.0.0.1:11434
• The granite4:micro chat model and the nomic-embed-text embedding model, already pulled
• The built in LanceDB vector database, storing vectors on the VM's disk
• nginx terminating HTTPS on 443 with a per-VM certificate, redirecting 80 to HTTPS
• An nftables rule that lets only the VM itself reach AnythingLLM (3001), its collector (8888) and Ollama (11434)
• AnythingLLM telemetry turned off and Ollama's cloud features disabled
• Unattended security upgrades left on for the operating system

Prerequisites

• An Azure subscription with permission to create VMs
• A Standard_B4ms VM (4 vCPU, 16 GiB) or larger is recommended. The models run on the CPU, so more vCPUs give faster answers: Standard_B8ms (8 vCPU, 32 GiB) roughly doubles the speed and leaves room for a larger model. Smaller sizes are not recommended
• Inbound TCP 443 (and 80 for the redirect) open only to the networks you connect from, plus 22 for SSH administration
• An SSH key pair for administrative access

Step 1: Deploy from the Azure Portal

1. In the Azure Portal, search the Marketplace for Ollama + AnythingLLM on Ubuntu 24.04 LTS and select Create.
2. Choose your subscription, resource group and region.
3. Set the VM size to Standard_B4ms or larger.
4. Under Administrator account, select SSH public key and supply your own key.
5. Under Inbound port rules, allow HTTPS (443), HTTP (80) and SSH (22), then restrict all three to your own address ranges on the networking blade.
6. Review and create.

Step 2: Deploy from the Azure CLI

az vm image terms accept --urn cloudimg1647283583153:ollama-anythingllm-ubuntu-24-04:default:latest

az vm create \
  --resource-group my-resource-group \
  --name my-private-ai \
  --image cloudimg1647283583153:ollama-anythingllm-ubuntu-24-04:default:latest \
  --size Standard_B4ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --public-ip-sku Standard

# Open HTTPS (and the HTTP redirect) to YOUR network only.
MY_CIDR="203.0.113.0/24"   # replace with your own range
az network nsg rule create \
  --resource-group my-resource-group \
  --nsg-name my-private-aiNSG \
  --name allow-https-from-office --priority 1010 \
  --source-address-prefixes "${MY_CIDR}" \
  --destination-port-ranges 80 443 --protocol Tcp --access Allow

Step 3: Connect to your VM

ssh azureuser@<vm-ip>

Step 4: Confirm the services are running

First boot takes about a minute. It generates this VM's admin password, secrets and certificate, starts the services and proves the sign in gate before it finishes.

# Every unit reports "active". The firstboot and firewall units are oneshots that stay active (exited).
systemctl is-active ollama-anythingllm-firstboot ollama anythingllm anythingllm-collector nginx ollama-anythingllm-firewall

# Ollama listens on loopback only; nginx is the only public listener.
sudo ss -lntp | grep -E ':(80|443|11434)\b'

Expected output:

active
active
active
active
active
active
LISTEN 0      4096       127.0.0.1:11434      0.0.0.0:*    users:(("ollama",...))
LISTEN 0      511          0.0.0.0:80         0.0.0.0:*    users:(("nginx",...))
LISTEN 0      511          0.0.0.0:443        0.0.0.0:*    users:(("nginx",...))
LISTEN 0      511             [::]:80            [::]:*    users:(("nginx",...))
LISTEN 0      511             [::]:443           [::]:*    users:(("nginx",...))

Step 5: Read this VM's credentials

Everything in this file was generated on this VM at first boot.

sudo cat /root/ollama-anythingllm-credentials.txt

# Ollama + AnythingLLM - per-VM credentials generated on first boot (...).
# Keep this file private. Change the password in AnythingLLM: Settings > Users (or your profile).
ANYTHINGLLM_URL=https://<vm-ip>/
ANYTHINGLLM_USERNAME=admin
ANYTHINGLLM_PASSWORD=<ANYTHINGLLM_PASSWORD>

Step 6: Sign in to AnythingLLM

1. Open ANYTHINGLLM_URL from the credentials file in your browser. The certificate is self-signed and generated for this VM, so your browser will warn you the first time. Accept it for this address, or follow Step 12 to use your own domain and a trusted certificate.
2. Sign in as admin with ANYTHINGLLM_PASSWORD.

1. The first time you sign in, AnythingLLM shows a set of recovery codes for the admin account. Download them and keep them somewhere safe: together with the username they let you reset the password if you lose it. AnythingLLM shows them only once.

After signing in you land on the AnythingLLM home page. Workspaces are listed in the sidebar; the settings (the spanner icon at the bottom of the sidebar) hold the model, embedding, vector database, user and security configuration.

Step 7: Understand the access model

• Sign in. AnythingLLM runs in multi user mode. Every page and API route needs a signed in user, except the sign in page itself and a few public status endpoints. First boot created the only account, admin, with a password stored as a bcrypt hash in AnythingLLM's database; the plain text exists only in the root-only credentials file. Add users for your team under Settings > Users, with the default, manager or admin role.
• Network. nginx is the only public listener: it serves AnythingLLM on port 443 with a certificate generated for this VM, and port 80 only redirects to HTTPS. AnythingLLM's server (3001) and document collector (8888) cannot be told to bind to loopback, so an nftables table drops traffic to them, and to Ollama (11434), from anything but the VM itself. Ollama also binds to 127.0.0.1 only.
• Brute force. nginx limits sign in attempts to ten a minute per client.
• Privacy. AnythingLLM's anonymous telemetry is switched off (DISABLE_TELEMETRY="true") and Ollama runs with OLLAMA_NO_CLOUD=1. Chats and documents are processed by the models on this VM.

You can check all of it from the VM itself:

code() { curl -sk -o /dev/null -w '%{http_code}' "$@"; }
token() { curl -sk -H 'Content-Type: application/json' -d "{\"username\":\"admin\",\"password\":\"$1\"}" https://127.0.0.1/api/request-token; }
# Multi user mode is on, and anonymous callers are refused (401).
curl -sk https://127.0.0.1/api/system/multi-user-mode; echo
echo "anonymous /api/workspaces : $(code https://127.0.0.1/api/workspaces)"
# A wrong password is refused, this VM's password is accepted.
echo "wrong password            : $(token wrong-password-123 | python3 -c 'import sys,json;print("valid:",json.load(sys.stdin)["valid"])')"
echo "per-VM password           : $(token '<ANYTHINGLLM_PASSWORD>' | python3 -c 'import sys,json;print("valid:",json.load(sys.stdin)["valid"])')"

Expected output:

{"multiUserMode":true}
anonymous /api/workspaces : 401
wrong password            : valid: False
per-VM password           : valid: True

Step 8: The local models

AnythingLLM is configured to use the Ollama server on this VM for both chat and embeddings. You can see the models and how the app is wired from the shell (AnythingLLM rewrites its own .env when you change settings in the web app, so the order of the lines may differ):

ollama --version
ollama list
sudo grep -E '^(LLM_PROVIDER|OLLAMA_BASE_PATH|OLLAMA_MODEL_PREF|EMBEDDING_ENGINE|EMBEDDING_MODEL_PREF|VECTOR_DB|DISABLE_TELEMETRY)=' /opt/anythingllm/server/.env

ollama version is 0.40.4
NAME                       ID              SIZE      MODIFIED
nomic-embed-text:latest    0a109f422b47    274 MB    ...
granite4:micro             89962fcc7523    2.1 GB    ...
DISABLE_TELEMETRY="true"
LLM_PROVIDER="ollama"
OLLAMA_BASE_PATH="http://127.0.0.1:11434"
OLLAMA_MODEL_PREF="granite4:micro"
EMBEDDING_ENGINE="ollama"
EMBEDDING_MODEL_PREF="nomic-embed-text:latest"
VECTOR_DB="lancedb"

The same settings are visible in the web app under Settings > AI Providers > LLM, where the provider is Ollama and the model is granite4:micro.

Ollama's own API is available to anything running on the VM. For example, ask the chat model a question directly:

curl -s http://127.0.0.1:11434/api/chat -d '{"model":"granite4:micro","stream":false,
  "options":{"temperature":0,"seed":7,"num_predict":120},
  "messages":[{"role":"user","content":"In two sentences, what is retrieval augmented generation?"}]}' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["message"]["content"])'

The models run on the CPU. On a Standard_B4ms a short answer takes a few seconds to start and then streams at a readable pace; the first request after the model has been idle for an hour also loads it into memory, which takes a few seconds more.

Step 9: Chat with your documents

1. Select New Workspace in the sidebar and give it a name. A workspace is a separate set of documents and chats.
2. Open the workspace and select the upload icon next to its name. Upload files (PDF, Word, text, Markdown, CSV and more) or paste a link, then move them into the workspace and select Save and Embed. The document collector extracts the text and the nomic-embed-text model turns it into vectors stored in LanceDB on this VM.
3. Ask a question in the chat box. AnythingLLM retrieves the most relevant passages from your documents, the granite4:micro model answers from them, and the answer lists the documents it drew on under Citations.

Each workspace has a chat mode in its settings: Automatic (the default for new workspaces) and Chat blend the model's general knowledge with your documents, and Query answers only from your documents and says so when they do not contain the answer.

Agents. Type @agent at the start of a message to hand it to AnythingLLM's agent, which can use tools such as web scraping, document summaries and saving files. Because the models run on the CPU, this image sets PROVIDER_DISABLE_NATIVE_TOOL_CALLING="ollama" in /opt/anythingllm/server/.env, so ordinary messages are answered directly from your documents instead of going through the agent's tool-calling loop on every message, which is much slower on a CPU. Agents still work when you ask for them with @agent. On a large VM or a GPU VM you can remove that line and run sudo systemctl restart anythingllm to let Automatic mode call tools natively.

Step 10: Use the developer API

AnythingLLM has a REST API for building your own tools on top of the workspace. Create an API key under Settings > Tools > Developer API, or from the shell as below. The interactive API reference is at https://<vm-ip>/api/docs.

This example signs in as admin, creates an API key, creates a workspace, adds a document to it and asks a question about it, all on this VM:

B=https://127.0.0.1/api
TOKEN=$(curl -sk -H 'Content-Type: application/json' -d '{"username":"admin","password":"<ANYTHINGLLM_PASSWORD>"}' $B/request-token | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
KEY=$(curl -sk -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"name":"guide"}' $B/admin/generate-api-key | python3 -c 'import sys,json;print(json.load(sys.stdin)["apiKey"]["secret"])')
SLUG=$(curl -sk -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d '{"name":"API demo"}' $B/v1/workspace/new | python3 -c 'import sys,json;print(json.load(sys.stdin)["workspace"]["slug"])')
curl -sk -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d "{\"textContent\":\"The staff car park opens at 7am and closes at 9pm. Visitors park on level 2.\",\"addToWorkspaces\":\"$SLUG\",\"metadata\":{\"title\":\"car-park.txt\"}}" \
  $B/v1/document/raw-text | python3 -c 'import sys,json;d=json.load(sys.stdin);print("document added:",d["success"])'
curl -sk -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"message":"Where do visitors park?","mode":"query"}' $B/v1/workspace/$SLUG/chat \
  | python3 -c 'import sys,json;d=json.load(sys.stdin);print("answer:",d["textResponse"]);print("sources:",[s["title"] for s in d["sources"]])'

document added: True
answer: Visitors park on level 2.
sources: ['car-park.txt']

Developer API keys act with full access, so store them like passwords and delete the ones you no longer need under Settings > Tools > Developer API.

Step 11: Add or change models

Ollama can run any model from its library that fits in the VM's memory. Pull a model from the shell, then select it in Settings > AI Providers > LLM (the model list is read from Ollama), or per workspace in the workspace's Chat Settings:

# Example: a larger general purpose model (about 5 GB; needs a VM with 16 GiB or more).
ollama pull qwen3:8b
ollama list

• Check each model's licence before using it commercially: ollama show <model> --license.
• Models are stored under /var/lib/ollama/models. The OS disk has room for a few more small models; resize the OS disk in Azure, or attach a data disk and point OLLAMA_MODELS at it in a systemd drop-in, if you plan to keep several large models.
• To remove a model you no longer use: ollama rm <model>.
• If you change the embedding model, AnythingLLM has to re-embed your documents, because vectors from different models are not compatible.
• On a GPU VM size (NC or ND series) with the NVIDIA driver installed, for example through the Azure NVIDIA GPU Driver extension, Ollama uses the GPU automatically; its CUDA runtime is already included.

Step 12: Use your own domain and certificate

The self-signed certificate is fine for a trial. For regular use, point a DNS name at the VM's public IP and install a trusted certificate. nginx already serves the Let's Encrypt HTTP-01 challenge path on port 80.

sudo apt-get update && sudo apt-get install -y certbot
sudo certbot certonly --webroot -w /var/www/html -d ai.example.com
sudo sed -i \
  -e 's#/etc/ollama-anythingllm/tls/server.crt#/etc/letsencrypt/live/ai.example.com/fullchain.pem#' \
  -e 's#/etc/ollama-anythingllm/tls/server.key#/etc/letsencrypt/live/ai.example.com/privkey.pem#' \
  /etc/nginx/sites-available/ollama-anythingllm
sudo nginx -t && sudo systemctl reload nginx

Certbot renews the certificate automatically; add --deploy-hook "systemctl reload nginx" to the certonly command so nginx picks up each renewal.

Step 13: Day-2 operations

# Restart the app (the collector restarts with it if needed) or Ollama.
sudo systemctl restart anythingllm anythingllm-collector
sudo systemctl restart ollama

# Recent logs (add -f to follow them live).
sudo journalctl -u anythingllm -u anythingllm-collector -u ollama -n 20 --no-pager

# What is installed.
cat /usr/share/ollama-anythingllm/VERSIONS

• Data. AnythingLLM keeps its database (users, workspaces, chats, settings), uploaded documents and LanceDB vectors in /opt/anythingllm/server/storage, and its configuration in /opt/anythingllm/server/.env. Back up both, together with /var/lib/ollama/models if you pulled extra models, or take Azure disk snapshots of the VM.
• Users and passwords. Add, suspend and reset users under Settings > Users. Change your own password from your profile. If you lose the admin password, use the recovery codes from your first sign in.
• Operating system updates. Unattended security upgrades are on. Reboot when /var/run/reboot-required exists.
• Upgrading AnythingLLM or Ollama. Both are installed from pinned upstream releases. Take a snapshot first, then follow the upstream release notes; cloudimg support can help plan an upgrade.

Troubleshooting

• The browser shows a certificate warning. Expected with the per-VM self-signed certificate; see Step 12.
• The page does not load at all. Check that the VM's network security group allows 443 from your address, and that systemctl is-active anythingllm nginx reports active.
• Answers are slow. The models run on the CPU, so speed depends on the VM size. Use Standard_B4ms or larger, and remember that B series VMs are burstable: under sustained use, a D series size such as Standard_D4s_v5 keeps its full CPU speed.
• A chat says the model is not available. Check systemctl is-active ollama and ollama list; the workspace's model must be one that Ollama has pulled.
• Uploads fail. Check systemctl is-active anythingllm-collector and sudo journalctl -u anythingllm-collector -n 50. Web pages are scraped with a headless browser that is not included in the image; uploading files works without it.

Support

cloudimg provides 24/7 support for this image by email at support@cloudimg.co.uk, covering deployment, model selection, VM sizing, document chat, the developer API, TLS and upgrades.