whisper.cpp Server on Ubuntu 24.04 LTS User Guide

Overview

This guide covers deploying and using whisper.cpp Server on Microsoft Azure with cloudimg's pre configured Ubuntu 24.04 LTS virtual machine image.

whisper.cpp is a high performance implementation of OpenAI's Whisper automatic speech recognition model, written in plain C and C++. It runs entirely on the CPU, with no GPU, no Python runtime and no external service dependencies. This image ships the project's bundled HTTP server, so the machine becomes a private speech to text service the moment it boots: post an audio file and receive a transcript back.

Because everything runs locally, audio never leaves the virtual machine you control, which suits recordings that cannot be sent to a third party transcription service.

What's included:

• whisper.cpp 1.9.1, built from the pinned upstream source tag
• The bundled whisper.cpp HTTP server, listening on the loopback interface only
• The ggml-base.en speech recognition model, preinstalled and verified by checksum
• nginx 1.24 as an authenticating reverse proxy on port 80
• ffmpeg, for automatic conversion of common audio and video formats
• A short test clip, so you can prove the service works before using your own audio
• 24/7 cloudimg support

Key facts:

Item	Value
Platform	Microsoft Azure
Operating system	Ubuntu 24.04 LTS
Default SSH user	azureuser
Web interface and API	Port 80, HTTP Basic authentication
Inference engine	127.0.0.1:8080, loopback only, not reachable from the network
Model file	/var/lib/whisper-cpp-server/models/ggml-base.en.bin
Credentials file	/root/whisper-cpp-server-credentials.txt
Licence notices	/opt/whisper/

About the bundled model

whisper.cpp is a program, not a model. It needs a separate model file to do anything, and that file carries its own licence. This image ships ggml-base.en, and its provenance was traced to its root rather than assumed from a repository badge:

This image includes the ggml-base.en speech recognition model, a format conversion of OpenAI's original Whisper base.en checkpoint into the GGML format used by whisper.cpp. It is not a fine tune and not a distillation of any third party model. Both OpenAI's Whisper weights and whisper.cpp itself are distributed under the MIT License, and copies of both licences ship in the image at /opt/whisper/, together with a written provenance record.

The model file is pinned by name and by SHA 256 checksum at build time, so the image cannot silently ship a different model.

Prerequisites

Before deploying, ensure you have:

1. An active Microsoft Azure subscription
2. An active Azure Marketplace subscription for this offer
3. An SSH key pair for connecting to the virtual machine
4. Permission to create virtual machines and network security groups in your target region

Recommended VM size: Standard_B2s (2 vCPU, 4 GB RAM). See Performance for what that delivers and when to choose something larger.

Network security group configuration

Protocol	Port	Description
TCP	22	SSH access
TCP	80	whisper.cpp web interface and transcription API, authenticated

Port 8080 is deliberately not in this table. The inference engine binds to the loopback interface only and is never reachable from the network. See Security model.

Step 1: Deploy the virtual machine

Option A: Azure Portal

1. In the Azure Portal, go to Marketplace and search for whisper.cpp Server on Ubuntu 24.04 LTS.
2. Select the offer and click Create.
3. Choose your subscription, resource group and region.
4. Set the VM size to Standard_B2s or larger.
5. Choose SSH public key as the authentication type and set the username to azureuser.
6. Under Inbound port rules, allow SSH (22) and HTTP (80).
7. Click Review + create, then Create.

Option B: Azure CLI

az vm create \
  --resource-group my-resource-group \
  --name my-whisper-vm \
  --image cloudimg:whisper-cpp-server-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard \
  --location eastus

Then open the web port:

az vm open-port --resource-group my-resource-group --name my-whisper-vm --port 80

Step 2: Connect via SSH

Find the public IP address of your virtual machine:

az vm show --resource-group my-resource-group --name my-whisper-vm \
  --show-details --query publicIps --output tsv

Connect:

ssh azureuser@<vm-ip>

Step 3: Retrieve your credentials

On the first boot of every virtual machine, this image generates a password that is unique to that machine. Nothing is shared between deployments and no password is baked into the image. Read it with:

sudo cat /root/whisper-cpp-server-credentials.txt

The file is readable by root only, and contains your username, your password and the URL of your service:

WHISPER_USER=whisper
WHISPER_PASSWORD=...
WHISPER_URL=http://<vm-ip>/
WHISPER_API_URL=http://<vm-ip>/inference

Confirm the file exists with the correct ownership and permissions. The command below shows that the file is present and root only without printing the password itself:

sudo ls -l /root/whisper-cpp-server-credentials.txt

Step 4: Confirm the service is running

sudo systemctl status whisper-server.service --no-pager

sudo systemctl status nginx.service --no-pager

Both units should report active (running).

Check that the inference engine is answering on the loopback interface:

curl -s -o /dev/null -w 'health: HTTP %{http_code}\n' http://127.0.0.1:8080/health

Expected output:

health: HTTP 200

Step 5: Transcribe the bundled test clip

The image ships a short test clip so you can prove the service works end to end before introducing your own audio. It was synthesised on the build machine with eSpeak NG, so no third party audio is bundled with this image.

Transcribe it, substituting the password from Step 3:

sudo curl -s -u whisper:<your-password> \
  -F file=@/usr/local/share/whisper-cpp-server/samples/cloudimg-test.wav \
  -F response_format=json \
  http://127.0.0.1/inference

Expected output:

{"text":" A quick brown fox jumps over the lazy dog.\n"}

If you get a transcript back containing those words, the model loaded correctly and the whole path works. An empty text value with an HTTP 200 would mean the model failed to load, so check the transcript content rather than the status code.

Step 6: Use the browser interface

Open http://<vm-ip>/ in a browser. You will be prompted for the username and password from Step 3.

Once authenticated, the interface documents the API and provides an upload form.

Choose an audio file, pick a response format and click Submit.

The transcript is returned directly in the browser.

Note that the interface page documents a /load endpoint. That endpoint is deliberately blocked on this image and returns HTTP 403. See Security model for why.

Step 7: Transcribe your own audio

Copy a file to the virtual machine:

scp <your-audio-file> azureuser@<vm-ip>:/tmp/

Then transcribe it. ffmpeg is installed and the server converts input automatically, so MP3, M4A, FLAC, OGG, WAV and the audio track of most video files are all accepted:

sudo curl -s -u whisper:<your-password> \
  -F file=@/tmp/<your-audio-file> \
  -F response_format=text \
  http://127.0.0.1/inference

Response formats

Pass response_format to choose the output shape:

Value	Output
text	Plain transcript text
json	{"text": "..."}
verbose_json	Transcript plus per segment and per word timings and probabilities
srt	SubRip subtitles
vtt	WebVTT subtitles

Generate subtitles for a recording:

sudo curl -s -u whisper:<your-password> \
  -F file=@/usr/local/share/whisper-cpp-server/samples/cloudimg-test.wav \
  -F response_format=srt \
  http://127.0.0.1/inference

Expected output:

1
00:00:00,000 --> 00:00:03,600
 A quick brown fox jumps over the lazy dog.

Calling the API from another machine

Replace the loopback address with your virtual machine's public IP and supply the same credentials:

curl -s -u whisper:<your-password> \
  -F file=@recording.mp3 -F response_format=text \
  http://<vm-ip>/inference

Performance

whisper.cpp runs on the CPU, so transcription speed scales with the number of vCPUs. These figures were measured on this image running on Standard_B2s (2 vCPU) with the bundled ggml-base.en model:

Input	Wall clock	Ratio
3.5 second clip	approximately 2.9 seconds	dominated by fixed startup cost
90.7 seconds of speech	approximately 17.4 seconds	about 5 times faster than real time

In practice, on Standard_B2s, expect roughly 5 times faster than real time for sustained audio, so a 10 minute recording takes around 2 minutes. Short clips are dominated by a fixed per request cost of a couple of seconds.

Guidance for choosing a size:

• Standard_B2s is a good fit for occasional transcription, evaluation and low volume API use. Note that B series machines are burstable, so sustained batch work can exhaust CPU credits and slow down.
• For sustained or batch transcription, choose a compute optimised size such as Standard_F4s_v2 or Standard_F8s_v2. Throughput scales roughly with vCPU count.
• Memory is not the constraint with this model. The ggml-base.en model uses a few hundred megabytes, and 4 GB of RAM is sufficient.

Requests are processed one at a time. If you need concurrency, place a queue in front of the API or run several virtual machines behind a load balancer.

Using a different model

ggml-base.en balances accuracy and speed, and is English only. Larger models are more accurate and slower; multilingual models drop the .en suffix. To use a different one, download it and point the service at it:

sudo curl -fL -o /var/lib/whisper-cpp-server/models/ggml-small.en.bin \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-small.en.bin
sudo chown whisper:whisper /var/lib/whisper-cpp-server/models/ggml-small.en.bin

Then edit the ExecStart line in /etc/systemd/system/whisper-server.service to reference the new file, and reload:

sudo systemctl daemon-reload
sudo systemctl restart whisper-server.service

Check the licence of any model you add. The models in the repository above are conversions of OpenAI's original Whisper checkpoints and are MIT licensed, but models elsewhere, including community fine tunes, may carry more restrictive terms that prohibit commercial use.

Security model

whisper.cpp's bundled server has no authentication of any kind, and the upstream project's own documentation warns against exposing it. It accepts user file uploads and shells out to ffmpeg for conversion, and its /load endpoint allows a caller to load an arbitrary server side file path as a model. Left open on a public IP address, that is an abuse and cost surface, and transcription endpoints handle audio that is often sensitive.

This image does not expose it. The protections below are applied in the image and verified on a freshly launched virtual machine before the image is published:

1. The inference engine binds to the loopback interface only. It listens on 127.0.0.1:8080 and is not reachable from the network, whatever your network security group allows.
2. Every path requires authentication. nginx fronts the engine on port 80 and demands HTTP Basic credentials. There is no anonymous route to any endpoint.
3. The /load endpoint is blocked outright. It returns HTTP 403 even with valid credentials, so the remote model swap primitive is unreachable rather than merely protected.
4. No default credential exists. The password is generated on each machine's first boot. Until that happens the credential store is empty, which fails closed: every request is refused.
5. The service runs unprivileged as the whisper system user, under systemd hardening including NoNewPrivileges, ProtectSystem=strict and PrivateTmp.
6. Audio and transcripts are not persisted. Uploads are handled per request and nothing but the model file remains on disk afterwards.

You can confirm points 1 to 3 yourself. The engine must refuse a connection on the machine's own routable address:

curl -s -o /dev/null -m 5 -w '%{http_code}\n' http://$(hostname -I | awk '{print $1}'):8080/health || echo "connection refused, as expected"

A status of 000 means curl could not connect at all, which is the desired result: nothing is listening on that address.

An anonymous request must be refused:

curl -s -o /dev/null -w 'anonymous: HTTP %{http_code}\n' http://127.0.0.1/inference

A wrong password must be refused:

curl -s -o /dev/null -w 'wrong password: HTTP %{http_code}\n' -u whisper:definitely-not-the-password http://127.0.0.1/

And the model swap endpoint must be blocked even with the correct credentials:

sudo curl -s -o /dev/null -w 'load endpoint: HTTP %{http_code}\n' -u whisper:<your-password> http://127.0.0.1/load

Expected output from those four commands:

000
connection refused, as expected
anonymous: HTTP 401
wrong password: HTTP 401
load endpoint: HTTP 403

Recommended additional hardening

• Restrict port 80 by source address in your network security group so only your own networks can reach the service.
• Add TLS. Basic authentication sends credentials in a reversible encoding, so on any untrusted network you should terminate TLS in front of it. The nginx configuration at /etc/nginx/sites-available/cloudimg-whisper-cpp-server is ready for a certificate.
• Change the password if you wish, using sudo htpasswd /etc/nginx/whisper-cpp-server.htpasswd whisper, and record the new value safely. nginx picks the change up immediately with no reload.
• Keep the engine on loopback. Do not change --host 127.0.0.1 in the service unit.

Verifying the bundled model

The model shipped with this image can be checked against the checksum recorded at build time:

sha256sum /var/lib/whisper-cpp-server/models/ggml-base.en.bin

Expected output:

a03779c86df3323075f5e796cb2ce5029f00ec8869eee3fdfb897afe36c6d002  /var/lib/whisper-cpp-server/models/ggml-base.en.bin

The licence notices and the written provenance record are here:

ls -l /opt/whisper/

cat /opt/whisper/MODEL-PROVENANCE.txt

Server components

Component	Version	Install path
whisper.cpp server	1.9.1	/usr/local/bin/whisper-server
Speech recognition model	ggml-base.en	/var/lib/whisper-cpp-server/models/ggml-base.en.bin
nginx	1.24	/usr/sbin/nginx
ffmpeg	6.1.1	/usr/bin/ffmpeg
eSpeak NG	1.51	/usr/bin/espeak-ng

Filesystem layout

Mount point	Description
/	Root filesystem
/boot	Operating system kernel files
/boot/efi	UEFI boot partition, Generation 2 Hyper V
/mnt	Azure temporary resource disk

Key directories:

Path	Purpose
/var/lib/whisper-cpp-server/models/	Speech recognition model files
/usr/local/share/whisper-cpp-server/samples/	The bundled test clip
/opt/whisper/	MIT licence notices and the model provenance record
/etc/nginx/whisper-cpp-server.htpasswd	Credential store for the reverse proxy
/root/whisper-cpp-server-credentials.txt	Per machine credentials, root only

Managing the service

sudo systemctl status whisper-server.service --no-pager

Restart the inference engine:

sudo systemctl restart whisper-server.service

View the engine log:

sudo journalctl -u whisper-server.service --no-pager -n 20

Scripts and log files

Path	Purpose
/usr/local/sbin/whisper-cpp-server-firstboot.sh	Generates the per machine credential on first boot
/var/log/cloudimg-firstboot.log	First boot log
/var/lib/cloudimg/whisper-cpp-server-firstboot.done	First boot completion marker

On startup

On the first boot only, whisper-cpp-server-firstboot.service runs before the application and nginx start. It generates the password unique to this machine, writes it into the reverse proxy credential store and into the root only credentials file, resolves the machine's public address for the login banner, and then writes its completion marker. Subsequent boots skip it, so your password is never regenerated behind your back.

Read the first boot log with:

sudo cat /var/log/cloudimg-firstboot.log

Troubleshooting

Every request returns HTTP 401

Confirm you are using the password from this machine. Passwords are per machine and are not shared between deployments:

sudo grep -c WHISPER_PASSWORD /root/whisper-cpp-server-credentials.txt

If the credentials file is missing, first boot did not complete. Check the log:

sudo journalctl -u whisper-cpp-server-firstboot.service --no-pager -n 20

A request returns HTTP 200 but the transcript is empty

An empty transcript with a healthy status usually means the model file is missing or corrupt. Verify it against its checksum as shown in Verifying the bundled model. Genuinely silent audio also returns an empty transcript, which is correct behaviour, so confirm your input actually contains speech.

The /load endpoint returns HTTP 403

This is intentional. The endpoint is blocked on this image because it allows an arbitrary server side file to be loaded as a model. To change the model, edit the service unit as described in Using a different model.

Transcription is slower than expected

Confirm the machine is not out of CPU credits if you are on a B series size, and consider a compute optimised size. See Performance.

The browser will not prompt for credentials again

Browsers cache Basic authentication credentials for the session. Open a private window to be prompted again.

Security recommendations

• Restrict port 80 to trusted source addresses in your network security group.
• Terminate TLS in front of the service before using it across any untrusted network.
• Keep the credentials file readable by root only and do not copy it into shared storage.
• Leave the inference engine bound to the loopback interface.
• Apply operating system updates regularly. Unattended security upgrades are enabled in this image.

Support

For assistance, contact cloudimg support:

• Email: support@cloudimg.co.uk
• Response time: 24/7 with a guaranteed 24 hour response SLA
• Website: https://www.cloudimg.co.uk