Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and use of Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images.

Distributed tracing needs four separate pieces, and Tempo is only one of them. Tempo has no user interface of its own and no ingest path, so a virtual machine running nothing but Tempo cannot be used. This image is the four pieces, pre-wired and proven end to end:

• Grafana Alloy 1.19.2 — the collector. It accepts OpenTelemetry (OTLP) spans over gRPC and over HTTP and forwards them to Tempo.
• Grafana Tempo 3.0.3 — the trace store, running as a single binary (-target=all) against a local filesystem backend.
• Grafana 13.2.2 — the query interface, with the Tempo datasource already provisioned, so trace search and the waterfall view work the first time you sign in.
• nginx 1.30.5 — a TLS front door, and the only service besides SSH reachable from outside the virtual machine.

What this is, and what it is not. Single-binary Tempo on local filesystem storage is a complete, genuinely useful tracing backend for one virtual machine. It is not a scale-out cluster and it is not object-storage backed: capacity is the size of this machine's disk, there is no replication, and there is no horizontal query fan-out. Scaling out explains exactly what to change when you outgrow it.

The stack is secure by default. Tempo and Alloy have no authentication of their own, so they are bound to loopback and are never reachable from outside the machine. The only off-box TCP ports are 22 (SSH) and 443 (nginx over TLS); port 80 exists solely to redirect to HTTPS and serves no content. On every fresh virtual machine, tempo-grafana-stack-firstboot.service generates a unique Grafana administrator password, a unique Grafana secret_key, a unique TLS key pair and a unique OTLP ingest credential before Grafana or nginx serves a single request. No credential is baked into the image — the image ships with no Grafana database at all, so even the administrator account's password salt is unique to your machine.

What is included:

• Grafana Tempo 3.0.3 (AGPL-3.0), single binary, local filesystem backend, loopback-bound on 127.0.0.1:3200

• Grafana Alloy 1.19.2 (Apache-2.0) with OTLP gRPC (127.0.0.1:4317) and OTLP HTTP (127.0.0.1:4318) receivers

• Grafana 13.2.2 Community Edition (AGPL-3.0) on 127.0.0.1:3000 with the Tempo datasource pre-provisioned

• nginx 1.30.5 (BSD-2-Clause) terminating TLS on 443, with a credentialed OTLP ingest path at /otlp/

• Per-virtual-machine TLS certificate, Grafana administrator password, Grafana secret_key and OTLP credential, all generated at first boot

• A demo-trace generator, an end-to-end self-test and a health check, all shipped on the machine

Prerequisites

• An Azure subscription with permission to create virtual machines
• An SSH key pair
• A virtual machine size of Standard_B2ms or larger (2 vCPU, 8 GiB RAM)
• Inbound TCP 22, 80 and 443 allowed in the network security group

Step 1: Deploy from the Azure Portal

1. In the Azure Portal, search the Marketplace for Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 LTS by cloudimg.
2. Select Create.
3. Choose your subscription, resource group and region.
4. Set the size to Standard_B2ms or larger.
5. Set the authentication type to SSH public key and provide your key. The administrator username is azureuser.
6. On the Networking tab, allow inbound 22, 80 and 443.
7. Select Review + create, then Create.

Step 2: Deploy from the Azure CLI

# Accept the Marketplace terms once per subscription, before the first deployment.
az vm image terms accept --publisher cloudimg --offer tempo-grafana-stack-ubuntu-24-04 --plan default

az group create --name tempo-stack-rg --location eastus

az vm create \
  --resource-group tempo-stack-rg \
  --name tempo-stack-vm \
  --image cloudimg:tempo-grafana-stack-ubuntu-24-04:default:latest \
  --size Standard_B2ms \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

az vm open-port --resource-group tempo-stack-rg --name tempo-stack-vm --port 22 --priority 1001
az vm open-port --resource-group tempo-stack-rg --name tempo-stack-vm --port 80 --priority 1002
az vm open-port --resource-group tempo-stack-rg --name tempo-stack-vm --port 443 --priority 1003

Step 3: Connect via SSH

ssh azureuser@<vm-public-ip>

First boot takes about 40 seconds to generate this machine's secrets. Wait for the sentinel before going further:

sudo test -f /var/lib/cloudimg/tempo-grafana-stack-firstboot.done && echo "first boot complete"

Step 4: Verify the Services

All four components should be active, and the ports reachable from outside should be exactly 22, 80 and 443:

for u in tempo alloy grafana-server nginx tempo-grafana-stack-firstboot; do
  printf '%-34s %s\n' "$u" "$(systemctl is-active $u)"
done

sudo /usr/local/sbin/tempo-grafana-stack-health.sh

Tempo (3200, 4327), Alloy (4317, 4318, 12345) and Grafana (3000) are all bound to 127.0.0.1. That is deliberate: Tempo and Alloy have no authentication mechanism at all, so exposing them would publish an unauthenticated write and read path to your trace data.

Step 5: Retrieve This Machine's Credentials

Every secret is generated on your own virtual machine at first boot and written to a root-only file:

sudo cat /root/tempo-grafana-stack-credentials.txt

The file records the Grafana URL and administrator password, the OTLP ingest endpoint with its own user and password, and the SSH-tunnel command for OTLP gRPC.

grafana.address_source tells you how the machine worked out its own public address. On Azure, the instance metadata service returns an empty response for a Standard-SKU public IP, so the appliance falls back to the load-balancer metadata view, which does report it. If that also returns nothing (a machine with no public IP), the file records the private address and says so.

Step 6: Prove the Whole Path End to End

The image ships a self-test that exercises the real product path rather than checking that processes are running. It sends a span into Alloy over OTLP gRPC and again over OTLP HTTP, confirms Tempo stored each one, and reads each back by trace ID through Grafana's provisioned Tempo datasource — the same path the browser uses:

sudo /usr/local/sbin/tempo-grafana-stack-selftest.sh

The self-test begins with a control: it queries a trace ID that was never sent and requires it to return nothing. Without that, a query path which answered for any input would look like a working round trip.

Step 7: Generate Some Demo Traces

A new appliance has nothing to look at. This generator sends a realistic seven-span checkout trace across five services, so the waterfall and the service graph have something meaningful in them:

sudo python3 /usr/local/share/cloudimg/tempo-grafana-stack/emit-demo-trace.py 15

It posts OTLP/JSON to Alloy's HTTP receiver exactly as an instrumented application would, so it exercises the real ingest path rather than writing to Tempo directly.

Step 8: First Login

Open https://<vm-public-ip>/ in a browser and sign in as admin with the grafana.password value from Step 5.

The certificate is self-signed and generated for your machine, so the browser warns on the first visit. Step 14 replaces it with your own.

Grafana keeps its brute-force protection enabled: five consecutive failed sign-ins block the account for a few minutes, and during that window even the correct password is refused. If you are locked out, wait rather than resetting anything.

Step 9: Search for Traces

Go to Explore, choose the Tempo datasource — it is already selected, because it is the default — pick the TraceQL query type, and run:

{ resource.service.name = "frontend" }

A span is retrievable by trace ID within about two seconds, but becomes searchable only after Tempo's live store cuts and completes a block. With this image's settings that takes roughly a minute. If a trace you just sent is not in the search results yet, wait and re-run — nothing is wrong.

You can confirm the same query outside the browser, straight through Grafana's datasource proxy:

NOW=$(date +%s)
curl -sG -u "admin:<GRAFANA_PASSWORD>" \
  "http://127.0.0.1:3000/api/datasources/proxy/uid/cloudimg-tempo/api/search" \
  --data-urlencode 'q={ resource.service.name = "frontend" }' \
  --data-urlencode "start=$((NOW-3600))" --data-urlencode "end=$((NOW+120))" \
  --data-urlencode 'limit=5' | head -c 400

Step 10: Render a Trace

Select any trace from the search results. Grafana draws the full waterfall: every span, its parent, its service and its duration.

This is the product. The trace above entered through Alloy's OTLP receiver, was stored by Tempo, and was fetched back by Grafana through the provisioned datasource — four components, one path.

To fetch the same trace as JSON:

TID=$(sudo python3 /usr/local/share/cloudimg/tempo-grafana-stack/emit-demo-trace.py 1 | awk '/DEMO_TRACE/{print $2}')
sleep 5
curl -s "http://127.0.0.1:3200/api/traces/${TID}" | head -c 300

Step 11: The Pre-Provisioned Tempo Datasource

Go to Connections → Data sources → Tempo. It is already configured against Tempo's loopback API, and it is the default datasource.

The datasource is provisioned from a file, not from the database:

sudo cat /etc/grafana/provisioning/datasources/cloudimg-tempo.yaml

Because it is file-provisioned, Grafana re-applies it on every start, so it cannot drift and it survives a database reset.

Streaming search is deliberately turned off. Grafana's streaming TraceQL results travel over a WebSocket that Grafana rejects when the browser's origin does not match the root_url recorded at first boot — which happens whenever you reach the machine by a different hostname, through a reverse proxy, or over an SSH tunnel. The failure is silent: the search panel simply shows nothing. The non-streaming path has no such dependency.

Step 12: Send Traces From Your Own Applications

There are two supported paths, and they differ because Alloy has no authentication of its own.

OTLP/HTTP over TLS — use this from anywhere. nginx terminates TLS and requires HTTP basic authentication, using the otlp.user and otlp.password values from Step 5:

curl -ks -o /dev/null -w 'ingest: HTTP %{http_code}\n' -X POST \
  -u "<OTLP_USER>:<OTLP_PASSWORD>" \
  -H 'Content-Type: application/json' \
  --data '{"resourceSpans":[]}' \
  https://127.0.0.1/otlp/v1/traces

For an OpenTelemetry SDK, set:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://<vm-public-ip>/otlp/v1/traces
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic <base64 of user:password>

An anonymous or wrong-password push is refused:

curl -ks -o /dev/null -w 'anonymous push: HTTP %{http_code}\n' -X POST \
  -H 'Content-Type: application/json' --data '{"resourceSpans":[]}' \
  https://127.0.0.1/otlp/v1/traces

OTLP/gRPC — over an SSH tunnel. Alloy's gRPC receiver is unauthenticated, so it stays on loopback. Run this on your own machine, then point your exporter at 127.0.0.1:4317 locally:

ssh -L 4317:127.0.0.1:4317 azureuser@<vm-public-ip>

If you need unauthenticated gRPC ingest from a private network, restrict it with a network security group rule scoped to that subnet before changing the bind address — never expose it to the internet.

Step 13: Scaling Beyond One Virtual Machine

This image runs Tempo as a single process against local disk. That is the right shape for one machine and the wrong shape for a fleet. The honest limits:

• Capacity is this machine's disk. There is no object storage behind it, so retention is bounded by free space.
• There is no replication. If the disk is lost, the traces are lost.
• There is no query fan-out. One process serves ingest, compaction and queries.

To move to object storage, change the backend in /etc/tempo/tempo.yaml from local to azure (or s3) and restart Tempo. To move beyond one machine, run Tempo's components separately with -target=distributor, -target=ingester and so on, which also introduces the Kafka-backed ingest path that Tempo 3 uses in its microservices mode.

grep -A6 '^storage:' /etc/tempo/tempo.yaml

Note that Tempo 3 keeps state in several directories under /var/lib/tempo — the write-ahead log and blocks, plus the live store and the backend scheduler — and this image pins all of them under that one root:

sudo ls -1 /var/lib/tempo

Step 14: Install Your Own TLS Certificate

The shipped certificate is self-signed and unique to this machine. Replace it with your own:

sudo ls -l /etc/ssl/cloudimg/

Copy your certificate to /etc/ssl/cloudimg/tempo-grafana-stack.crt and your key to /etc/ssl/cloudimg/tempo-grafana-stack.key (mode 0600, owned by root), then check and reload:

sudo nginx -t

If you also change the hostname customers use, update root_url in /etc/grafana/grafana.ini to match, or Grafana Live features will be refused for that origin.

Step 15: Managing the Services

systemctl --no-pager --lines=0 status tempo alloy grafana-server nginx

Logs for each component. These are filtered to the startup lines rather than dumping the raw journal, because Tempo is chatty at idle:

sudo journalctl -u tempo --no-pager | grep -E 'Starting Tempo|server listening' | tail -3

sudo journalctl -u alloy --no-pager | grep -E 'server now listening|finished node evaluation' | tail -3

no jobs found in Tempo's log is expected and harmless. Tempo 3's backend worker polls its scheduler for compaction work, and on a quiet appliance there is none, so it logs error calling scheduler ... no jobs found and backs off. It is not a fault; the worker retries on its own.

Alloy's own diagnostics interface is on loopback at 127.0.0.1:12345. Reach it over an SSH tunnel in the same way as the gRPC receiver.

Step 16: Troubleshooting

Traces do not appear in search. Confirm they reached Tempo by trace ID first — search needs a completed block, which takes about a minute:

curl -s "http://127.0.0.1:3200/api/search/tag/service.name/values" | head -c 300

The ingest endpoint returns 401. The credential is the otlp.user and otlp.password pair from Step 5, not the Grafana login.

The ingest endpoint returns 400. The body is not valid OTLP JSON. {"resourceSpans":[]} is the smallest valid payload.

Grafana refuses the correct password. Five consecutive failures block the account for a few minutes. Wait for the window to expire.

Nothing is listening on 443. nginx and Grafana are both gated on the first-boot marker and will not start before this machine's secrets exist:

sudo test -f /var/lib/cloudimg/tempo-grafana-stack-ready && echo "bootstrap marker present"

sudo journalctl -u tempo-grafana-stack-firstboot --no-pager | tail -20

Step 17: Security Notes

• Tempo, Alloy and Grafana all bind to loopback. Only nginx (443), its redirect (80) and SSH (22) are reachable from outside.
• The image contains no Grafana database, so no administrator account, password hash or salt is shared between machines. Your machine mints its own at first boot.
• The TLS key pair, the Grafana secret_key and the OTLP credential are all generated on your machine and never baked into the image.
• The default admin/admin login, a blank password, common guesses and the build-time password are all refused.
• Credentials live in /root/tempo-grafana-stack-credentials.txt, mode 0600, readable only by root.
• Unattended security upgrades are enabled. The four components are pinned to their minor lines, so security patches flow while a major version change does not happen unattended.

Step 18: Support and Licensing

Grafana Tempo and Grafana Community Edition are licensed under AGPL-3.0; Grafana Alloy under Apache-2.0; nginx under the 2-clause BSD licence. This image ships the AGPL-licensed open-source Grafana build — never Grafana Enterprise.

for p in tempo grafana alloy nginx; do printf '%-8s %s\n' "$p" "$(dpkg-query -W -f='${Version}' $p)"; done

cloudimg provides 24/7 support for the image itself: deployment, first boot, the pre-wired configuration and the Azure integration.

Deploy on Azure

Find Grafana Tempo 3 Tracing Stack on Ubuntu 24.04 LTS by cloudimg in the Azure Marketplace.

Need Help?

Contact cloudimg support through the Azure Marketplace listing.