VictoriaTraces on Ubuntu 24.04 on Azure User Guide

Overview

VictoriaTraces is a distributed tracing database from the VictoriaMetrics project. It receives OpenTelemetry spans over OTLP, stores them compactly on disk with no external database behind it, and serves them back through Jaeger compatible query APIs and a built in web UI. Where a traditional tracing stack needs a collector, a storage backend such as Cassandra or Elasticsearch, and a separate query service, VictoriaTraces is a single static Go binary that does the storage and the querying itself.

The cloudimg image installs the pinned VictoriaTraces 0.9.4 binary running under systemd, then locks it down for a marketplace appliance. VictoriaTraces has no built in authentication of any kind, which matters more for a tracing backend than for most services: an open ingest endpoint lets anyone write or poison your trace store, and an open query endpoint exposes every span in it, and spans routinely carry hostnames, URL paths, user identifiers, SQL statements and internal service topology. So the binary is bound to loopback 127.0.0.1:10428 where it is never reachable from outside the VM, and an nginx reverse proxy on port 80 puts a per-VM HTTP Basic Auth gate (user admin, unique password generated on first boot) in front of both the ingest path and the query paths. The only unauthenticated route is a static /healthz used by load balancer probes. All stored spans live on a dedicated Azure data disk mounted at /var/lib/victoria-traces. Backed by 24/7 cloudimg support.

What is included:

• VictoriaTraces 0.9.4 installed as a single Go binary running as the victoriatraces systemd service under a dedicated non-login account
• OTLP over HTTP span ingestion at /insert/opentelemetry/v1/traces, accepting protobuf or JSON
• Jaeger compatible query APIs under /select/jaeger/api/ and the built in web UI at /select/vmui/
• Per-VM HTTP Basic Auth (user admin) protecting the web UI, the ingest endpoint and every query API, with a unique password generated on first boot and no default login
• An unauthenticated /healthz endpoint for Azure Load Balancer health probes
• A dedicated Azure data disk mounted at /var/lib/victoria-traces holding all span data, with a 7 day default retention and an 80 percent disk usage cap you can tune
• A shipped self test at /usr/local/sbin/victoriatraces-selftest.sh that proves the auth gate and a full ingest to query round trip
• A fully patched Ubuntu 24.04 LTS base with unattended security upgrades enabled

Project maturity

VictoriaTraces is a young component of the VictoriaMetrics family, and you should factor that in before you make it the system of record for production tracing.

The project's first release was in July 2025 and it is under active development, but upstream has not yet declared a generally available release. Its own roadmap lists items that must be completed before GA, including "finalize the data structure and commit to backward compatibility", and the version numbering is still 0.x. Upstream does not describe the product as beta or experimental, but it is explicitly pre-GA by its own statement.

The practical consequence is that the on disk span format is not yet covered by a backward compatibility commitment, so a future major upgrade may require re-ingesting your spans rather than upgrading in place. Traces are usually short lived, high volume and reproducible, which makes this far less painful than it would be for a metrics or billing store, but it is a real consideration. Two individual features are separately marked experimental upstream and are deliberately not enabled in this image: the Jaeger service dependency graph API and the Grafana Tempo query APIs.

VictoriaTraces and VictoriaMetrics are trademarks of their respective owners. This image is provided by cloudimg and is not affiliated with, endorsed by, or sponsored by VictoriaMetrics.

Prerequisites

• An Azure subscription and permission to create a VM
• An SSH key pair (ssh-keygen -t ed25519)
• Inbound access to port 22 (SSH) and port 80 (the web UI, ingest and query APIs) on the VM, and optionally 443 if you add TLS
• A recommended size of Standard_B2s or larger

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for VictoriaTraces by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name victoriatraces \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name victoriatraces --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

Two units make up the appliance: victoriatraces.service is the tracing database itself, bound to loopback, and nginx.service is the authenticating reverse proxy that is the only thing listening on a public interface.

systemctl is-active victoriatraces.service nginx.service

Both should report active. You can confirm the listener layout and the dedicated data disk at the same time:

ss -tlnp | grep -E '127.0.0.1:10428 |:80 '
findmnt -no SOURCE,TARGET,FSTYPE,SIZE /var/lib/victoria-traces

Note that VictoriaTraces appears on 127.0.0.1:10428 only. It has no public listener, which is deliberate.

Step 5 - Retrieve your per-VM password

Every VM generates its own web password on first boot. Nothing is baked into the image, so there is no default login to change.

sudo cat /root/victoriatraces-credentials.txt

This file is 0600 root:root and contains the instance URL, the OTLP endpoint, the username admin and the generated password. It also records the pinned runtime configuration.

Step 6 - Understand the security gate

The unauthenticated health endpoint is open so Azure Load Balancer probes work. It is served by nginx itself and never touches VictoriaTraces, so it exposes nothing about your spans.

curl -s http://localhost/healthz

Everything else requires the per-VM credentials. This is worth verifying yourself, because it is the single most important property of the appliance. The following proves that both halves are closed to anonymous callers and that only the correct password opens them:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
echo "unauth ingest : $(curl -s -o /dev/null -w '%{http_code}' -X POST -H 'Content-Type: application/json' --data '{}' http://127.0.0.1/insert/opentelemetry/v1/traces)"
echo "unauth query  : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/select/jaeger/api/services)"
echo "unauth web UI : $(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1/select/vmui/)"
echo "wrong pw      : $(curl -s -o /dev/null -w '%{http_code}' -u admin:wrong-pw http://127.0.0.1/select/jaeger/api/services)"
echo "correct pw    : $(curl -s -o /dev/null -w '%{http_code}' -u admin:$PW http://127.0.0.1/select/vmui/)"

The first four should all return 401 and the last should return 200.

A browser that visits the VM without credentials sees the same gate before any part of the interface loads.

Step 7 - Send your first spans over OTLP

The ingest endpoint is /insert/opentelemetry/v1/traces and it speaks standard OTLP over HTTP. Set Content-Type: application/json to send OTLP/JSON, or application/x-protobuf to send the binary encoding that most collectors use by default.

This sends a single span describing a checkout operation in a service called guide-demo:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
TID=$(openssl rand -hex 16); SID=$(openssl rand -hex 8); NOW=$(date +%s)
curl -s -o /dev/null -w 'ingest -> HTTP %{http_code}\n' -u admin:$PW \
  -H 'Content-Type: application/json' --data-binary "{
  \"resourceSpans\": [{
    \"resource\": {\"attributes\": [{\"key\": \"service.name\", \"value\": {\"stringValue\": \"guide-demo\"}}]},
    \"scopeSpans\": [{\"spans\": [{
      \"traceId\": \"$TID\", \"spanId\": \"$SID\", \"name\": \"checkout\", \"kind\": 2,
      \"startTimeUnixNano\": \"${NOW}000000000\", \"endTimeUnixNano\": \"$((NOW+1))000000000\",
      \"attributes\": [{\"key\": \"http.response.status_code\", \"value\": {\"intValue\": \"200\"}}]
    }]}]
  }]
}" http://127.0.0.1/insert/opentelemetry/v1/traces
echo "$TID" | sudo tee /tmp/guide-trace-id >/dev/null

A 200 means the span was accepted.

Step 8 - Query your spans back

Spans are not queryable the instant they are accepted. VictoriaTraces buffers incoming spans and flushes them to the searchable store on an interval, which is typically ten to twenty seconds. Any automation you write should poll rather than assume the span is immediately visible.

List the services that have reported spans:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
for i in $(seq 1 30); do
  BODY=$(curl -s -u admin:$PW http://127.0.0.1/select/jaeger/api/services)
  case "$BODY" in *guide-demo*) echo "$BODY"; break ;; esac
  sleep 3
done

Now fetch the trace itself. The Jaeger API takes start and end as unix timestamps in microseconds, which is a common source of empty results:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
NOW=$(date +%s); S=$(( (NOW - 3600) * 1000000 )); E=$(( (NOW + 3600) * 1000000 ))
for i in $(seq 1 30); do
  BODY=$(curl -s -u admin:$PW "http://127.0.0.1/select/jaeger/api/traces?service=guide-demo&start=$S&end=$E&limit=5")
  case "$BODY" in *checkout*) break ;; esac
  sleep 3
done
echo "$BODY" | python3 -m json.tool | head -25

You can also list the operations a given service has recorded, which is how you discover what is worth filtering on:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
curl -s -u admin:$PW http://127.0.0.1/select/jaeger/api/services/guide-demo/operations

To fetch one specific trace by id, use the single trace endpoint:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
TID=$(sudo cat /tmp/guide-trace-id)
curl -s -u admin:$PW "http://127.0.0.1/select/jaeger/api/traces/$TID" | python3 -m json.tool | head -20

Step 9 - Explore traces in the web UI

Open http://<vm-public-ip>/select/vmui/ in a browser and sign in as admin with the password from Step 5. The Traces Explorer shows the spans arriving over time, with a breakdown by operation and service underneath.

The Trace query box filters spans. Because the service name field contains dots and a colon, quote the whole field name. To see only the spans from one service:

"resource_attr:service.name":"checkout-service"

Press Execute Query and the chart, the totals and the results below all narrow to that service.

To filter by operation as well, add the span name. This shows only the place-order spans from checkout-service:

"resource_attr:service.name":"checkout-service" AND name:"place-order"

The Table tab lays the matching spans out as rows, which is the quickest way to scan timings across many spans.

The JSON tab shows each span in full, including its duration and every attribute your instrumentation attached. This is where you confirm that the attributes you expect, such as HTTP status codes or database statements, are actually arriving.

Step 10 - Point an OpenTelemetry Collector at this VM

In production you will normally send spans through an OpenTelemetry Collector rather than with curl. Add this VM as an OTLP HTTP exporter in your collector configuration. The auth extension supplies the same Basic Auth credentials the API expects:

extensions:
  basicauth/victoriatraces:
    client_auth:
      username: admin
      password: <VICTORIATRACES_PASSWORD>

exporters:
  otlphttp/victoriatraces:
    traces_endpoint: http://<vm-public-ip>/insert/opentelemetry/v1/traces
    auth:
      authenticator: basicauth/victoriatraces
    encoding: proto

service:
  extensions: [basicauth/victoriatraces]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/victoriatraces]

To point an instrumented application straight at the VM with no collector in between, set the standard OpenTelemetry SDK environment variables. Every OpenTelemetry SDK understands these:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://<vm-public-ip>/insert/opentelemetry/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Basic <base64-of-admin:password>
OTEL_SERVICE_NAME=my-application

Generate the header value for the Authorization header with:

PW=$(sudo sed -n 's/^victoriatraces.password=//p' /root/victoriatraces-credentials.txt)
printf 'Authorization=Basic %s\n' "$(printf 'admin:%s' "$PW" | base64 -w0)"

Because everything travels over plain HTTP by default, put the VM behind a private network, a VPN or a TLS terminating load balancer before you send real production spans across a public network. See Step 13.

Step 11 - Tune retention and size the data disk

Span volume is much higher than log or metric volume, so retention is the setting you are most likely to change. The image ships two limits working together:

• -retentionPeriod=7d drops spans older than seven days
• -retention.maxDiskUsagePercent=80 caps the store at 80 percent of the data disk, dropping the oldest partitions first

The second is a safety net: whatever retention period you choose, a sudden burst of spans cannot fill the volume and wedge the service. Check the current values and how much of the disk is in use:

systemctl show -p ExecStart --value victoriatraces.service | grep -oE '\-retentionPeriod=[^ ]+|\-retention.maxDiskUsagePercent=[^ ]+'
df -h /var/lib/victoria-traces
du -sh /var/lib/victoria-traces 2>/dev/null || true

To change the retention period, edit the systemd unit and restart:

sudo systemctl edit --full victoriatraces.service   # change -retentionPeriod=7d to e.g. 30d
sudo systemctl daemon-reload
sudo systemctl restart victoriatraces.service

Sizing the disk. Storage need scales with spans per second and retention, not with the number of services. As a planning starting point, VictoriaTraces stores a typical span in well under a kilobyte after compression, so roughly: spans per second x 86400 x retention days x span size. At 100 spans per second and 7 days that is on the order of a few gigabytes, which the 32 GiB default disk holds comfortably. If you raise retention to 30 days or push several hundred spans per second, grow the data disk in the Azure portal and then extend the filesystem. Watch the actual growth rate over your first few days with du -sh and size from your own measured numbers rather than an estimate.

Step 12 - Verify the appliance at any time

The image ships a self test that re-proves everything this guide has covered: that the health probe is open, that unauthenticated ingest and unauthenticated queries are both rejected, that a wrong password is rejected, and that a real span sent over OTLP comes back through the Jaeger query API with its operation name and attributes intact.

sudo /usr/local/sbin/victoriatraces-selftest.sh

It prints a single OK- line on success and a SELFTEST-FAIL: line naming the failing check otherwise. It is a useful smoke test after you change the nginx configuration, rotate the password or resize the disk.

A note on the spans it leaves behind. The image itself ships with a completely empty trace store, but this self test runs once automatically during first boot to prove the appliance works before you rely on it, and each run ingests one real span. So the first time you open the UI you will see a service named cloudimg-selftest-<timestamp> alongside your own, and another appears each time you run the self test by hand. They are harmless, they are the proof the round trip worked, and they disappear on their own once they pass the retention period. If you would rather start from a completely clean store, stop the service, clear the storage directory and start it again, before you point any application at the VM:

sudo systemctl stop victoriatraces
sudo find /var/lib/victoria-traces -mindepth 1 -maxdepth 1 ! -name lost+found -exec rm -rf {} +
sudo systemctl start victoriatraces

Step 13 - Production hardening

• Add TLS. Spans and credentials travel in clear text over HTTP. Put the VM behind an Azure Application Gateway or Load Balancer that terminates TLS, or add a certificate to nginx directly and open port 443. The image does not ship a certificate because it cannot know your domain.
• Restrict the network. Tracing backends are internal infrastructure. Scope the network security group for port 80 to the subnets your applications and collectors run in rather than leaving it open to the internet.
• Rotate the password. Replace the generated password by writing a new bcrypt entry, then update the credentials note so the self test and your collectors stay in sync:

sudo htpasswd -B /etc/nginx/.victoriatraces.htpasswd admin
sudo systemctl reload nginx

• Back up what matters. Spans are usually reproducible and short lived, so most teams do not back up a trace store. If yours is an exception, snapshot the data disk rather than copying files out from under a running service.
• Keep the OS patched. Unattended security upgrades are enabled by default; leave them on.

Troubleshooting

A query returns an empty result immediately after ingesting. This is the flush interval, not an error. Spans become searchable roughly ten to twenty seconds after they are accepted. Poll as shown in Step 8 rather than querying once.

A Jaeger query returns nothing even after waiting. Check the start and end parameters. They are unix timestamps in microseconds, not milliseconds or seconds. A window in the wrong unit silently matches nothing.

The UI query box returns no results for a service you know exists. Quote the whole field name: "resource_attr:service.name":"checkout-service". Leaving it unquoted is a parse error because the field name itself contains a colon and dots.

Your collector gets 401. The credentials cover the ingest path as well as the query paths. Confirm the exporter is sending Basic Auth and that the password matches /root/victoriatraces-credentials.txt.

There is a cloudimg-selftest-... service you did not create. That is the first boot self test described in Step 12, not stray build data. See that step for what it is and how to clear it.

The service will not start. Check the journal and confirm the data disk is mounted, since VictoriaTraces will not start if its storage path is missing:

systemctl status victoriatraces.service --no-pager | tail -20
mountpoint /var/lib/victoria-traces && echo "data disk mounted"

Support

cloudimg provides 24/7 support for this image. Contact support@cloudimg.co.uk.