VictoriaTraces on AWS User Guide

Overview

This image runs VictoriaTraces, the OpenTelemetry compatible distributed tracing database from the VictoriaMetrics family. It accepts trace spans over the standard OTLP HTTP protocol and serves them back through Jaeger compatible query APIs and a built in browser interface, from a single static Go binary with no external storage dependency. There is no Cassandra, no Elasticsearch and no object store to operate.

The VictoriaTraces binary is installed at /usr/local/bin/victoria-traces-prod and runs as a dedicated unprivileged victoriatraces system account under a systemd service that starts it on boot and restarts it on failure. Stored spans live at /var/lib/victoria-traces, which is a dedicated, independently resizable EBS data volume.

VictoriaTraces ships with no built in authentication of any kind. For a tracing backend that matters twice over: an open ingest endpoint lets anyone write or poison the trace store, and an open query endpoint leaks every span in it, including hostnames, URLs, user identifiers and internal topology. This image therefore binds the database to the loopback interface only and never exposes it directly. An nginx reverse proxy publishes the web interface, the OTLP ingest path and the Jaeger query APIs on port 80 behind HTTP Basic authentication. The password is generated on the first boot of every deployed instance, so two instances launched from the same Amazon Machine Image never share a password. It is written to /root/victoriatraces-credentials.txt with mode 0600 so only the root user can read it.

Project Maturity

VictoriaTraces is a newer component of the VictoriaMetrics family, first released in July 2025 and under active development. It is pre general availability by the project's own statement: upstream has not yet committed to backward compatibility for the on disk data format, so a future major upgrade may require re ingesting spans rather than an in place upgrade. Plan your adoption accordingly. The software itself is Apache 2.0 licensed and is installed unmodified from the official upstream release.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the web interface, the OTLP ingest path and the query APIs
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for VictoriaTraces. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the web interface and the APIs. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a few seconds after the instance state becomes Running and the status checks pass.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg VictoriaTraces Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=victoriatraces}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i /path/to/your-key.pem ubuntu@<public-ip>

Step 4: Retrieve the Password

The web interface password is generated on first boot and stored in a root only file. Read it on the instance with the command below. Keep it out of shell history and out of shared terminals.

sudo cat /root/victoriatraces-credentials.txt

The file records the web interface URL, the OTLP endpoint, the username admin and the generated password. The commands later in this guide read the password straight out of this file into a shell variable, so it is never printed to your terminal.

Confirm the credentials file is present and correctly locked down without printing its contents:

sudo test -f /root/victoriatraces-credentials.txt && echo "credentials file present"
sudo stat -c '%a %U:%G' /root/victoriatraces-credentials.txt
sudo grep -c '^victoriatraces.password=' /root/victoriatraces-credentials.txt

The mode must be 600 root:root and the count must be 1.

Step 5: Sign In to the Web Interface

Open http://<public-ip>/select/vmui/ in a browser. Sign in with the username admin and the password from the previous step. The built in Traces Explorer opens with a query bar, a volume histogram and the matching spans grouped by stream.

Enter a query in the Trace query box and select Execute Query to filter the stored spans. Because the OpenTelemetry resource attribute name contains a colon, quote the field name. This query returns every span belonging to a single service:

"resource_attr:service.name":"checkout-api"

The Table tab renders the same result set as a column view, which is the quickest way to scan span timings and stream labels side by side.

The JSON tab shows the full stored record for each span, including its trace and span identifiers, its duration in nanoseconds, and every resource and span attribute your instrumentation attached.

Step 6: Confirm VictoriaTraces Is Running

Check that both services are active and that the database is listening on the loopback interface only.

systemctl is-active victoriatraces.service
systemctl is-active nginx.service
ss -tln | awk '{print $4}' | grep -c '127.0.0.1:10428'
curl -s -o /dev/null -w 'healthz: HTTP %{http_code}\n' http://127.0.0.1/healthz

Both services must report active, the loopback listener count must be 1, and the health endpoint must return HTTP 200. The health endpoint is served by nginx itself and deliberately needs no credentials, so load balancer and target group health checks work without embedding a password.

Confirm the database is not reachable on any public interface. This is the single most important security property of the image:

ss -tln | awk '{print $4}' | grep -Ec '(^0\.0\.0\.0:10428$|^\*:10428$|^\[::\]:10428$)' || true

The count must be 0. Everything from outside the instance goes through nginx on port 80.

Step 7: Confirm the Authentication Gate

Both halves of the appliance, span ingestion and span querying, sit behind the same HTTP Basic authentication gate. Verify that unauthenticated access is refused and that the generated password is accepted. The password is read into a shell variable and never printed.

PW=$(sudo sed -n 's/^victoriatraces\.password=//p' /root/victoriatraces-credentials.txt)
curl -s -o /dev/null -w 'unauthenticated query:  HTTP %{http_code}\n' http://127.0.0.1/select/jaeger/api/services
curl -s -o /dev/null -w 'unauthenticated ingest: HTTP %{http_code}\n' -X POST -H 'Content-Type: application/json' --data-binary '{}' http://127.0.0.1/insert/opentelemetry/v1/traces
curl -s -o /dev/null -w 'wrong password:         HTTP %{http_code}\n' -u 'admin:not-the-password' http://127.0.0.1/select/jaeger/api/services
curl -s -o /dev/null -w 'correct password:       HTTP %{http_code}\n' -u "admin:$PW" http://127.0.0.1/select/jaeger/api/services

The first three must return HTTP 401 and the last must return HTTP 200.

Step 8: Send Spans over OpenTelemetry OTLP

Point any OpenTelemetry SDK or an OpenTelemetry Collector at the instance's OTLP HTTP endpoint, using the same HTTP Basic credentials. The endpoint accepts protobuf or JSON, selected by the Content-Type header.

The block below sends one complete span and then reads it back through the Jaeger compatible query API, proving the full ingest to query round trip on your own instance.

PW=$(sudo sed -n 's/^victoriatraces\.password=//p' /root/victoriatraces-credentials.txt)
SVC="guide-demo-$(date +%s)"
TID=$(openssl rand -hex 16); SID=$(openssl rand -hex 8)
S=$(date +%s); E=$((S + 1))
curl -s -o /dev/null -w 'ingest: HTTP %{http_code}\n' -u "admin:$PW" \
  -H 'Content-Type: application/json' \
  --data-binary "{\"resourceSpans\":[{\"resource\":{\"attributes\":[{\"key\":\"service.name\",\"value\":{\"stringValue\":\"$SVC\"}}]},\"scopeSpans\":[{\"scope\":{\"name\":\"guide\"},\"spans\":[{\"traceId\":\"$TID\",\"spanId\":\"$SID\",\"name\":\"guide-demo-span\",\"kind\":2,\"startTimeUnixNano\":\"${S}000000000\",\"endTimeUnixNano\":\"${E}000000000\",\"status\":{}}]}]}]}" \
  http://127.0.0.1/insert/opentelemetry/v1/traces
QS=$(( (S - 3600) * 1000000 )); QE=$(( (E + 3600) * 1000000 ))
FOUND=no
for _ in $(seq 1 20); do
  BODY=$(curl -s -u "admin:$PW" "http://127.0.0.1/select/jaeger/api/traces?service=$SVC&start=$QS&end=$QE&limit=20")
  case "$BODY" in *"$TID"*) FOUND=yes; break ;; esac
  sleep 2
done
echo "round trip: span $TID returned by the Jaeger query API: $FOUND"
[ "$FOUND" = yes ]

The ingest must report HTTP 200 and the round trip line must end in yes, meaning the exact span you sent was stored and read back. A newly ingested span becomes queryable within a few seconds; the loop above allows up to 40. Each run uses a fresh service name so the assertion stays exact however many spans the instance already holds.

To point an OpenTelemetry Collector at this instance, use an OTLP HTTP exporter with basic authentication. Replace the placeholders with your instance address and generated password:

exporters:
  otlphttp:
    endpoint: http://<public-ip>
    traces_endpoint: http://<public-ip>/insert/opentelemetry/v1/traces
    auth:
      authenticator: basicauth/traces

extensions:
  basicauth/traces:
    client_auth:
      username: admin
      password: <your-password>

Step 9: Query Traces with the Jaeger Compatible APIs

VictoriaTraces exposes Jaeger compatible query endpoints, so existing Jaeger tooling and Grafana's Jaeger data source can read from it. List the services that have reported spans, and the operations recorded for one of them:

PW=$(sudo sed -n 's/^victoriatraces\.password=//p' /root/victoriatraces-credentials.txt)
curl -s -u "admin:$PW" http://127.0.0.1/select/jaeger/api/services | head -c 400
echo
SVC=$(curl -s -u "admin:$PW" http://127.0.0.1/select/jaeger/api/services \
  | python3 -c 'import json,sys; d=json.load(sys.stdin)["data"] or ["none"]; print(d[0])')
echo "inspecting operations for service: $SVC"
curl -s -o /dev/null -w 'operations API: HTTP %{http_code}\n' -u "admin:$PW" \
  "http://127.0.0.1/select/jaeger/api/services/$SVC/operations"

To connect Grafana, add a Jaeger data source with the URL http://<public-ip>/select/jaeger and enable Basic auth with the username admin and your generated password.

Step 10: Verify the Appliance End to End

The image ships a self test you can run at any time. It proves function rather than liveness: the health endpoint, the negative controls (unauthenticated ingest refused, unauthenticated query refused, wrong password refused), and a full OTLP ingest to Jaeger query round trip that reads the exact span back with its operation name and span attribute.

sudo /usr/local/sbin/victoriatraces-selftest.sh

It prints a single OK-otlp-ingest-to-jaeger-query-roundtrip-and-basic-auth-gate-proven line on success and exits non zero on any failure.

Step 11: The Data Volume

Stored spans live on a dedicated EBS volume mounted at /var/lib/victoria-traces, separate from the operating system disk and independently resizable. It is referenced in /etc/fstab by filesystem UUID, so it remounts correctly on every boot regardless of NVMe device naming.

findmnt -no SOURCE,TARGET,FSTYPE /var/lib/victoria-traces
df -h /var/lib/victoria-traces | tail -1
grep -c 'victoria-traces' /etc/fstab

To grow the store, extend the EBS volume in the AWS console or with aws ec2 modify-volume, then grow the filesystem on the instance with sudo resize2fs. No VictoriaTraces configuration changes are needed.

Step 12: Retention and Disk Sizing

Span retention defaults to 7 days. Storage is additionally capped at 80 percent of the data volume, so a burst of spans can never fill the disk and wedge the service; VictoriaTraces drops the oldest partitions first. Both settings are single flags on the service unit:

systemctl cat victoriatraces.service | grep -E 'retentionPeriod|maxDiskUsagePercent|storageDataPath'

To change them, override the unit with sudo systemctl edit victoriatraces.service, supplying a replacement ExecStart with your preferred values, then reload and restart the service. Trace volume is typically far higher than log volume, so size the data volume against your actual span rate rather than your log rate.

Step 13: Enable HTTPS

The image serves plain HTTP on port 80 because it ships without a certificate. For production, put a domain in front of the instance and terminate TLS. Point a DNS A record at the instance's public address, open port 443 in the security group, then install a certificate with Let's Encrypt:

sudo apt-get update
sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d <your-domain>

Certbot rewrites the nginx server block to listen on 443, installs the certificate and sets up automatic renewal. The HTTP Basic authentication gate is unchanged and continues to protect both the ingest and the query paths. Once TLS is live, update your OpenTelemetry exporters to the https:// endpoint.

Step 14: Backup and Maintenance

The span store is at /var/lib/victoria-traces. Because it is a dedicated EBS volume, the simplest backup is an EBS snapshot, which is consistent and needs no application downtime. Snapshot it on whatever schedule matches your retention window.

Operating system packages are updated with the distribution's normal tooling. To move to a newer VictoriaTraces release, replace the binary at /usr/local/bin/victoria-traces-prod with the official upstream build and restart the service. Because the project has not yet committed to on disk format compatibility, read the upstream release notes before any major version upgrade.

Check service health and recent activity at any time:

systemctl is-active victoriatraces.service nginx.service
curl -s -o /dev/null -w 'healthz: HTTP %{http_code}\n' http://127.0.0.1/healthz

Support

This image is published and supported by cloudimg. Technical support is available 24 hours a day, 7 days a week by email at support@cloudimg.co.uk.

Support covers deployment and instance sizing, OpenTelemetry Collector and SDK configuration against the OTLP endpoint, retention and disk sizing, TLS termination, nginx and authentication configuration, VictoriaTraces version upgrades, and troubleshooting ingestion, query and connectivity issues. Include your instance ID and AWS Region when you get in touch so we can help faster.

VictoriaTraces and VictoriaMetrics are trademarks of their respective owners. cloudimg is not affiliated with or endorsed by VictoriaMetrics.