Observability Azure

Jaeger with OpenTelemetry Collector on Ubuntu 24.04 on Azure User Guide

| Product: Jaeger with OpenTelemetry Collector on Ubuntu 24.04 LTS on Azure

Overview

This image is a complete distributed tracing pipeline in a single virtual machine. Applications send OpenTelemetry (OTLP) telemetry to the OpenTelemetry Collector, which is the only publicly reachable ingest point. The collector limits memory, enriches every span with resource attributes, batches the stream, and exports it over OTLP to Jaeger, which stores the traces in an embedded local database and serves the trace search, timeline and service dependency views.

The two halves are wired together as one pipeline rather than shipped side by side. Jaeger does not listen on a routable interface at all: its query API, its ingest port and every health and metrics endpoint are bound to the loopback address, so every span you see in the UI physically travelled through the collector first. The trace interface is published by nginx over TLS behind a credential that is generated on your own instance the first time it boots.

What is included:

  • OpenTelemetry Collector contrib 0.158.0 (Apache-2.0) as the OTLP ingest and processing gateway
  • Jaeger 2.20.0 (Apache-2.0, a CNCF graduated project) as the trace store and UI
  • Embedded Badger trace storage on the instance itself, so the appliance runs standalone with no external database
  • A unique administrator credential, a unique telemetry ingest credential and a unique TLS certificate, all generated on first boot
  • OTLP over gRPC and OTLP over HTTP, both protected by TLS and by HTTP Basic authentication
  • An on instance pipeline self test that pushes a span through the collector and reads it back out of Jaeger
  • jaeger.service, otelcol-contrib.service and nginx.service as systemd units, enabled and active
  • 24/7 cloudimg support

Key facts
Default login user azureuser
Trace UI https://<vm-public-ip>/
OTLP HTTP ingest https://<vm-public-ip>:4318/v1/traces
OTLP gRPC ingest <vm-public-ip>:4317
Credentials file /etc/cloudimg-credentials.txt (mode 0600, root only)
TLS certificate /etc/ssl/cloudimg/jaeger-otel-collector.crt
Trace storage /var/lib/jaeger/badger

Prerequisites

An active Azure subscription, an SSH key pair, and a virtual network and subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is the recommended starting size; the whole pipeline idles at roughly 270 MB of memory.

Network security group inbound rules:

  1. 22/tcp from your management network, for SSH.
  2. 443/tcp from wherever engineers will read traces, for the Jaeger UI.
  3. 4317/tcp and 4318/tcp from your application subnet only, for telemetry ingest. These ports are the reason the product exists, so they are bound on the public interface, but they should be scoped to the workloads that send traces rather than opened to the internet. They are protected by TLS and a credential either way.

Port 80/tcp is optional and only issues a redirect to 443.

Step 1 Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Jaeger with OpenTelemetry Collector by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTPS (443). Then Review + create, then Create.

Add the two telemetry ingest ports afterwards, scoped to your application subnet.

Step 2 Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name jaeger-otel \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

Open the UI port, and the ingest ports scoped to the subnet your applications run in:

az vm open-port --resource-group <your-rg> --name jaeger-otel --port 443 --priority 1001

az network nsg rule create --resource-group <your-rg> \
  --nsg-name jaeger-otelNSG --name allow-otlp \
  --priority 1002 --access Allow --protocol Tcp --direction Inbound \
  --source-address-prefixes 10.0.1.0/24 \
  --destination-port-ranges 4317 4318

Step 3 Confirm the pipeline came up

Connect over SSH and check the three services. Every one of them is gated on the first boot bootstrap completing, so active on all three means your instance generated its own credentials and certificate successfully.

ssh azureuser@<vm-public-ip>

sudo systemctl is-active jaeger otelcol-contrib nginx

Expected output:

active
active
active

The three pipeline services reporting active, with the first boot marker and sentinel present and the credentials file owned by root with mode 0600

Both processes also expose a health endpoint on the loopback interface:

curl -s -o /dev/null -w 'jaeger health: %{http_code}\n' http://127.0.0.1:14269/status
curl -s -o /dev/null -w 'collector health: %{http_code}\n' http://127.0.0.1:13133/

Expected output:

jaeger health: 200
collector health: 200

The pinned upstream versions and the checksums they were verified against are recorded on the instance:

/opt/otelcol-contrib/otelcol-contrib --version
cat /etc/cloudimg-jaeger-otel-collector.release

Expected output:

otelcol-contrib version 0.158.0
JAEGER_VERSION=2.20.0
JAEGER_SOURCE=https://github.com/jaegertracing/jaeger/releases/download/v2.20.0/jaeger-2.20.0-linux-amd64.tar.gz
JAEGER_BINARY_SHA256=9ee6dc46b776137ea3fa4d56a54a14d64e4b97714441b4196fb519af2c25e5c8
OTELCOL_DISTRIBUTION=otelcol-contrib
OTELCOL_VERSION=0.158.0
OTELCOL_SOURCE=https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.158.0/otelcol-contrib_0.158.0_linux_amd64.tar.gz
OTELCOL_TARBALL_SHA256=7623348c295ec7b00d86c30040a30730f7e3537e813b34c880c1d5abb9bbe8d5

Both binaries reporting their pinned versions alongside the recorded upstream download URLs and SHA256 checksums

Step 4 Read your instance credentials

Nothing in this image ships with a password. Your instance generates its own administrator credential, its own telemetry ingest credential and its own TLS certificate the first time it boots, and writes them to a root only file. The endpoint addresses and usernames are:

sudo grep -E '^(JAEGER_UI_URL|JAEGER_UI_USERNAME|OTLP_HTTP_URL|OTLP_GRPC_ENDPOINT|OTLP_INGEST_USERNAME|OTLP_TLS_CERTIFICATE)=' /etc/cloudimg-credentials.txt

Example output (the address will be your own instance's):

JAEGER_UI_URL=https://10.0.0.10/
JAEGER_UI_USERNAME=cloudimg
OTLP_HTTP_URL=https://10.0.0.10:4318
OTLP_GRPC_ENDPOINT=10.0.0.10:4317
OTLP_INGEST_USERNAME=otel-ingest
OTLP_TLS_CERTIFICATE=/etc/ssl/cloudimg/jaeger-otel-collector.crt

Read the two passwords with:

sudo grep PASSWORD /etc/cloudimg-credentials.txt

JAEGER_UI_PASSWORD signs you in to the trace interface. OTLP_INGEST_PASSWORD is what your applications present when they send telemetry. They are different values, and neither exists anywhere else.

Azure's instance metadata service returns an empty value for a Standard SKU public IP, so the addresses written at first boot are your instance's private address. Use the public IP of the VM from the browser and from anything outside the virtual network.

Step 5 Prove the pipeline carries a span

The image ships with the same end to end check that cloudimg runs before release. It pushes a span into the collector with your instance's ingest credential, then reads that exact trace back out of Jaeger through the authenticated front door.

sudo /usr/local/sbin/cloudimg-jaeger-e2e-probe.sh

Expected output (your trace and span identifiers will differ):

E2E_PIPELINE_OK trace=019b1a6615b82cded612a6d60c64aa24 span=140756abee2060b6 operation=cloudimg-e2e-probe

If that line prints, telemetry ingest, the collector's processing pipeline, the export to Jaeger, trace storage and the authenticated query path are all working together.

The on instance pipeline self test reporting a span carried end to end, and the Jaeger service list showing the services that produced traces

Step 6 Send your own span

You can drive the same path by hand with curl. This sends one OTLP span over HTTP as a service called my-first-service, then reads the trace back from Jaeger.

CERT=/etc/ssl/cloudimg/jaeger-otel-collector.crt
TRACE_ID=$(openssl rand -hex 16)
SPAN_ID=$(openssl rand -hex 8)
NOW=$(date +%s)000000000
cat > /tmp/my-span.json <<JSON
{"resourceSpans":[{"resource":{"attributes":[
 {"key":"service.name","value":{"stringValue":"my-first-service"}}]},
 "scopeSpans":[{"scope":{"name":"manual"},"spans":[
 {"traceId":"$TRACE_ID","spanId":"$SPAN_ID","name":"my-first-span","kind":1,
  "startTimeUnixNano":"$NOW","endTimeUnixNano":"$NOW"}]}]}]}
JSON
INGEST_PASS=$(sudo sed -n 's/^OTLP_INGEST_PASSWORD=//p' /etc/cloudimg-credentials.txt)
UI_PASS=$(sudo sed -n 's/^JAEGER_UI_PASSWORD=//p' /etc/cloudimg-credentials.txt)
curl -s -o /dev/null -w 'ingest accepted: %{http_code}\n' --cacert "$CERT" \
  -u "otel-ingest:$INGEST_PASS" -H 'Content-Type: application/json' \
  --data-binary @/tmp/my-span.json https://127.0.0.1:4318/v1/traces
sleep 8
curl -s --cacert "$CERT" -u "cloudimg:$UI_PASS" \
  "https://127.0.0.1/api/traces/$TRACE_ID" | head -c 160
echo
rm -f /tmp/my-span.json

Expected output (the trace body will contain your generated identifiers):

ingest accepted: 200
{"data":[{"traceID":"...","spans":[{"traceID":"...","spanID":"...","operationName":"my-first-span",

Step 7 Browse the trace interface

Open https://<vm-public-ip>/ in a browser and sign in with cloudimg and JAEGER_UI_PASSWORD. Because the certificate is generated on your own instance and signed by itself, the browser will warn the first time; see Replacing the TLS certificate below to issue one for your own name.

The search view lists every service that has sent telemetry through the collector.

The Jaeger search view with the service selector listing the services whose telemetry arrived through the OpenTelemetry Collector

Choosing a service and searching returns the matching traces with their duration, span count and the services each one touched. Failed requests are flagged in the errors column and stand out in red on the duration scatter plot.

Jaeger search results showing fourteen checkout traces on a duration scatter plot, with span counts, the services each trace touched, and one trace flagged with errors

Opening a trace draws the full request timeline. This is what the pipeline is for: one request fanning out across five services, with each span's duration and parent relationship laid out so a slow call is obvious at a glance.

A Jaeger trace timeline for a 483ms checkout request, showing five spans across five services at three levels of depth, with each span's duration drawn on the waterfall

The System Architecture view builds a service dependency graph from the spans the collector delivered, with the call counts on each edge.

The Jaeger System Architecture view showing the service dependency graph derived from the collected traces, with call counts on each edge

Step 8 Point a real application at the collector

Any OpenTelemetry SDK or agent can send to this instance. Set the standard OTLP environment variables on your application, using your instance address, the ingest credential from Step 4, and the instance certificate as the trust anchor. Copy /etc/ssl/cloudimg/jaeger-otel-collector.crt from the VM to your application host first.

OTLP over HTTP:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://<vm-public-ip>:4318
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/jaeger-otel-collector.crt
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(printf 'otel-ingest:<OTLP_INGEST_PASSWORD>' | base64 -w0)"
export OTEL_SERVICE_NAME=my-service

OTLP over gRPC uses the same credential on port 4317:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://<vm-public-ip>:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_CERTIFICATE=/path/to/jaeger-otel-collector.crt
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(printf 'otel-ingest:<OTLP_INGEST_PASSWORD>' | base64 -w0)"
export OTEL_SERVICE_NAME=my-service

Traces appear in the search view within a few seconds of the collector batching them.

The listener and authentication model

Only four ports face the network, and both of the ones that accept data require a credential.

sudo ss -ltn | grep -E ':(80|443|4317|4318|16686|14317|14269|13133|8888)'

Expected output:

LISTEN 0      511          0.0.0.0:80         0.0.0.0:*
LISTEN 0      511          0.0.0.0:443        0.0.0.0:*
LISTEN 0      4096       127.0.0.1:14317      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:14269      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:16686      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:16685      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:13133      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:8888       0.0.0.0:*
LISTEN 0      4096                *:4318             *:*
LISTEN 0      4096                *:4317             *:*

The listening sockets on the instance, with Jaeger's query, ingest, health and metrics ports bound to the loopback address and only nginx and the collector's OTLP ports on the public interface

Port Interface What it is Protection
443 public Jaeger trace UI and query API via nginx TLS plus bcrypt basic authentication
80 public Redirect to 443 only no content served
4318 public Collector OTLP over HTTP TLS plus bcrypt basic authentication
4317 public Collector OTLP over gRPC TLS plus bcrypt basic authentication
16686, 16685 loopback Jaeger query HTTP and gRPC not reachable off the instance
14317 loopback Jaeger OTLP ingest, from the collector only not reachable off the instance
14269, 14271 loopback Jaeger health and metrics not reachable off the instance
13133, 8888 loopback Collector health and metrics not reachable off the instance

Neither process enables zPages or pprof. Confirm that unauthenticated callers are turned away:

CERT=/etc/ssl/cloudimg/jaeger-otel-collector.crt
echo "anonymous UI:          $(curl -s -o /dev/null -w '%{http_code}' --cacert $CERT https://127.0.0.1/)"
echo "anonymous OTLP ingest: $(curl -s -o /dev/null -w '%{http_code}' --cacert $CERT -X POST -H 'Content-Type: application/json' -d '{}' https://127.0.0.1:4318/v1/traces)"

Expected output:

anonymous UI:          401
anonymous OTLP ingest: 401

How first boot protects the instance

The image contains no credential and no certificate. On first boot jaeger-otel-collector-firstboot.service generates both passwords and the TLS keypair, writes them to the root only credentials file, and then creates a bootstrap marker. All three services carry ConditionPathExists on that marker and run a start time guard before they launch, so if the bootstrap has not run they do not start at all rather than starting with anything shipped in the image. The guard also refuses to run when a credential is too short, when it matches a published example value, or when the certificate and private key do not match.

systemctl is-enabled jaeger-otel-collector-firstboot.service jaeger.service otelcol-contrib.service nginx.service

Expected output:

enabled
enabled
enabled
enabled

Replacing the TLS certificate

The first boot certificate covers the loopback address, the instance hostname and the private IP, because Azure does not report a Standard SKU public IP through instance metadata. Re issue it with the names you actually use, which keeps the existing local names as well:

sudo /usr/local/sbin/cloudimg-jaeger-tls-reissue.sh <vm-public-ip> tracing.example.com

The helper writes the new keypair, reloads nginx and restarts the collector. For a certificate a browser trusts without a warning, point a DNS record at the instance and use certbot instead: install it with sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx, then run sudo certbot --nginx -d tracing.example.com. Update the collector's cert_file and key_file in /etc/otelcol-contrib/config.yaml to the certbot paths if you want the ingest ports to use the same certificate.

Trace storage

Traces are stored in an embedded Badger database on the instance, so nothing external needs provisioning. Live traces are kept for 72 hours and the archive store for 30 days.

sudo du -sh /var/lib/jaeger/badger
df -h /var/lib/jaeger | tail -1

Example output:

36K /var/lib/jaeger/badger
/dev/root        29G  2.8G   26G  10% /var/lib/jaeger

To keep traces for longer, raise ttl.spans under jaeger_storage in /etc/jaeger/config.yaml and restart jaeger.service. For volumes beyond a single instance, change the same backend block to Elasticsearch, OpenSearch or Cassandra following the Jaeger storage documentation; the collector configuration does not change, because it always exports to the local Jaeger endpoint.

Server components

Component Version Purpose
OpenTelemetry Collector contrib 0.158.0 OTLP ingest, memory limiting, resource detection, batching, export to Jaeger
Jaeger 2.20.0 Trace storage, query API, web interface
nginx 1.24.0 TLS termination and authentication in front of the trace interface
Ubuntu Server 24.04 LTS Operating system, fully patched with unattended security upgrades enabled

Filesystem layout

Path Purpose
/opt/jaeger Jaeger binary, symlinked to the pinned version directory
/opt/otelcol-contrib Collector binary, symlinked to the pinned version directory
/etc/jaeger/config.yaml Jaeger configuration: storage backends, query endpoints, health
/etc/otelcol-contrib/config.yaml Collector configuration: receivers, processors, exporter
/etc/otelcol-contrib/.htpasswd-ingest Telemetry ingest credential, bcrypt
/etc/nginx/.htpasswd-jaeger-otel Trace interface credential, bcrypt
/etc/ssl/cloudimg/ Per instance TLS certificate and key
/etc/cloudimg-credentials.txt Per instance credentials, mode 0600 root only
/var/lib/jaeger/badger Live trace storage
/var/lib/jaeger/badger-archive Archive trace storage
/var/lib/cloudimg/ First boot sentinel and bootstrap marker

Scripts and log files

Path Purpose
/usr/local/sbin/cloudimg-jaeger-e2e-probe.sh End to end pipeline self test
/usr/local/sbin/cloudimg-jaeger-tls-reissue.sh Re issue the instance TLS certificate with extra names
/usr/local/sbin/jaeger-otel-guard.sh Start time bootstrap guard, run before each service
/usr/local/sbin/jaeger-otel-collector-firstboot.sh First boot secret and certificate generation
/var/log/cloudimg-firstboot.log First boot log

Maintenance

  • Service logs: sudo journalctl -u otelcol-contrib.service -f follows the ingest side, sudo journalctl -u jaeger.service -f the storage and query side.
  • Restart: sudo systemctl restart jaeger.service otelcol-contrib.service nginx.service.
  • OS updates: unattended security upgrades stay enabled. Apply everything pending at any time with sudo apt-get update && sudo apt-get upgrade -y.
  • Rotating the trace interface password: sudo htpasswd -B /etc/nginx/.htpasswd-jaeger-otel cloudimg, then sudo systemctl reload nginx. Record the new value in /etc/cloudimg-credentials.txt so the self test keeps working.
  • Rotating the ingest credential: sudo htpasswd -B /etc/otelcol-contrib/.htpasswd-ingest otel-ingest, then sudo systemctl restart otelcol-contrib. Update your applications at the same time.

Troubleshooting

A service is inactive. All three are gated on the first boot bootstrap. Check it completed: sudo systemctl status jaeger-otel-collector-firstboot.service and ls -l /var/lib/cloudimg/. If the bootstrap marker is missing, the guard is doing its job and refusing to start the appliance with no credentials. Read /var/log/cloudimg-firstboot.log for the reason.

Spans are accepted but do not appear. The collector batches for up to two seconds and Jaeger writes asynchronously, so allow a few seconds. If they still do not appear, check the collector is reaching Jaeger with sudo journalctl -u otelcol-contrib.service -n 50, and confirm Jaeger's ingest port is listening with sudo ss -ltn | grep 14317.

Ingest returns 401. The credential is wrong, or the header is missing. Confirm with sudo grep OTLP_INGEST_PASSWORD /etc/cloudimg-credentials.txt and remember the username is otel-ingest, not cloudimg.

Ingest returns a TLS error. Your client does not trust the instance certificate. Copy /etc/ssl/cloudimg/jaeger-otel-collector.crt to the client and point OTEL_EXPORTER_OTLP_CERTIFICATE at it, or re issue the certificate for the name your client connects to.

The browser warns about the certificate. That is expected for a certificate the instance signed itself. Use certbot for a publicly trusted certificate, as described above.

Security recommendations

  1. Scope the network security group rules for 4317 and 4318 to your application subnet, and 443 to the networks your engineers use. Do not publish either to the internet.
  2. Replace the first boot certificate with one issued for a name you control, so clients verify it properly rather than pinning a self signed file.
  3. Keep /etc/cloudimg-credentials.txt at mode 0600 and owned by root. The start time guard refuses to launch the services if it is loosened.
  4. Rotate both credentials after handing the instance over, using the commands in Maintenance.
  5. Leave the loopback bindings alone. Jaeger's query API has no authentication of its own; nginx is what protects it.

Support

This image is maintained by cloudimg with 24/7 support. Contact support@cloudimg.co.uk for assistance.