Z3
Applications AWS

Zipkin 3.6.1 on AWS User Guide

| Product: Zipkin 3.6.1 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Zipkin 3.6.1 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg OpenTelemetry Collector, tracing and APM listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Zipkin is a distributed tracing system. Your instrumented services report spans (timed operations such as an HTTP request, a queue message or a database query) to Zipkin. Zipkin joins them into traces, so you can follow one request through every service it touched. In the web interface you find traces by service, span name, tag and duration, open a trace to see where the time went, and view the dependency graph that shows which service calls which. Zipkin accepts spans from Brave, OpenTelemetry (through the OpenTelemetry Collector's Zipkin exporter) and every other Zipkin compatible tracer.

The cloudimg image makes it safe to run on EC2:

  • Logins on everything. Stock Zipkin has no authentication: anyone who can reach port 9411 can read every trace and post spans. On this image nginx asks for a login on the web interface, the query API and span ingest. Your instance generates two logins on first boot and writes them to a root only file: admin (24 character random password) for the web interface and the query API, and spans (32 character random password) for sending spans. Each login works only on its own endpoints, so a password you put in an application's configuration cannot read your traces.
  • HTTPS from the first boot. nginx serves Zipkin on port 443 with a certificate generated on your instance's first boot, naming the public DNS name, the public and private IP addresses. Port 80 redirects to HTTPS. Nothing answers until first boot has finished.
  • Zipkin listens on localhost only (127.0.0.1:19411), so nginx is its only client. Port 9411 is not open, and the gRPC and Scribe collectors are off.
  • Failed logins are rate limited. After repeated failed logins from one address, nginx answers 429 for the rest of the minute.
  • Other web sites cannot act with your login. nginx refuses span posts and other changes that another web site's page sends in your browser, and the span ingest path refuses every request that comes from a browser at all.
  • No access to the instance metadata service. nginx and Zipkin are blocked at the kernel level from the EC2 instance metadata service (IPv4 and IPv6). Zipkin runs as its own unprivileged user with no Linux capabilities.
  • Patched libraries. The Zipkin 3.6.1 release jar bundles libraries that later received security fixes (Netty, Jackson, Spring and Micrometer among them). The image rebuilds the official Zipkin server jar with 48 of them replaced by the fixed releases of the same version line, each checked against a pinned checksum, and leaves out the optional Apache Pulsar collector and its 14 libraries. There were no known vulnerabilities in the shipped jar at build time. Zipkin runs on Ubuntu's OpenJDK 21 runtime, which Ubuntu patches; there is no JDK or compiler on the image.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Zipkin 3.6.1 (/opt/zipkin/zipkin-server.jar, settings in /etc/zipkin/zipkin.env), OpenJDK 21 runtime, nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair, and a VPC subnet with internet access.
  • A security group that allows TCP 22 (SSH), TCP 443 (Zipkin) and TCP 80 (redirects to HTTPS) from the networks you, your users and your instrumented applications work from.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Zipkin 3.6.1 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (8 GiB of memory: Zipkin's heap is capped at 1 GiB, Step 8), your key pair, a subnet and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=zipkin}]'

Step 2: Connect to your instance

Version SSH user
Zipkin 3.6.1 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot sets the host name, generates the certificate and the two logins; then Zipkin and nginx start:

systemctl is-active zipkin-firstboot zipkin nginx
sudo test -f /var/lib/cloudimg/zipkin-firstboot.done && echo FIRSTBOOT_DONE

All three units are active and the first boot marker is present:

active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on the network. Zipkin (the java process) listens on localhost only:

sudo ss -lntpH | awk '{print $4, $6}' | sed -E 's/users:\(\("([^"]+)".*/\1/' | grep -v '127.0.0.5' | sort -u

0.0.0.0:22 sshd
0.0.0.0:443 nginx
0.0.0.0:80 nginx
[::]:22 sshd
[::]:443 nginx
[::]:80 nginx
[::ffff:127.0.0.1]:19411 java

Step 4: Get your URLs and logins

sudo cat /var/lib/cloudimg/zipkin-credentials.txt

The file is readable by root only and holds the web interface URL with the admin login, and the span ingest URL with the spans login:

# Zipkin 3.6.1 - generated on first boot by zipkin-firstboot.service
# These logins are unique to this instance; none was baked into the image. Keep this file secret.

ZIPKIN_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/zipkin/
ZIPKIN_USERNAME=admin
ZIPKIN_PASSWORD=<ZIPKIN_PASSWORD>

ZIPKIN_INGEST_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/ingest/api/v2/spans
ZIPKIN_INGEST_USERNAME=spans
ZIPKIN_INGEST_PASSWORD=<ZIPKIN_INGEST_PASSWORD>
#
# Open ZIPKIN_URL in a browser (self-signed certificate minted on this instance's first boot; port 80 redirects to
# https) and sign in with ZIPKIN_USERNAME / ZIPKIN_PASSWORD: the UI and the query API (/api/v2/...).
# Applications send spans (Zipkin JSON v2, POST) to ZIPKIN_INGEST_URL with HTTP basic auth ZIPKIN_INGEST_USERNAME /
# ZIPKIN_INGEST_PASSWORD. Inside the VPC use the private address instead: https://<private-ip>/ingest/api/v2/spans
# (the certificate names it). The ingest login cannot read traces; the admin login cannot post spans through /ingest/.
# Traces are kept IN MEMORY (MEM_MAX_SPANS in /etc/zipkin/zipkin.env) and are lost when Zipkin restarts.
# Self-test:  sudo zipkin-selftest     New password:  sudo zipkin-rotate-password admin|spans
# Change the served name:  sudo zipkin-set-hostname <your-domain>
# User guide: https://www.cloudimg.co.uk/guides/zipkin-3-6-1-on-ubuntu-24-04-aws/

The URLs use the instance's public DNS name from EC2 instance metadata. Opening https://<public-ip>/ sends you on to that name.

The image includes a self test. Through nginx it checks that both logins are required and kept apart, sends a span with the ingest login and reads it back with the admin login, and checks the request guards, the service's privileges and the metadata block:

sudo zipkin-selftest

SELFTEST-OK https 301 + healthz + query API 401/429 without a login, with a wrong password and with the ingest login + UI + health UP (in-memory) + ingest 401/429 without a login, with the admin login and a wrong password + 403 browser ingest + span ingested (202) and queried back + 403 span POST without same-origin proof + 403 cross-origin POST + 444 unknown host + no CORS + zipkin uid zipkin, no capabilities, NoNewPrivs, listening 127.0.0.1:19411 only + metadata 000 inside zipkin and nginx (control 200)

The self test's span appears in Zipkin under the service name cloudimg-selftest.

Step 5: Open the web interface

Open the ZIPKIN_URL in your browser. The certificate is self signed, so your browser asks you to confirm it the first time (or install your own certificate, Step 10). The browser then asks for a user name and password: enter ZIPKIN_USERNAME and ZIPKIN_PASSWORD.

Find a trace lists recent traces. Choose a service name, and optionally a span name, tags, a minimum duration and a time range, then Run query. Each row shows the root span, the number of spans, the services involved and the total duration:

The Zipkin 3.6.1 Find a trace page on cloudimg listing five traces for the frontend service, each with the root span get /checkout, its duration and the services frontend, orders and postgres

Open a trace to see its spans on a timeline, with each span's service, duration, tags and remote endpoint:

A Zipkin trace on cloudimg: frontend get /checkout calls orders post /orders, which runs the postgres insert order query, 42 ms in total

Step 6: Send spans from your applications

Applications send spans to the ZIPKIN_INGEST_URL (/ingest/api/v2/spans, Zipkin JSON v2 or protobuf; /ingest/api/v1/spans takes the older v1 formats) with an HTTP POST and HTTP basic authentication as spans with the ZIPKIN_INGEST_PASSWORD. Zipkin answers 202 when it accepts the spans.

This example sends a small trace from the instance itself, as a client in your VPC would. The password goes to curl on its standard input, never on the command line:

F=/var/lib/cloudimg/zipkin-credentials.txt
URL=$(sudo grep '^ZIPKIN_INGEST_URL=' "$F" | cut -d= -f2-)
PASS=$(sudo grep '^ZIPKIN_INGEST_PASSWORD=' "$F" | cut -d= -f2-)
H=$(echo "$URL" | sed -E 's#^https://([^/]+)/.*#\1#')
T=$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n'); NOW=$(( $(date +%s) * 1000000 ))
SPANS=$(printf '[{"traceId":"%s","id":"%s","name":"get /checkout","kind":"SERVER","timestamp":%s,"duration":42000,"localEndpoint":{"serviceName":"frontend"},"tags":{"http.method":"GET","http.path":"/checkout"}},{"traceId":"%s","parentId":"%s","id":"%s","name":"post /orders","kind":"CLIENT","timestamp":%s,"duration":30000,"localEndpoint":{"serviceName":"frontend"},"remoteEndpoint":{"serviceName":"orders"}},{"traceId":"%s","parentId":"%s","id":"%s","name":"post /orders","kind":"SERVER","timestamp":%s,"duration":28000,"localEndpoint":{"serviceName":"orders"}},{"traceId":"%s","parentId":"%s","id":"%s","name":"insert order","kind":"CLIENT","timestamp":%s,"duration":9000,"localEndpoint":{"serviceName":"orders"},"remoteEndpoint":{"serviceName":"postgres"},"tags":{"db.system":"postgresql"}}]' \
  "$T" "${T:0:16}" "$NOW" "$T" "${T:0:16}" "${T:16:16}" "$((NOW+5000))" "$T" "${T:0:16}" "${T:16:16}" "$((NOW+6000))" "$T" "${T:16:16}" "${T:8:16}" "$((NOW+12000))")
printf 'user = "spans:%s"\n' "$PASS" | curl -sk -K - -o /dev/null -w 'span ingest: %{http_code}\n' \
  --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' --data "$SPANS" "$URL"
echo "trace id: $T"

span ingest: 202
trace id: 981c22833bfde6db342bb9635cd88f26

Trust the instance's certificate. Clients check the certificate, so give them the instance's certificate (it is public; only the key is secret), or install your own certificate (Step 10). Copy it to your machine with:

scp -i <your-key.pem> ubuntu@<public-ip>:/etc/ssl/zipkin/zipkin.crt zipkin.crt

The certificate names the public DNS name, the public IP address and the private IP address, so clients in the same VPC can send to https://<private-ip>/ingest/api/v2/spans and keep span traffic off the internet. Allow port 443 in the security group from those clients.

OpenTelemetry Collector. The usual way to connect OpenTelemetry instrumented services: they export OTLP to a Collector, and the Collector's Zipkin exporter (in the otelcol-contrib distribution) sends the spans to this instance with the ingest login. The basicauth extension holds the credentials:

extensions:
  basicauth/zipkin:
    client_auth:
      username: spans
      password: <ZIPKIN_INGEST_PASSWORD>

exporters:
  zipkin:
    endpoint: https://<private-ip>/ingest/api/v2/spans
    tls:
      ca_file: /etc/otelcol-contrib/zipkin.crt
    auth:
      authenticator: basicauth/zipkin

service:
  extensions: [basicauth/zipkin]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [zipkin]

Zipkin reporters and other tools. Any reporter that can set an HTTP header can send directly: add the header Authorization: Basic <base64 of spans:ZIPKIN_INGEST_PASSWORD> and point it at the ingest URL. A reporter that cannot set headers (or cannot be given the certificate) sends through a Collector as above, or through an OpenTelemetry Collector on the same host as the application.

Things to know:

  • Use /ingest/, not /api/v2/spans. The /api/v2/spans route on the web interface's address needs the admin login and proof that the request comes from this instance's own web pages, so that another web site cannot post spans with your browser's saved login. Span reporters send no such proof, so they get 403 there. Applications always use /ingest/.
  • The ingest path takes POST only, refuses any request carrying browser headers (Origin, Referer or Sec-Fetch-*) with 403, and accepts up to 20 MB per request.
  • A wrong or missing ingest password gets 401, and after repeated failures from one address 429 until the minute has passed. Check the password in your application if you see either.

Step 7: Dependencies and the query API

Dependencies draws the services that called each other in the chosen time range. Choose a time range and Run query:

The Zipkin dependency graph on cloudimg: frontend calls orders, and orders calls postgres

Scripts and dashboards can read traces through Zipkin's query API (/api/v2/...) with the admin login. This lists the services and fetches the latest frontend trace:

F=/var/lib/cloudimg/zipkin-credentials.txt
H=$(sudo sed -n 's#^ZIPKIN_URL=https://\([^/]*\)/.*#\1#p' "$F")
APASS=$(sudo grep '^ZIPKIN_PASSWORD=' "$F" | cut -d= -f2-)
sleep 2
printf 'user = "admin:%s"\n' "$APASS" | curl -sk -K - --resolve "$H:443:127.0.0.1" "https://$H/api/v2/services"; echo
printf 'user = "admin:%s"\n' "$APASS" | curl -sk -K - --resolve "$H:443:127.0.0.1" "https://$H/api/v2/traces?serviceName=frontend&limit=1" | python3 -c 'import json,sys; t=json.load(sys.stdin)[0]; print(len(t), "spans:", ", ".join(sorted({s["localEndpoint"]["serviceName"]+" "+s["name"] for s in t})))'

["cloudimg-selftest","frontend","orders"]
4 spans: frontend get /checkout, frontend post /orders, orders insert order, orders post /orders

The ingest login gets 401 on the query API, so an application's credentials cannot be used to read traces.

Step 8: Storage and memory

Zipkin keeps traces in memory. That suits development, testing and short term troubleshooting: it needs no database and is fast, but traces are lost when Zipkin or the instance restarts, and the oldest traces are dropped once the limit is reached. The settings are in /etc/zipkin/zipkin.env:

grep -E '^(STORAGE_TYPE|MEM_MAX_SPANS|ZIPKIN_JAVA_OPTS)=' /etc/zipkin/zipkin.env

STORAGE_TYPE=mem
MEM_MAX_SPANS=200000
ZIPKIN_JAVA_OPTS=-Xms256m -Xmx1g -XX:+ExitOnOutOfMemoryError -XX:-UsePerfData -Dcom.linecorp.armeria.useJdkDnsResolver=true
  • MEM_MAX_SPANS is how many spans Zipkin keeps (200,000 by default).
  • -Xmx1g caps Zipkin's memory at 1 GiB. If Zipkin ever runs out of it, it exits and systemd starts it again (with the stored traces gone).

To keep more traces, raise both together, for example MEM_MAX_SPANS=500000 with -Xmx2g on an instance with enough memory, then restart Zipkin:

sudo nano /etc/zipkin/zipkin.env
sudo systemctl restart zipkin

Step 9: Change the passwords

To give either login a new random password, run the helper. It updates nginx and the credentials file and prints nothing secret; read the new password from the file (Step 4):

sudo zipkin-rotate-password admin
sudo zipkin-rotate-password spans

After rotating spans, update the password in every application or Collector that sends spans. Passwords are stored in nginx only as SHA-512 crypt hashes (/etc/nginx/zipkin-admin.htpasswd and /etc/nginx/zipkin-ingest.htpasswd).

Step 10: Use your own domain and certificate

  1. In your DNS, point <your-domain> (an A record) at the instance's public IP address. An Elastic IP keeps that address stable.
  2. Tell the instance to use it. This sets the name nginx accepts, the request guards and the URLs in the credentials file, and generates a new self signed certificate for it:
sudo zipkin-set-hostname <your-domain>
  1. For a certificate your browser and clients trust, put the certificate chain for <your-domain> in /etc/ssl/zipkin/zipkin.crt and its private key in /etc/ssl/zipkin/zipkin.key (root only, mode 600), and run sudo systemctl reload nginx. Alternatively put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate that forwards to port 443 on the instance.

Step 11: Security checks

Without a login, the web interface, the query API and span ingest are refused; a span post from a browser page is refused; a request for any other host name gets no answer (nginx closes the connection); and Zipkin's own ports are closed to the network:

H=$(sudo sed -n 's#^ZIPKIN_URL=https://\([^/]*\)/.*#\1#p' /var/lib/cloudimg/zipkin-credentials.txt)
curl -sk -o /dev/null -w 'UI without a login: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/zipkin/"
curl -sk -o /dev/null -w 'query API without a login: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/api/v2/services"
curl -sk -o /dev/null -w 'span POST without a login: %{http_code}\n' --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' --data '[]' "https://$H/ingest/api/v2/spans"
curl -sk -o /dev/null -w 'span POST from a browser page: %{http_code}\n' --resolve "$H:443:127.0.0.1" -H 'Content-Type: application/json' -H 'Origin: https://evil.example' --data '[]' "https://$H/ingest/api/v2/spans"
curl -sk -o /dev/null -w 'unknown host name: %{http_code}\n' --resolve evil.example:443:127.0.0.1 https://evil.example/ || true
for p in 9411 19411; do if timeout 3 bash -c "exec 3<>/dev/tcp/$(hostname -I | awk '{print $1}')/$p" 2>/dev/null; then echo "port $p: open"; else echo "port $p: closed"; fi; done

UI without a login: 401
query API without a login: 401
span POST without a login: 401
span POST from a browser page: 403
unknown host name: 000
port 9411: closed
port 19411: closed

(If you have just made several failed logins from the instance, the first three show 429 instead of 401 for up to a minute.)

nginx and Zipkin are blocked from the instance metadata service:

for u in nginx zipkin; do printf '%s ' "$u"; systemctl show -p IPAddressDeny --value "$u" | tr ' ' '\n' | sort | tr '\n' ' '; echo; done

nginx 168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128
zipkin 168.63.129.16/32 169.254.169.254/32 fd00:ec2::254/128

Zipkin runs as the zipkin user with no Linux capabilities and cannot gain any:

P=$(systemctl show -p MainPID --value zipkin)
ps -o user= -p "$P"
grep -E '^(CapEff|CapBnd|CapAmb|NoNewPrivs)' /proc/"$P"/status

zipkin
CapEff: 0000000000000000
CapBnd: 0000000000000000
CapAmb: 0000000000000000
NoNewPrivs: 1

The credentials file and the certificate key are root only; nginx can read the password hashes but not change them:

sudo stat -c '%a %U:%G %n' /var/lib/cloudimg/zipkin-credentials.txt /etc/nginx/zipkin-admin.htpasswd /etc/nginx/zipkin-ingest.htpasswd /etc/ssl/zipkin/zipkin.key /etc/zipkin/zipkin.env /var/lib/zipkin

600 root:root /var/lib/cloudimg/zipkin-credentials.txt
640 root:www-data /etc/nginx/zipkin-admin.htpasswd
640 root:www-data /etc/nginx/zipkin-ingest.htpasswd
600 root:root /etc/ssl/zipkin/zipkin.key
644 root:root /etc/zipkin/zipkin.env
700 zipkin:zipkin /var/lib/zipkin

Other notes:

  • Who can reach it. Open ports 80 and 443 in the security group only to the networks your users and applications work from. Spans can contain request paths, tags and error messages, so treat the admin login like any other access to your application data.
  • Headers. nginx sends a strict Content Security Policy, HSTS, X-Content-Type-Options: nosniff and X-Frame-Options: DENY, and removes Zipkin's Access-Control-Allow-Origin: *, because the web interface is served from the same address.
  • Client addresses. nginx passes Zipkin the address of the connection, not one a client claims in X-Forwarded-For, and never passes your login on to Zipkin.
  • The build. /usr/share/doc/cloudimg-zipkin/BUILD-INFO.txt records the release and how the jar was rebuilt; jar-rebuild-plan.txt lists each replaced library with its checksum, and bundled-jars.txt every library in the shipped jar.
  • Logs. nginx logs are in /var/log/nginx (without query strings); Zipkin logs to the journal (journalctl -u zipkin).

Step 12: Keep it up to date

Ubuntu security updates, including the OpenJDK runtime, install automatically. To apply all updates now:

sudo apt-get update && sudo apt-get upgrade -y

New Zipkin releases arrive as new versions of this listing, built and checked the same way. Because traces are kept in memory there is nothing to migrate: launch the new version, point your applications or Collector at it, and stop the old instance. Read the Zipkin release notes before you move.

Support and licensing

Zipkin is open source under the Apache License 2.0 (The OpenZipkin Authors); the licence text is in /usr/share/doc/cloudimg-zipkin/Zipkin-LICENSE, and the source code is at github.com/openzipkin/zipkin. The libraries in the Zipkin server are open source, mainly under the Apache 2.0, MIT, BSD and EPL licences (bundled-jars.txt in the same directory lists them), and OpenJDK is licensed under the GPL version 2 with the Classpath Exception. There is no per user licence fee. The AMI price covers cloudimg's build, hardening, first boot setup and support.

  • Email: support@cloudimg.co.uk
  • Website: https://www.cloudimg.co.uk
  • Support: 24/7, with a 24 hour response SLA

The other versions on this listing are on the OpenTelemetry Collector, tracing and APM guide.