Observability AWS

Grafana Pyroscope 2.2.3 on AWS User Guide

| Product: Grafana Pyroscope 2.2.3 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Grafana Pyroscope 2.2.3 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg tracing and APM listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Grafana Pyroscope is an open source continuous profiling database. Your applications, or a collector such as Grafana Alloy, send it CPU, memory, lock and other profiles all the time, and Pyroscope stores them and shows them as flamegraphs, so you can see which functions use your CPU and memory and how that changes from one deploy to the next.

What the cloudimg image adds:

  • No default login, for people and for agents. Pyroscope has no authentication of its own, so this image never exposes it. nginx serves the web UI, the ingest API that agents push profiles to and the query APIs over HTTPS with HTTP basic authentication, and the password is generated on your instance's first boot and written to /root/grafana-pyroscope-credentials.txt (root only). The image ships no password, no password hash and no TLS key.
  • Pyroscope on localhost only. Every port Pyroscope opens (4040 HTTP, 9095 gRPC, 7946 cluster gossip, 9099 metadata store) is bound to 127.0.0.1; nginx on port 443 is the only way in. The unauthenticated /healthz path answers ok for load balancer health checks and reveals nothing else.
  • HTTPS from the first boot. The certificate is generated on your instance for its own addresses, port 80 redirects to HTTPS, and requests for a host name the instance does not serve are dropped. Cross-site form posts from other web sites are refused.
  • Fail closed. nginx and Pyroscope start only after first boot has finished. If it cannot finish, nothing is served.
  • Nothing sent home. Pyroscope's anonymous usage statistics and its self-profiling are switched off, so the database holds only the profiles you send it and the instance makes no calls to Grafana.
  • No metadata access. pyroscope.service, nginx.service and the first boot service are blocked from the instance metadata service, and the AMI requires IMDSv2.
  • Honest client addresses. nginx hands Pyroscope a fixed host name and the real client address from the connection, never forwarding headers a client sends.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Grafana Pyroscope 2.2.3 (the upstream release binary, checksum verified) in single-binary mode with local disk storage, nginx 1.24 and OpenSSL 3.0 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

The Pyroscope explore view with the CPU timeline, the top functions table and the flamegraph of a service

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows TCP 22 (SSH) from your admin network and TCP 443 (web UI and ingest) only from the people and the agents or applications that send profiles. Port 80 only redirects to HTTPS, so you can leave it closed. Never open 4040, 9095, 7946 or 9099: Pyroscope listens on the loopback address only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Grafana Pyroscope 2.2.3 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above. Profiles are stored on the root volume under /var/lib/pyroscope/data, so give it room for your retention period (31 days by default).

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=pyroscope}]'

First boot takes a few seconds after the instance is running; Pyroscope reports ready about a minute after it starts.

Step 2: Connect to your instance

Version SSH user
Grafana Pyroscope 2.2.3 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot sets the password and the certificate, then nginx and Pyroscope start:

systemctl is-active pyroscope nginx pyroscope-firstboot
sudo test -f /var/lib/cloudimg/pyroscope-firstboot.done && echo FIRSTBOOT_DONE

All three units are active and the first boot marker is present:

active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network ports; everything Pyroscope opens is on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:4040 users:(("pyroscope"))
127.0.0.1:7946 users:(("pyroscope"))
127.0.0.1:9095 users:(("pyroscope"))
127.0.0.1:9099 users:(("pyroscope"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

Pyroscope's own readiness endpoint, on the loopback address:

curl -s http://127.0.0.1:4040/ready; echo

ready

Step 4: Get your URL and password

sudo cat /root/grafana-pyroscope-credentials.txt

The file is readable by root only:

# Grafana Pyroscope 2.2.3 on Ubuntu 24.04 (cloudimg) - generated on this instance's first boot.
# This password is unique to this instance; none was baked into the image. Keep this file secret.

PYROSCOPE_URL=https://<public-ip>/
PYROSCOPE_USER=admin
PYROSCOPE_PASSWORD=<PYROSCOPE_PASSWORD>

# Web UI:  https://<host>/   (HTTP basic auth with the user and password above)
# Ingest:  agents and SDKs push to https://<host>/ with the SAME basic-auth user and password, e.g. Grafana
#          Alloy:  pyroscope.write "cloudimg" { endpoint { url = "https://<host>"
#                    basic_auth { username = "admin"  password_file = "/etc/alloy/pyroscope.pass" } } }
# Health:  https://<host>/healthz   (no login; answers "ok")
# The certificate is self signed (minted on this instance): accept the browser warning once, give agents
# the certificate (/etc/nginx/tls/pyroscope.crt), or install your own. Serve a DNS name instead:
#   sudo pyroscope-set-hostname <your-dns-name>
# Pyroscope listens on 127.0.0.1 only; nginx on 443 is the only way in. Data: /var/lib/pyroscope/data
# Self-test: sudo /usr/local/sbin/pyroscope-selftest.sh
# User guide: https://www.cloudimg.co.uk/guides/grafana-pyroscope-2-2-3-on-ubuntu-24-04-aws/

The URL uses the instance's public IP address from the EC2 instance metadata. The certificate also names the public DNS name and the private IP address, so https://<public-dns-name>/ and https://<private-ip>/ work too. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it. To serve Pyroscope under your own DNS name, see Step 9.

A request without the password gets nginx's 401, and the health check answers without one:

curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/
curl -sk https://127.0.0.1/healthz

401
ok

Step 5: Send a first profile

Every way of sending profiles (the Pyroscope SDKs, Grafana Alloy, or a plain HTTP upload) goes to https://<host>/ with the user admin and the password from the credentials file. To see it work on the instance, upload an example CPU profile in the folded stack format (one call stack per line, followed by its sample count) for a service called shop-api, six times, one for each 10 second window of the last minute.

The small shell function pyro passes the password from the credentials file to curl on standard input, so it never appears on a command line:

pyro() { sudo sed -n 's/^PYROSCOPE_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/grafana-pyroscope-credentials.txt | curl -sgk -K - "$@"; }
cat > /tmp/shop-api.folded <<'EOF'
main;net/http.(*conn).serve;shop/api.(*Server).Checkout;shop/db.(*Pool).Query;database/sql.(*DB).QueryContext 910
main;net/http.(*conn).serve;shop/api.(*Server).Checkout;shop/cart.Total;shop/pricing.ApplyDiscounts 420
main;net/http.(*conn).serve;shop/api.(*Server).Checkout;encoding/json.Marshal 260
main;net/http.(*conn).serve;shop/api.(*Server).Search;shop/search.Rank;sort.Slice 640
main;net/http.(*conn).serve;shop/api.(*Server).Search;shop/db.(*Pool).Query;database/sql.(*DB).QueryContext 380
main;net/http.(*conn).serve;shop/api.(*Server).Health 15
runtime.gcBgMarkWorker;runtime.gcDrain;runtime.scanobject 190
EOF
NOW=$(date +%s)
for i in 0 1 2 3 4 5; do
  FROM=$((NOW - 60 + i * 10))
  pyro -o /dev/null -w '%{http_code} ' --data-binary @/tmp/shop-api.folded \
    "https://127.0.0.1/ingest?name=shop-api.cpu{env=demo}&from=${FROM}&until=$((FROM + 10))&format=folded&sampleRate=100&spyName=gospy"
done; echo

Each upload is accepted:

200 200 200 200 200 200

The same upload without the password is refused by nginx before it reaches Pyroscope:

curl -sk -o /dev/null -w '%{http_code}\n' --data-binary @/tmp/shop-api.folded "https://127.0.0.1/ingest?name=intruder.cpu&format=folded"

401

Step 6: Explore the profile

Ask Pyroscope which services it has data for, then read the flamegraph back through the query API:

pyro() { sudo sed -n 's/^PYROSCOPE_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/grafana-pyroscope-credentials.txt | curl -sgk -K - "$@"; }
NOWMS=$(( $(date +%s) * 1000 ))
pyro -X POST -H 'Content-Type: application/json' \
  -d "{\"name\":\"service_name\",\"start\":$((NOWMS - 3600000)),\"end\":${NOWMS}}" \
  https://127.0.0.1/querier.v1.QuerierService/LabelValues; echo

{"names":["shop-api"]}

pyro() { sudo sed -n 's/^PYROSCOPE_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/grafana-pyroscope-credentials.txt | curl -sgk -K - "$@"; }
pyro -G https://127.0.0.1/pyroscope/render \
  --data-urlencode 'query=process_cpu:cpu:nanoseconds:cpu:nanoseconds{service_name="shop-api"}' \
  --data-urlencode 'from=now-15m' --data-urlencode 'until=now' --data-urlencode 'format=json' \
  | jq -r '.flamebearer as $f | "numTicks=\($f.numTicks)", ($f.names[1:] | sort | .[])'

numTicks=168900000000
database/sql.(*DB).QueryContext
encoding/json.Marshal
main
net/http.(*conn).serve
runtime.gcBgMarkWorker
runtime.gcDrain
runtime.scanobject
shop/api.(*Server).Checkout
shop/api.(*Server).Health
shop/api.(*Server).Search
shop/cart.Total
shop/db.(*Pool).Query
shop/pricing.ApplyDiscounts
shop/search.Rank
sort.Slice

In the browser, open the URL from the credentials file. Your browser warns once about the self signed certificate; accept it (or install your own certificate, Step 9), then sign in as admin with the password from the credentials file. The explore view opens on a service: the timeline at the top shows CPU use over the selected time range, and below it the top functions table and the flamegraph show where the time went. Here database/sql.(*DB).QueryContext is the widest leaf, so database queries are where shop-api spends most of its CPU.

Switch the view to Flame Graph to see the call tree on its own; click a frame to zoom into it, and use Search to highlight a function:

The flamegraph of the shop-api example service, with Checkout and Search under main and the database query as the widest leaf

The service menu at the top lists every service that has sent profiles, and each service's profile types (CPU, memory allocations, goroutines, locks and so on, depending on what the agent sends):

The service and profile type menu listing the shop-api service

The example data is yours to delete: it ages out with the retention period, or stop Pyroscope, empty /var/lib/pyroscope/data and start it again for a clean store.

Step 7: Send profiles from your applications

Point your SDKs or Alloy at https://<host>/ (the private IP or your DNS name, which the certificate names), with the user admin and the password from the credentials file. Copy the instance's certificate (/etc/nginx/tls/pyroscope.crt) to the senders so they can verify it, or install a certificate your systems already trust (Step 9).

Grafana Alloy (for example with pyroscope.ebpf or pyroscope.scrape feeding pyroscope.write.cloudimg.receiver):

pyroscope.write "cloudimg" {
  endpoint {
    url = "https://<host>"
    basic_auth {
      username      = "admin"
      password_file = "/etc/alloy/pyroscope.pass"
    }
    tls_config {
      ca_file = "/etc/alloy/pyroscope.crt"
    }
  }
}

Go with github.com/grafana/pyroscope-go:

pyroscope.Start(pyroscope.Config{
    ApplicationName:   "my-service",
    ServerAddress:     "https://<host>",
    BasicAuthUser:     "admin",
    BasicAuthPassword: os.Getenv("PYROSCOPE_PASSWORD"),
})

Java with the Pyroscope Java agent: set PYROSCOPE_SERVER_ADDRESS=https://<host>, PYROSCOPE_BASIC_AUTH_USER=admin and PYROSCOPE_BASIC_AUTH_PASSWORD in the application's environment, and add the instance's certificate to the JVM trust store.

Keep the password in a secrets store or a file only the sender can read, never in source code. The Pyroscope documentation covers every language SDK and the Alloy components. To look at the profiles from Grafana, add a Grafana Pyroscope data source with the URL https://<host>, basic authentication with the same user and password, and the instance's certificate as the CA certificate.

Step 8: Storage and retention

Pyroscope runs in single-binary mode: one process with every component, storing profiles on the local disk under /var/lib/pyroscope/data, with no separate database or object store. It keeps profiles for 31 days. The settings are command-line flags in /etc/systemd/system/pyroscope.service; change them with a drop-in so your change survives, for example a 14 day retention:

sudo systemctl edit pyroscope
# in the editor, add:
[Service]
ExecStart=
ExecStart=/usr/local/bin/pyroscope <all the flags from pyroscope.service> -retention-period=336h

then sudo systemctl restart pyroscope. Keep the 127.0.0.1 address flags as they are. For long retention or a scale-out cluster, Pyroscope can store profiles in Amazon S3 instead (-storage.backend=s3 with -storage.s3.bucket-name, -storage.s3.endpoint and -storage.s3.region); see the object storage documentation. pyroscope.service is blocked from the instance metadata service, so it cannot use an instance role: give it an access key for a bucket-only IAM user through a root-only environment file (EnvironmentFile= in the drop-in, with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY).

Step 9: Your own DNS name and certificate

Point a DNS record at the instance (an Elastic IP keeps the address stable), then set the name Pyroscope is served as. This mints a new self signed certificate for the name, updates nginx, the list of origins the web UI may post from and the URL in /root/grafana-pyroscope-credentials.txt, and keeps the name across reboots:

sudo pyroscope-set-hostname <your-domain>

To use a certificate from a public or internal certificate authority instead (for example one issued by Let's Encrypt for <your-domain>), copy it over the generated pair and reload nginx (running pyroscope-set-hostname again replaces it with a self signed one):

sudo install -m 0644 <your-domain>.crt /etc/nginx/tls/pyroscope.crt
sudo install -m 0600 <your-domain>.key /etc/nginx/tls/pyroscope.key
sudo nginx -t && sudo systemctl reload nginx

Step 10: Security checks

The shipped self-test checks the exact set of listening ports and that every Pyroscope port is on the loopback address, that an unknown host name is dropped, that requests without a password, with admin/admin or with a wrong password are refused by nginx while this instance's password is accepted, that a cross-site post is refused, and that the instance metadata service cannot be reached from inside the pyroscope and nginx services while it can from your shell. The password is handed to curl in a root only temporary file, never on a command line:

sudo /usr/local/sbin/pyroscope-selftest.sh

selftest OK: listening tcp lo:4040 lo:53 lo:7946 lo:9095 lo:9099 pub:22 pub:443 pub:80 | pyroscope listeners 127.0.0.1:4040 127.0.0.1:7946 127.0.0.1:9095 127.0.0.1:9099 (loopback only); healthz ok, unknown Host dropped (444); anonymous UI+ingest 401, admin/admin and wrong passwords 401 (nginx auth), per-instance password 200 (UI + query API), cross-origin POST 403; instance metadata blocked inside pyroscope and nginx (000), reachable from the host shell (200)

Add --ingest to also push a tiny profile for a service called cloudimg-selftest and read it back.

The metadata block is a systemd setting on each service:

systemctl show -p IPAddressDeny pyroscope nginx pyroscope-firstboot

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128

A request for a host name the instance does not serve gets no answer at all (nginx closes the connection, so curl exits with code 52):

curl -sk -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000
curl exit 52

The certificate was generated on this instance for its own addresses:

sudo openssl x509 -in /etc/nginx/tls/pyroscope.crt -noout -subject -ext subjectAltName

subject=CN = <public-ip>, O = cloudimg
X509v3 Subject Alternative Name: 
    IP Address:<public-ip>, DNS:<public-dns-name>, IP Address:<private-ip>, DNS:<private-dns-name>, IP Address:127.0.0.1, DNS:localhost

Step 11: Versions and files

/usr/local/bin/pyroscope --version 2>/dev/null | head -1
nginx -v 2>&1

pyroscope, version 2.2.3 (branch: HEAD, revision: a13c2d994)
nginx version: nginx/1.24.0 (Ubuntu)
Path What it is
/usr/local/bin/pyroscope The Pyroscope binary, run by pyroscope.service as the pyroscope user
/etc/systemd/system/pyroscope.service Pyroscope's settings (command-line flags): loopback addresses, storage, retention
/var/lib/pyroscope/data Profile storage (segments, blocks and the metadata store)
/etc/nginx/sites-available/cloudimg-pyroscope nginx site: HTTPS, basic authentication, proxy to 127.0.0.1:4040
/etc/nginx/pyroscope.htpasswd The basic authentication user and password hash (SHA-512 crypt)
/etc/nginx/tls/ This instance's certificate and private key
/root/grafana-pyroscope-credentials.txt URL, user and password (root only)
/usr/local/share/pyroscope/LICENSE The GNU Affero General Public License v3 that Pyroscope is released under

Step 12: Changing the password

Run these lines in your SSH session. The new password is read at a prompt and reaches openssl on standard input, so it never appears on a command line or in your shell history:

read -rsp 'New password: ' NEWPASS; echo
printf 'admin:%s\n' "$(printf '%s\n' "$NEWPASS" | openssl passwd -6 -stdin)" | sudo tee /etc/nginx/pyroscope.htpasswd >/dev/null
unset NEWPASS

nginx reads the file on every request, so the new password works at once; update it on every agent and SDK that sends profiles. Then update or delete PYROSCOPE_PASSWORD in /root/grafana-pyroscope-credentials.txt (sudo nano /root/grafana-pyroscope-credentials.txt). To give each team or agent its own login, append another user:hash line in the same format.

Step 13: Security recommendations

  • Keep port 443 open only to the networks that need the web UI and the agents that send profiles, and 22 to your admin network. Never open 4040, 9095, 7946 or 9099.
  • Use your own DNS name and a certificate from a certificate authority your agents trust (Step 9).
  • Keep the password in a secrets store and change it when people leave (Step 12). Profiles reveal function names and code structure, so treat them like source code.
  • The instance applies Ubuntu security updates automatically. To update Pyroscope itself, replace /usr/local/bin/pyroscope with a newer upstream release after checking its checksum against the release's checksums.txt, then restart the service; read the release notes first, and back up /var/lib/pyroscope/data.

Support and licensing

Grafana Pyroscope is open source software licensed under the GNU Affero General Public License v3.0 (AGPL-3.0-only); this image ships the unmodified upstream 2.2.3 release, whose source code is published at github.com/grafana/pyroscope. nginx is licensed under the 2-clause BSD license. cloudimg provides 24/7 support for this image by email at support@cloudimg.co.uk and through the cloudimg website: we help with launching, connecting SDKs and Alloy, reading flamegraphs, storage and retention, TLS with your own domain, upgrades and troubleshooting.

Grafana and Pyroscope are trademarks of Grafana Labs. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.