Coroot Community Edition 1.29.2 on AWS User Guide

Overview

This guide covers launching and using the Coroot Community Edition 1.29.2 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg tracing and APM listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Coroot is an open source observability platform. Its node agent uses eBPF to watch every process on a host without any change to your applications: it records TCP connections, HTTP, database and other protocol requests with their latency and errors, logs, and CPU profiles. Coroot turns that into a live service map, an application list with health checks and SLOs, node dashboards, traces, logs and flamegraphs.

What the cloudimg image adds:

• A complete stack on one instance, already reporting. Coroot, Prometheus (metrics), ClickHouse (traces, logs and profiles) and the node agent are installed, connected and running when the instance boots. The instance monitors itself from the first minute, and you add more hosts with one command (Step 7).
• No default login and no shared secret. The Coroot admin password, the API key that agents and OpenTelemetry SDKs use, the ClickHouse password and the Prometheus password are all generated on your instance's first boot and written to /var/lib/cloudimg/coroot-credentials.txt (root only). The image ships no password, no API key and no TLS key.
• Everything on localhost, HTTPS in front. Coroot (8080), Prometheus (9090), ClickHouse (8123, 9000) and the node agent (10300) listen on the loopback address only. nginx on port 443 is the only way in, with a certificate generated on your instance. Port 80 only redirects to HTTPS, and requests for a host name the instance does not serve are dropped.
• A rate-limited login and guarded APIs. Sign-in is rate limited, every API call needs a session, the ingestion and Prometheus-compatible APIs refuse a missing or malformed API key at nginx, cross-site form posts and cross-site reads are refused, and Coroot's ClickHouse connection editor is closed so the web UI cannot be used to point the server at another address.
• A node agent with the least privilege it needs. The agent runs as root but its capability bounding set holds only the five capabilities its eBPF tracing needs (Step 9). The build proved each one: with any one removed, the agent loses a named function.
• No metadata access. Coroot, the node agent, Prometheus, ClickHouse and nginx are blocked from the instance metadata service, and the AMI requires IMDSv2. First boot hands the agent its region, zone and instance type instead.
• Nothing sent home. Coroot's usage statistics and update checks are off. The AI root cause analysis is a Coroot Cloud feature and stays off unless you connect it.
• Built from source and checked. Coroot and the node agent are built from their tagged upstream source with patched dependencies, and every shipped binary was scanned for known vulnerabilities. Ubuntu 24.04 LTS is fully patched at build time, with unattended security upgrades switched on.

What is on the image: Coroot Community Edition 1.29.2 (Apache-2.0, no Enterprise code), coroot-node-agent 1.38.1, Prometheus 3.16.0, ClickHouse 26.8 LTS, nginx 1.24 and OpenSSL 3.0 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH) from your admin network and TCP 443 (web UI, node agents and OpenTelemetry) only from the people and hosts that use it. Port 80 only redirects to HTTPS, so you can leave it closed. Never open 8080, 9090, 8123, 9000 or 10300: they listen on the loopback address only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Coroot Community Edition 1.29.2 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger as you add hosts), your key pair and a security group as above. Metrics, traces, logs and profiles are stored on the root volume, so give it room: 50 GiB is a good start for a handful of hosts.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=coroot}]'

First boot takes about half a minute after the instance is running; the first data shows in Coroot a minute or two later.

Step 2: Connect to your instance

Version	SSH user
Coroot Community Edition 1.29.2 on Ubuntu 24.04	ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

The login message shows where the credentials are and how to run the self-test.

Step 3: Check the services

First boot sets the passwords, the API key and the certificate, then ClickHouse, Prometheus, Coroot, the node agent and nginx start:

systemctl is-active clickhouse-server prometheus coroot coroot-node-agent nginx coroot-firstboot
sudo test -f /var/lib/cloudimg/coroot-firstboot.done && echo FIRSTBOOT_DONE

All six units are active and the first boot marker is present:

active
active
active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network ports; Coroot, Prometheus, ClickHouse and the node agent are on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:10300 users:(("coroot-node-age"))
127.0.0.1:8080 users:(("coroot"))
127.0.0.1:8123 users:(("clickhouse-serv"))
127.0.0.1:9000 users:(("clickhouse-serv"))
127.0.0.1:9090 users:(("prometheus"))
[::1]:8123 users:(("clickhouse-serv"))
[::1]:9000 users:(("clickhouse-serv"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

(The number of nginx entries follows the number of vCPUs.)

Step 4: Get your URL, login and API key

sudo cat /var/lib/cloudimg/coroot-credentials.txt

The file is readable by root only:

# Coroot Community Edition 1.29.2 - generated on first boot by coroot-firstboot.service
# These credentials are unique to this instance; none was baked into the image. Keep this file secret.

COROOT_URL=https://<public-dns-name>/
COROOT_USER=admin
COROOT_PASSWORD=<COROOT_PASSWORD>
COROOT_API_KEY=<COROOT_API_KEY>
CLICKHOUSE_USER=default
CLICKHOUSE_PASSWORD=<CLICKHOUSE_PASSWORD>
PROMETHEUS_USER=coroot
PROMETHEUS_PASSWORD=<PROMETHEUS_PASSWORD>
#
# Open COROOT_URL in a browser (self-signed certificate minted on this instance's first boot; port 80 redirects to
# https) and sign in as COROOT_USER / COROOT_PASSWORD (change it under your user menu). This host's node agent already
# reports with COROOT_API_KEY; point node agents on other hosts at https://<public-dns-name>/ with the same key.
# ClickHouse (127.0.0.1:8123/9000) and Prometheus (127.0.0.1:9090) listen on loopback only; their passwords are here
# and in /etc/coroot/coroot.yaml.
# Self-test:  sudo coroot-selftest      Change the served name:  sudo coroot-set-hostname <your-domain>
# User guide: https://www.cloudimg.co.uk/guides/coroot-community-edition-1-29-2-on-ubuntu-24-04-aws/

• COROOT_PASSWORD is your sign-in password for the web UI.
• COROOT_API_KEY is the key node agents and OpenTelemetry SDKs send with their data, and the key for Coroot's Prometheus-compatible API.
• You do not need the ClickHouse and Prometheus passwords for normal use: Coroot uses them to reach its own databases.

Coroot is served under one name: the instance's public DNS name at first boot (its public or private IP address when it has no DNS name). The certificate also names the IP addresses, and a request to an IP address is redirected to the name. If the name changes (for example you stop and start the instance without an Elastic IP), or you want your own DNS name, set it with coroot-set-hostname (Step 11).

The health check answers without a login, and the instance's own address redirects to the served name:

curl -sk https://localhost/healthz
curl -sk -o /dev/null -w '%{http_code}\n' https://localhost/

ok
301

Step 5: Sign in to the web UI

Open COROOT_URL in your browser. The certificate is self signed (generated on your instance), so accept the browser warning once, or install your own certificate (Step 11). Sign in with admin in the Email field and COROOT_PASSWORD:

Coroot opens the Applications list of the default project. The node agent on the instance has already found the services running on it (the screenshots in this guide were taken with the demo from Step 6 running). The Service Map shows which services talk to which, with the traffic between them:

Nodes lists every host that reports to Coroot, starting with this one, with its CPU, memory, network, availability zone and instance type:

Change the admin password from the user menu (bottom left, Admin > Change password), and add a user for each person under Settings > Organization.

Step 6: See an application appear

To watch Coroot pick up an application with no instrumentation, start a small web server and a client that calls it once a second (both as throwaway systemd services), and give the agent a minute and a half to report:

sudo systemd-run --quiet --unit=demo-web -p DynamicUser=yes python3 -m http.server 8000 --bind 127.0.0.1 --directory /usr/share/doc/cloudimg-coroot
sudo systemd-run --quiet --unit=demo-client -p DynamicUser=yes /bin/sh -c 'while :; do curl -s -o /dev/null http://127.0.0.1:8000/; curl -s -o /dev/null http://127.0.0.1:8000/missing; sleep 1; done'
sleep 90

demo-web and demo-client now show in Applications and on the Service Map; open demo-client to see the requests it makes, their latency and the share that fail (the /missing page answers 404):

The same data is available from Coroot's Prometheus-compatible API, which takes the API key in the X-API-Key header. The key goes to curl in a file readable only by you, so it never appears on a command line:

( umask 077; printf 'header = "X-API-Key: %s"\n' "$(sudo sed -n 's/^COROOT_API_KEY=//p' /var/lib/cloudimg/coroot-credentials.txt)" > ~/.coroot-api-key.curl )
HOST="$(sudo sed -n 's#^COROOT_URL=https://\([^/]*\)/$#\1#p' /var/lib/cloudimg/coroot-credentials.txt)"
curl -sk -K ~/.coroot-api-key.curl --resolve "${HOST}:443:127.0.0.1" -G "https://${HOST}/api/v1/query_range" \
  --data-urlencode 'query=sum by (status) (rate(container_http_requests_total{destination="127.0.0.1:8000"}[1m]))' \
  --data-urlencode "start=$(( $(date +%s) - 120 ))" --data-urlencode "end=$(date +%s)" --data-urlencode step=60 \
  | grep -o '"status":"[0-9]xx"'

"status":"2xx"
"status":"4xx"

Without the key, nginx refuses the request:

HOST="$(sudo sed -n 's#^COROOT_URL=https://\([^/]*\)/$#\1#p' /var/lib/cloudimg/coroot-credentials.txt)"
curl -sk -o /dev/null -w '%{http_code}\n' --resolve "${HOST}:443:127.0.0.1" "https://${HOST}/api/v1/series"

401

Stop the demo when you are done (Coroot keeps its history until it ages out):

sudo systemctl stop demo-web demo-client
rm -f ~/.coroot-api-key.curl

Step 7: Monitor your other hosts

Every host you want to see in Coroot runs the node agent and sends its data to this instance over HTTPS with the API key. On each host:

1. Allow it to reach this instance on TCP 443 in the security group.
2. Make it trust this instance's certificate. The certificate is its own certificate authority, so copy /etc/ssl/coroot/coroot.crt from this instance to the host and add it to the host's trust store (on Ubuntu or Debian):

sudo install -m 0644 coroot.crt /usr/local/share/ca-certificates/coroot.crt
sudo update-ca-certificates

1. Install the agent with Coroot's installer, pointing it at COROOT_URL with COROOT_API_KEY (pin the version this instance runs):

curl -sfL https://raw.githubusercontent.com/coroot/coroot-node-agent/main/install.sh \
  | COLLECTOR_ENDPOINT=https://<public-dns-name> API_KEY=<COROOT_API_KEY> sh -s -- --version 1.38.1

The host appears under Nodes a minute later, and its services join the application list and the service map. (The Add nodes button on the Nodes page shows the same commands, filled in with this instance's address and key.) For Kubernetes, Docker and other installs see the Coroot documentation; use the same endpoint and key. If you install a certificate from a public certificate authority (Step 11), step 2 is not needed.

Step 8: Send OpenTelemetry traces and logs

Applications instrumented with OpenTelemetry can send traces and logs to Coroot over OTLP/HTTP on the same address. Set, for example:

OTEL_EXPORTER_OTLP_ENDPOINT=https://<public-dns-name>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=x-api-key=<COROOT_API_KEY>

The SDK posts to /v1/traces and /v1/logs. nginx refuses any request to these paths without a well-formed key, and Coroot refuses a wrong one.

Step 9: Node agent privileges

The node agent loads eBPF programs into the kernel and reads other processes' network and file information, so it runs as root. Its systemd unit limits it to the capabilities below and nothing else: no ambient capabilities, no new privileges (NoNewPrivileges=yes), a read-only system (ProtectSystem=strict) except its own write-ahead log directory, no kernel module loading and no access to the instance metadata service. When the image was built, the agent was run in this sandbox with each capability removed in turn and the named function stopped working every time:

Capability	Why the agent needs it	Without it
CAP_SYS_ADMIN	Load the eBPF programs and maps (on this kernel it also covers CAP_BPF and CAP_PERFMON), and enter the host's namespaces	the agent cannot start: failed to get uname: operation not permitted
CAP_SYS_PTRACE	Read /proc/<pid>/ns, /proc/<pid>/fd and the memory maps of other users' processes to tie connections and files to services	the agent cannot start: failed to get uname: open /proc/1/ns/uts: permission denied
CAP_DAC_READ_SEARCH	Open the log files that services owned by other users write in their own private directories	those services' logs are not collected
CAP_NET_ADMIN	Read the kernel's connection tracking table over netlink, to see the real destination behind NAT and load balancer addresses	netlink receive: operation not permitted
CAP_NET_RAW	Open an ICMP socket to measure the network round trip time to the hosts your services talk to	failed to open IPConn: listen ip4:icmp 0.0.0.0: socket: operation not permitted

Check the running agent's sets:

grep -E '^Cap(Eff|Bnd|Amb)|^NoNewPrivs' /proc/$(systemctl show -p MainPID --value coroot-node-agent)/status
capsh --decode=0000000000283004

CapEff: 0000000000283004
CapBnd: 0000000000283004
CapAmb: 0000000000000000
NoNewPrivs: 1
0x0000000000283004=cap_dac_read_search,cap_net_admin,cap_net_raw,cap_sys_ptrace,cap_sys_admin

Coroot and Prometheus run as their own users with no capabilities at all; ClickHouse runs as clickhouse with only CAP_IPC_LOCK and CAP_SYS_NICE:

systemctl show -p User -p CapabilityBoundingSet -p NoNewPrivileges coroot prometheus clickhouse-server

CapabilityBoundingSet=
User=coroot
NoNewPrivileges=yes

CapabilityBoundingSet=
User=prometheus
NoNewPrivileges=yes

CapabilityBoundingSet=cap_ipc_lock cap_sys_nice
User=clickhouse
NoNewPrivileges=yes

The agent normally asks the instance metadata service which region, zone and instance type it runs in. Here that request is blocked, and first boot writes those values to /etc/coroot-node-agent/agent.env instead, so the node still shows its availability zone and instance type in Coroot.

Step 10: Storage and retention

Data	Where	Kept for
Metrics	Prometheus, /var/lib/prometheus	7 days (--storage.tsdb.retention.time=7d in /etc/systemd/system/prometheus.service)
Traces, logs and profiles	ClickHouse, /var/lib/clickhouse	7 days (Coroot's default; set ttl under traces:, logs: and profiles: in /etc/coroot/coroot.yaml and restart coroot to change it)
Projects, users, settings and incidents	Coroot, /var/lib/coroot	until you delete them

To keep metrics longer, raise the retention and restart Prometheus, and grow the root volume to match:

sudo sed -i 's/--storage.tsdb.retention.time=7d/--storage.tsdb.retention.time=30d/' /etc/systemd/system/prometheus.service
sudo systemctl daemon-reload && sudo systemctl restart prometheus

Step 11: Your own DNS name and certificate

Point a DNS record at the instance (an Elastic IP keeps the address stable), then set the name Coroot is served as. This mints a new self signed certificate for the name, updates nginx and the cross-site and Origin guards, updates COROOT_URL in the credentials file, and keeps the name across reboots:

sudo coroot-set-hostname <your-domain>

Run the same command with the new public DNS name if the instance's address changes and you have no DNS name of your own. Node agents on other hosts need the new certificate (Step 7) and the new endpoint.

To use a certificate from a public or internal certificate authority instead (for example one issued by Let's Encrypt for <your-domain>), copy it over the generated pair and reload nginx (running coroot-set-hostname again replaces it with a self signed one):

sudo install -m 0644 <your-domain>.crt /etc/ssl/coroot/coroot.crt
sudo install -m 0600 <your-domain>.key /etc/ssl/coroot/coroot.key
sudo nginx -t && sudo systemctl reload nginx

Step 12: Security checks

The shipped self-test checks the HTTPS redirect and the health check, that every Coroot API call needs a session and that a wrong password is refused while the admin password works, that the ingestion API refuses a missing or all-zero key and accepts this instance's key, that the ClickHouse connection editor is closed, that the node metrics reach Prometheus and the traces reach ClickHouse (and that both refuse a request without their password), that cross-site requests and unknown host names are refused, that Coroot, Prometheus, ClickHouse and the agent listen on the loopback address only, and that the instance metadata service cannot be reached from inside any of the five services while it can from your shell. Passwords are handed to curl in root-only temporary files, never on a command line:

sudo coroot-selftest

SELFTEST-OK https 301 + healthz + UI shell with pinned CSP + /api/user 401 anonymous, 404 wrong password, set_admin_password 401, admin login 200 (Secure HttpOnly cookie) + project <project-id> status ok (prometheus, node agent) + /v1 API key 401 none/zero, 200 instance key + clickhouse-config/connect refused + node metrics in Prometheus, Coroot tables in ClickHouse (both refuse without password) + 403 cross-origin POST + 403 cross-site /stats + 444 unknown host + loopback-only coroot/prometheus/clickhouse/agent + metadata 000 inside 5 services (control 200)

The metadata block is a systemd setting on each service (the first boot service is the one exception: it reads the instance's address, region and zone once, before the others start):

systemctl show -p IPAddressDeny clickhouse-server prometheus coroot coroot-node-agent nginx

IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32

IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32

IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32

IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32

IPAddressDeny=168.63.129.16/32 fd00:ec2::254/128 169.254.169.254/32

A request for a host name the instance does not serve gets no answer at all (nginx closes the connection, so curl exits with code 52):

curl -sk --http1.1 -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000
curl exit 52

The certificate was generated on this instance for its own names and addresses:

sudo openssl x509 -in /etc/ssl/coroot/coroot.crt -noout -subject -ext subjectAltName

subject=CN = <public-dns-name>, O = cloudimg
X509v3 Subject Alternative Name: 
    DNS:<public-dns-name>, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost

Step 13: Versions and files

head -1 /usr/share/doc/cloudimg-coroot/BUILD-INFO.txt
/usr/local/bin/coroot-node-agent --version
/usr/local/bin/prometheus --version 2>&1 | head -1
clickhouse-server --version
nginx -v 2>&1

Coroot CE         1.29.2 (tag v1.29.2, commit b5e20cc1d1fe8d1ef8bfeecc5c8cc53348ce859d), Apache-2.0, edition Community
Version: 1.38.1
prometheus, version 3.16.0 (branch: HEAD, revision: e2e28adeee2c5883d000fc82aca6008cfbf7e96d)
ClickHouse server version 26.8.34.13 (official build).
nginx version: nginx/1.24.0 (Ubuntu)

Path	What it is
/usr/local/bin/coroot	Coroot (server and web UI), run by coroot.service as the coroot user
/etc/coroot/coroot.yaml	Coroot's settings: loopback listener, the Prometheus and ClickHouse connections, the project and its API key
/usr/local/bin/coroot-node-agent	The node agent, run by coroot-node-agent.service
/etc/coroot-node-agent/agent.env	The agent's endpoint, API key, region, zone and instance type (root only)
/usr/local/bin/prometheus	Prometheus, run by prometheus.service as the prometheus user; settings in /etc/prometheus/
/etc/clickhouse-server/config.d/cloudimg.xml	ClickHouse settings: loopback listeners
/etc/nginx/sites-available/cloudimg-coroot	nginx site: HTTPS, the API key check, the login rate limit, proxy to 127.0.0.1:8080
/etc/ssl/coroot/	This instance's certificate and private key
/var/lib/cloudimg/coroot-credentials.txt	URL, logins, passwords and API key (root only)
/usr/share/doc/cloudimg-coroot/	Build information, checksums, the licenses, the Go module lists and the agent's eBPF C sources

Step 14: Changing passwords and the API key

• Admin password: sign in and use the user menu (Admin > Change password). Then update or delete COROOT_PASSWORD in /var/lib/cloudimg/coroot-credentials.txt (sudo nano /var/lib/cloudimg/coroot-credentials.txt).
• API key: the project's keys are read from /etc/coroot/coroot.yaml (apiKeys under projects). Add a new key there, restart Coroot (sudo systemctl restart coroot), move every agent and SDK to the new key (this host's agent reads it from /etc/coroot-node-agent/agent.env; restart coroot-node-agent after editing), then remove the old key and restart Coroot again.

Step 15: Security recommendations

• Keep port 443 open only to the people and the hosts that send data, and 22 to your admin network. Never open 8080, 9090, 8123, 9000 or 10300.
• Use your own DNS name and a certificate from a certificate authority your agents trust (Step 11).
• Keep the credentials file's values in a secrets store, give each person their own Coroot user, and change the API key when a host that held it is retired (Step 14). Logs and traces can contain personal or sensitive data, so treat the instance like the systems it watches.
• The instance applies Ubuntu security updates automatically. Upgrade Coroot, the agent and Prometheus by replacing their binaries with newer upstream releases after reading the release notes; back up /var/lib/coroot first.

Support and licensing

Coroot Community Edition and coroot-node-agent are open source software licensed under the Apache License 2.0 (the agent's eBPF programs are licensed under the GPL, and their C sources are on the image); this image builds the unmodified upstream Coroot v1.29.2 and coroot-node-agent v1.38.1 sources and contains no Coroot Enterprise Edition code. Prometheus and ClickHouse are licensed under the Apache License 2.0, and nginx under the 2-clause BSD license. cloudimg provides 24/7 support for this image by email at support@cloudimg.co.uk and through the cloudimg website: we help with launching, adding hosts and Kubernetes clusters, OpenTelemetry, storage and retention, TLS with your own domain, upgrades and troubleshooting.

Coroot is a trademark of Coroot, Inc. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.