Cloudprober 0.14.3 on AWS User Guide

Overview

This guide covers launching and using the Cloudprober 0.14.3 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg network operations and monitoring listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Cloudprober is an open source active monitoring tool. Instead of waiting for someone to report an outage, it probes your endpoints on a schedule over HTTP and HTTPS, DNS, TCP, PING, gRPC and external commands, and turns every result into availability and latency metrics in Prometheus format, with a live status page that graphs the success ratio of each probe. Everything it does is described in one readable configuration file.

What the cloudimg image adds:

• No default login. Cloudprober's own web server has no authentication, so this image never exposes it. nginx serves the status page and the metrics over HTTPS with HTTP basic authentication, and the password is generated on your instance's first boot and written to /root/cloudprober-credentials.txt (root only). The image ships no password, no password hash and no TLS key.
• Cloudprober on localhost only. Cloudprober listens on 127.0.0.1:9313; nginx on port 443 is the only way in. The unauthenticated /healthz path answers ok for load balancer health checks and reveals nothing else.
• HTTPS from the first boot. The certificate is generated on your instance for its own addresses, port 80 redirects to HTTPS, and requests for a host name the instance does not serve are dropped.
• Fail closed. nginx and Cloudprober start only after first boot has finished. If it cannot finish, nothing is served.
• No metadata access for probes. Cloudprober can probe any address you give it, so cloudprober.service, nginx.service and the first boot service are blocked from the instance metadata service, Cloudprober's own cloud metadata lookups are switched off, and the AMI requires IMDSv2.
• Honest client addresses. nginx hands Cloudprober a fixed host name and the real client address from the connection, never forwarding headers a client sends.
• Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Cloudprober 0.14.3 (the upstream release binary, checksum verified), nginx 1.24 and OpenSSL 3.0 (from Ubuntu 24.04), cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH) from your admin network and TCP 443 (status page and metrics) only from the people and Prometheus servers that need it. Port 80 only redirects to HTTPS, so you can leave it closed. Never open 9313: Cloudprober listens on the loopback address only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Cloudprober 0.14.3 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (a smaller instance is fine for a few dozen probes), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=cloudprober}]'

First boot takes a few seconds after the instance is running.

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot sets the password and the certificate, then nginx and Cloudprober start:

systemctl is-active cloudprober nginx cloudprober-firstboot
sudo test -f /var/lib/cloudimg/cloudprober-firstboot.done && echo FIRSTBOOT_DONE

All three units are active and the first boot marker is present:

active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on network ports; Cloudprober listens on the loopback address:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:9313 users:(("cloudprober"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

Step 4: Get your URL and password

sudo cat /root/cloudprober-credentials.txt

The file is readable by root only:

# Cloudprober 0.14.3 on Ubuntu 24.04 (cloudimg) - generated on this instance's first boot.
# This password is unique to this instance; none was baked into the image. Keep this file secret.

CLOUDPROBER_URL=https://<public-ip>/
CLOUDPROBER_USER=cloudimg
CLOUDPROBER_PASSWORD=<CLOUDPROBER_PASSWORD>

# Status page:  https://<host>/status    Metrics: https://<host>/metrics (Prometheus format)
# Health check: https://<host>/healthz   (no login; answers "ok")
# The certificate is self signed (minted on this instance): accept the browser warning once, or install
# your own. Serve a DNS name instead:  sudo cloudprober-set-hostname <your-dns-name>
# Cloudprober listens on 127.0.0.1:9313 only; nginx on 443 is the only way in.
# Probes live in /etc/cloudprober.cfg; apply changes with: sudo systemctl restart cloudprober
# Self-test: sudo /usr/local/sbin/cloudprober-selftest.sh
# User guide: https://www.cloudimg.co.uk/guides/cloudprober-0-14-3-on-ubuntu-24-04-aws/

The URL uses the instance's public IP address from the EC2 instance metadata. The certificate also names the public DNS name and the private IP address, so https://<public-dns-name>/ and https://<private-ip>/ work too. If you stop and start the instance without an Elastic IP, the boot service picks up the new address and issues a new certificate for it. To serve Cloudprober under your own DNS name, see Step 8.

Step 5: Open the status page

Browse to the URL from the credentials file. Your browser warns once about the self signed certificate; accept it (or install your own certificate, Step 8). Sign in as cloudimg with the password from the credentials file.

The status page shows every probe with its success ratio over the last minute, five minutes, half hour and longer, and a graph per probe. The links at the top lead to the running configuration, Cloudprober's recent log lines and the raw metrics. The image ships one probe, cloudimg_selfcheck, which checks Cloudprober's own status endpoint every five seconds so metrics flow from the first minute; the page below shows it alongside the example probes from Step 7.

A request without the password gets nginx's 401, and the health check answers without one:

curl -sk -o /dev/null -w '%{http_code}\n' https://127.0.0.1/metrics
curl -sk https://127.0.0.1/healthz

401
ok

Step 6: Scrape the metrics with Prometheus

Every probe exports total, success, failure, timeouts, latency (cumulative microseconds) and, for HTTP probes, resp_code counters. Read them on the instance:

curl -sk -u "cloudimg:$(sudo sed -n 's/^CLOUDPROBER_PASSWORD=//p' /root/cloudprober-credentials.txt)" https://127.0.0.1/metrics | grep 'probe="cloudimg_selfcheck"'

total{ptype="http",probe="cloudimg_selfcheck",dst="127.0.0.1"} 30
success{ptype="http",probe="cloudimg_selfcheck",dst="127.0.0.1"} 30
latency{ptype="http",probe="cloudimg_selfcheck",dst="127.0.0.1"} 29967.474
timeouts{ptype="http",probe="cloudimg_selfcheck",dst="127.0.0.1"} 0
resp_code{ptype="http",probe="cloudimg_selfcheck",dst="127.0.0.1",code="200"} 30
failure{ptype="http",probe="cloudimg_selfcheck",dst="127.0.0.1"} 0

To scrape it from a Prometheus server, add a job like this to prometheus.yml, with the password from the credentials file in a file only Prometheus can read. Use the address in the certificate (the private IP or your DNS name), and either trust the instance's certificate with ca_file or install your own certificate (Step 8):

scrape_configs:
  - job_name: cloudprober
    scheme: https
    basic_auth:
      username: cloudimg
      password_file: /etc/prometheus/cloudprober.pass
    tls_config:
      ca_file: /etc/prometheus/cloudprober.crt   # a copy of /etc/nginx/tls/cloudprober.crt
    static_configs:
      - targets: ['<private-ip>:443']

The success ratio of a probe over five minutes is rate(success[5m]) / rate(total[5m]), and its mean latency is rate(latency[5m]) / rate(success[5m]).

Step 7: Add probes for your own services

Probes live in /etc/cloudprober.cfg. Open it with sudo nano /etc/cloudprober.cfg and add a probe block per check, for example an HTTPS check of a website, a DNS check against the VPC resolver and a TCP check of a port:

probe {
  name: "website_https"
  type: HTTP
  targets {
    host_names: "www.cloudimg.co.uk"
  }
  http_probe {
    protocol: HTTPS
    relative_url: "/"
  }
  interval_msec: 15000
  timeout_msec: 5000
}

probe {
  name: "vpc_dns"
  type: DNS
  targets {
    host_names: "169.254.169.253"
  }
  dns_probe {
    resolved_domain: "aws.amazon.com."
  }
  interval_msec: 15000
  timeout_msec: 3000
}

probe {
  name: "ssh_port"
  type: TCP
  targets {
    host_names: "127.0.0.1"
  }
  tcp_probe {
    port: 22
  }
  interval_msec: 15000
  timeout_msec: 3000
}

Check the file and apply it. Cloudprober reads its configuration at start, so restart it:

sudo /opt/cloudprober/cloudprober --config_file=/etc/cloudprober.cfg --configtest && echo CONFIG_OK
sudo systemctl restart cloudprober

CONFIG_OK

The config link on the status page shows the configuration Cloudprober is running:

Cloudprober also has PING, gRPC, UDP and external (run a command) probe types, target discovery and many more options; see the Cloudprober documentation. The instance metadata service (169.254.169.254) is blocked for Cloudprober by design, so a probe aimed at it always fails. The VPC DNS resolver (169.254.169.253) and Amazon Time Sync (169.254.169.123) are not blocked.

Step 8: Your own DNS name and certificate

Point a DNS record at the instance (an Elastic IP keeps the address stable), then set the name Cloudprober is served as. This mints a new self signed certificate for the name, updates nginx and the URL in /root/cloudprober-credentials.txt, and keeps the name across reboots:

sudo cloudprober-set-hostname <your-domain>

To use a certificate from a public or internal certificate authority instead (for example one issued by Let's Encrypt for <your-domain>), copy it over the generated pair and reload nginx (running cloudprober-set-hostname again replaces it with a self signed one):

sudo install -m 0644 <your-domain>.crt /etc/nginx/tls/cloudprober.crt
sudo install -m 0600 <your-domain>.key /etc/nginx/tls/cloudprober.key
sudo nginx -t && sudo systemctl reload nginx

Step 9: Security checks

The shipped self-test checks the exact set of listening ports, that an unknown host name is dropped, that a request without a password and one with a wrong password are both refused by nginx while this instance's password is accepted, that the self-check probe is succeeding, and that the instance metadata service cannot be reached from inside the cloudprober and nginx services while it can from your shell. The password is handed to curl in a root only temporary file, never on a command line:

sudo /usr/local/sbin/cloudprober-selftest.sh

selftest OK: listening tcp lo:53 lo:9313 pub:22 pub:443 pub:80 | healthz ok, unknown Host dropped (444); anonymous 401 and wrong password 401 (nginx auth), per-instance password 200; selfcheck probe success=30; instance metadata blocked inside cloudprober and nginx (000), reachable from the host shell (200)

The metadata block is a systemd setting on each service:

systemctl show -p IPAddressDeny cloudprober nginx cloudprober-firstboot

IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32

IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32

IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32

A request for a host name the instance does not serve gets no answer at all (nginx closes the connection, so curl exits with code 52):

curl -sk -o /dev/null -w '%{http_code}\n' -H 'Host: attacker.example' https://127.0.0.1/healthz; echo "curl exit $?"

000
curl exit 52

The certificate was generated on this instance for its own addresses:

sudo openssl x509 -in /etc/nginx/tls/cloudprober.crt -noout -subject -ext subjectAltName

subject=CN = <public-ip>, O = cloudimg
X509v3 Subject Alternative Name: 
    IP Address:<public-ip>, DNS:<public-dns-name>, IP Address:<private-ip>, DNS:<private-dns-name>, IP Address:127.0.0.1, DNS:localhost

Step 10: Versions and files

/opt/cloudprober/cloudprober --version
nginx -v 2>&1

v0.14.3
nginx version: nginx/1.24.0 (Ubuntu)

Step 11: Changing the password

Run these lines in your SSH session. The new password is read at a prompt and reaches openssl on standard input, so it never appears on a command line or in your shell history:

read -rsp 'New password: ' NEWPASS; echo
printf 'cloudimg:%s\n' "$(printf '%s\n' "$NEWPASS" | openssl passwd -6 -stdin)" | sudo tee /etc/nginx/cloudprober.htpasswd >/dev/null
unset NEWPASS

nginx reads the file on every request, so the new password works at once. Then update or delete CLOUDPROBER_PASSWORD in /root/cloudprober-credentials.txt (sudo nano /root/cloudprober-credentials.txt). To add a second user, append another user:hash line in the same format.

Step 12: Security recommendations

• Keep port 443 open only to the networks that need the status page and metrics, and 22 to your admin network. Never open 9313.
• Use your own DNS name and a certificate from a certificate authority your Prometheus servers trust (Step 8).
• Keep the password in a secrets store and change it when people leave (Step 11).
• Probe targets you are allowed to probe, and keep probe intervals sensible for third party services.
• The instance applies Ubuntu security updates automatically. To update Cloudprober itself, replace /opt/cloudprober/cloudprober with a newer upstream release after checking its checksum, then restart the service.

Support and licensing

Cloudprober is open source software licensed under the Apache License 2.0. nginx is licensed under the 2-clause BSD license. cloudimg provides 24/7 support for this image by email at support@cloudimg.co.uk and through the cloudimg website: we help with launching, probe and target design, Prometheus and Grafana integration, TLS with your own domain, upgrades and troubleshooting.

Cloudprober is a trademark of its respective owner. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.