Observability AWS

Grafana CE with Prometheus on AWS User Guide

| Product: Grafana CE with Prometheus

Overview

Grafana is the open source standard for visualising time series data, and Prometheus is the industry standard engine for collecting and storing it. This cloudimg image ships both, already wired together, alongside node_exporter for detailed host metrics. Prometheus scrapes node_exporter and itself, Grafana has Prometheus set as its default datasource, and a starter host metrics dashboard is provisioned, so live charts render the first time you sign in rather than an empty datasource picker.

The appliance is secure by default. Grafana will not serve at all until first boot has generated a unique admin password and a fresh encryption secret for that specific instance, so no shared or default credential is ever baked into the image, and the well known default login is actively rejected. Prometheus and node_exporter bind to the loopback interface only, so neither is reachable from the network; you reach Prometheus data through Grafana, or over an SSH tunnel. The Prometheus time series database and Grafana's dashboard state each live on their own EBS volume, so retention and application data can be resized independently of the OS disk.

What is included:

  • Grafana Community Edition 12.4.6, installed from Grafana's official apt repository, serving the UI on TCP 3000
  • Prometheus 3.13.1 (/usr/local/bin/prometheus), bound to 127.0.0.1:9090, with its tsdb on a dedicated EBS volume at /var/lib/prometheus
  • node_exporter 1.12.1 (/usr/local/bin/node_exporter), bound to 127.0.0.1:9100, running as a dedicated non root user under a hardened systemd unit
  • A provisioned Prometheus datasource (set as default) and a starter "Node Metrics (cloudimg)" dashboard
  • grafana-firstboot.service, which rotates the admin password and the [security] secret_key to fresh per instance values before Grafana ever serves
  • Dedicated EBS volumes for /var/lib/prometheus and /var/lib/grafana
  • Ubuntu 24.04 LTS base, latest patches, unattended security upgrades enabled

Prerequisites

An active AWS account, an EC2 key pair, and a VPC with a subnet. Recommended instance type: m5.large. Prometheus memory use grows with the number of active time series, so scale up as you add scrape targets.

Step 1: Launch from the AWS Marketplace

Find Grafana CE with Prometheus on the AWS Marketplace, subscribe, and launch an instance. In the launch wizard, attach a security group that allows inbound TCP 22 (SSH) and TCP 3000 (the Grafana UI) from your client networks only. Prometheus and node_exporter are loopback bound and need no inbound rules of their own.

Step 2: Launch from the AWS CLI

AMI_ID="<marketplace-ami-id>"        # from the Marketplace listing, in your region
KEY_NAME="my-key"
SUBNET_ID="subnet-xxxxxxxx"
SG_ID="sg-xxxxxxxx"                  # allows 22 + 3000 from your networks only

aws ec2 run-instances \
  --image-id "$AMI_ID" \
  --instance-type m5.large \
  --key-name "$KEY_NAME" \
  --subnet-id "$SUBNET_ID" \
  --security-group-ids "$SG_ID" \
  --associate-public-ip-address \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=grafana-prometheus}]'

Connecting to your instance

Connect over SSH as the login user for your image's operating system. grafana-server.service, prometheus.service and node_exporter.service all start automatically on first boot.

Operating system SSH login user Connect
Ubuntu 24.04 LTS ubuntu ssh -i my-key.pem ubuntu@<instance-public-ip>

Step 3: Retrieve the Per-Instance Admin Password

First boot generates a random admin password for this instance alone and writes it to a root only file. Read it with:

sudo cat /stage/scripts/grafana-credentials.log

The file records the username, the generated password, and the URL to open:

# Grafana CE with Prometheus - Per-VM Credentials
#
ADMIN_USER=admin
ADMIN_PASSWORD=<ADMIN_PASSWORD>
GRAFANA_PORT=3000
GRAFANA_URL=http://<instance-public-ip>:3000

Step 4: Verify the Services

All three units should report active:

systemctl is-active grafana-server prometheus node_exporter

active
active
active

Confirm the versions that shipped in this image:

/usr/local/bin/prometheus --version 2>&1 | head -1
/usr/local/bin/node_exporter --version 2>&1 | head -1
dpkg-query -W -f='grafana ${Version}\n' grafana

prometheus, version 3.13.1 (branch: HEAD, revision: 73ff57ce2b8161059ac7fe5188f03f1c3d22b29a)
node_exporter, version 1.12.1 (branch: HEAD, revision: 6044da783597cc3b57aef7580ddcdcff58a4ee99)
grafana 12.4.6

Grafana's own health endpoint should report its database as ok:

curl -s http://127.0.0.1:3000/api/health

{
  "database": "ok",
  "version": "12.4.6",
  "commit": "16fbc24965a6dbeb7ed7facafb090062214ba33e"
}

Step 5: Confirm Prometheus and node_exporter Are Loopback Only

This is the security posture of the appliance: only the authenticated Grafana UI is exposed. Prometheus and node_exporter listen on 127.0.0.1 and are not reachable from the network.

sudo ss -tlnp | grep -E '3000|9090|9100' | sed 's/users:.*//'

LISTEN 0      4096       127.0.0.1:9090      0.0.0.0:*
LISTEN 0      4096       127.0.0.1:9100      0.0.0.0:*
LISTEN 0      4096               *:3000            *:*

Only port 3000 is bound to all interfaces. To reach the Prometheus expression browser from your workstation, use an SSH tunnel rather than opening the port:

ssh -i my-key.pem -L 9090:127.0.0.1:9090 ubuntu@<public-ip>

Then browse to http://localhost:9090 on your own machine.

Step 6: Verify Both Scrape Targets Are Up

Prometheus is pre configured to scrape node_exporter and itself. Both targets should report up:

curl -s 'http://127.0.0.1:9090/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | "\(.labels.job)\t\(.scrapeUrl)\t\(.health)"'

node    http://127.0.0.1:9100/metrics   up
prometheus  http://127.0.0.1:9090/metrics   up

Step 7: Confirm the Dedicated Data Volumes

The Prometheus tsdb and Grafana's state are each on their own EBS volume, mounted by filesystem UUID so the layout survives reboots and re launches:

df -h /var/lib/prometheus /var/lib/grafana

Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme1n1     49G  360K   47G   1% /var/lib/prometheus
/dev/nvme2n1     20G   49M   19G   1% /var/lib/grafana

Either volume can be grown in the EC2 console and extended with resize2fs without touching the OS disk.

Step 8: Sign In to Grafana

Open http://<instance-public-ip>:3000/ in your browser. Sign in as admin with the password from Step 3.

Grafana sign in page

Step 9: Open the Provisioned Dashboard

Go to Dashboards and open Node Metrics (cloudimg) in the cloudimg folder. It renders live host metrics immediately: CPU busy, memory used, root filesystem used, load average, CPU usage broken down by mode, and network throughput.

Node Metrics dashboard showing live host metrics

Step 10: Review the Prometheus Datasource

Go to Connections > Data sources > Prometheus. The datasource is already provisioned, set as the default, and pointed at the loopback Prometheus on http://127.0.0.1:9090.

Provisioned Prometheus datasource set as default

Step 11: Prove the Datasource Round-Trip

Grafana proxies queries to Prometheus on the server side. This runs a PromQL query through Grafana's datasource proxy and returns a live value, which proves the whole chain: your credential, Grafana's auth, the proxy, and Prometheus itself.

PW=$(sudo grep '^ADMIN_PASSWORD=' /stage/scripts/grafana-credentials.log | cut -d= -f2-)
curl -s -u "admin:$PW" 'http://127.0.0.1:3000/api/datasources/proxy/uid/cloudimg-prometheus/api/v1/query?query=node_load1' | jq -r '.data.result[0].value[1]'

A numeric load average is returned, for example:

0.89

Step 12: Confirm the Default Login Is Rejected

No shared credential ships in this image. The well known admin/admin default is rejected, while your per instance password is accepted:

PW=$(sudo grep '^ADMIN_PASSWORD=' /stage/scripts/grafana-credentials.log | cut -d= -f2-)
echo "default admin/admin -> $(curl -s -o /dev/null -w '%{http_code}' -u 'admin:admin' http://127.0.0.1:3000/api/user)"
echo "per-instance password -> $(curl -s -o /dev/null -w '%{http_code}' -u "admin:$PW" http://127.0.0.1:3000/api/user)"

default admin/admin -> 401
per-instance password -> 200

Step 13: Add Your Own Scrape Targets

Prometheus reads its configuration from /etc/prometheus/prometheus.yml. Add a job under scrape_configs for any application /metrics endpoint, another host's node_exporter, or any community exporter:

  - job_name: 'my-application'
    static_configs:
      - targets: ['10.0.1.25:8080']

Validate the file, then hot reload Prometheus by sending it SIGHUP. This re reads the configuration in place, so the tsdb stays open and no metrics are dropped:

sudo promtool check config /etc/prometheus/prometheus.yml
sudo systemctl kill -s HUP prometheus.service

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Confirm the reload landed in the journal:

sudo journalctl -u prometheus -n 3 --no-pager | grep -i "loading configuration\|Completed loading"

New targets appear in Grafana immediately and can be charted with PromQL.

Step 14: Managing the Services

sudo systemctl status grafana-server
sudo systemctl restart grafana-server
sudo journalctl -u grafana-server -n 50 --no-pager
sudo journalctl -u prometheus -n 50 --no-pager

Grafana's own log lives at /var/log/grafana/grafana.log. Prometheus and node_exporter log to the journal.

Step 15: Import a Community Dashboard

Grafana has a large library of ready made dashboards. In the UI choose Dashboards > New > Import, enter a dashboard ID from grafana.com, and select the Prometheus datasource when prompted. Dashboard 1860 (Node Exporter Full) is a comprehensive host metrics dashboard that works with the node_exporter in this image.

Step 16: Add TLS (Optional)

Grafana serves plain HTTP on port 3000 by default. For production, put your own domain and a TLS certificate in front of it. Either terminate TLS at an AWS Application Load Balancer and forward to port 3000, or install a reverse proxy on the instance and point it at 127.0.0.1:3000. If you terminate TLS upstream, set root_url in /etc/grafana/grafana.ini to your public HTTPS URL so generated links and redirects are correct.

Step 17: Security Recommendations

  • Restrict TCP 3000 and TCP 22 in your security group to known client networks only; never leave them open to the whole internet
  • Leave Prometheus and node_exporter loopback bound, and reach them over an SSH tunnel as in Step 5
  • Change the generated admin password to one your team manages, and add individual Grafana users rather than sharing the admin account
  • Put TLS in front of the UI before exposing it beyond a trusted network (Step 16)
  • Keep the OS patched; unattended security upgrades are enabled by default
  • Take EBS snapshots of the Prometheus and Grafana volumes on a schedule that matches your retention needs

Step 18: Support and Licensing

Grafana is a trademark of Raintank, Inc. dba Grafana Labs and is distributed under the GNU Affero General Public License v3.0. Prometheus and node_exporter are Cloud Native Computing Foundation projects distributed under the Apache License 2.0. All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.

Deploy on AWS

Find Grafana CE with Prometheus on the AWS Marketplace to launch this image in your own account.

Need Help?

cloudimg provides 24/7 technical support for this image, covering deployment, upgrades, adding scrape targets and exporters, dashboard and alert configuration, and fronting the UI with your own domain and TLS. Email support@cloudimg.co.uk with your instance ID and a description of the issue.