Observability AWS

Grafana Loki 3.7.1 on AWS User Guide

Last updated: 2026-10-19 | Product: Grafana Loki 3.7.1 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Grafana Loki 3.7.1 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg log management listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Loki is a log aggregation system. Agents such as Grafana Alloy, Fluent Bit or Vector push log lines with a few labels (job, host, app), Loki stores them compressed on disk and indexes only the labels, and you search them with LogQL from Grafana, logcli or plain HTTP. Because it does not index the full text, it is cheap to run and needs little memory.

An upstream Loki binary listens on all interfaces with no authentication. The cloudimg image changes that:

HTTPS from the first boot. nginx is the only network entry point and serves Loki on port 443 with a certificate generated on your instance's first boot. Port 80 only redirects to HTTPS.
A password unique to your instance. Every Loki path (push, query, tail, ready, metrics) needs HTTP Basic Auth as the user loki. The password is generated on first boot and written to the root only file /root/loki-credentials.txt; nginx stores only its sha512 hash. No password is baked into the image and no two instances share one.
Loki on loopback only. Loki's HTTP port 3100 and gRPC port 9096 are bound to 127.0.0.1. They are never reachable from the network, so never open them in a security group.
Fail closed. nginx refuses to start until first boot has written the certificate and the password, so Loki is never served without a login.
Hardened service. Loki runs as its own unprivileged user in a systemd sandbox (read only system, no capabilities, no new privileges) and may only talk to loopback, so it cannot reach the instance metadata service or any other address. Requests for a host name the instance does not serve are dropped, and cross site browser requests are refused.
Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Grafana Loki 3.7.1 (/usr/local/bin/loki, the unmodified upstream release binary, checksum verified, AGPL-3.0), single tenant filesystem storage (TSDB index, schema v13) in /var/lib/loki, nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent. Loki has no web interface of its own; use Grafana (Step 8) or the HTTP API to read your logs.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
A security group that allows TCP 22 (SSH) from your own networks, and TCP 443 (the Loki API) from the networks your log agents and Grafana run in. TCP 80 only redirects to HTTPS and is optional. If your agents run inside the same VPC, allow 443 from the VPC CIDR only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Grafana Loki 3.7.1 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above. Logs are stored on the root volume under /var/lib/loki, so give it room for the log volume you expect (Step 10).

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=loki}]'

Step 2: Connect to your instance

Version	SSH user
Grafana Loki 3.7.1 on Ubuntu 24.04	`ubuntu`

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot generates the certificate and the password, then nginx starts in front of Loki:

systemctl is-active loki nginx loki-firstboot
sudo test -f /var/lib/cloudimg/loki-firstboot.done && echo FIRSTBOOT_DONE
/usr/local/bin/loki -version | head -1

active
active
active
FIRSTBOOT_DONE
loki, version 3.7.1 (branch: release-3.7.x, revision: 2c8fff22)

Only SSH and nginx listen on public addresses; Loki listens on loopback:

sudo ss -Hlntp | awk '{print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:3100 users:(("loki"))
127.0.0.1:9096 users:(("loki"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))

Step 4: Get your endpoints and password

The endpoints are in the root only credentials file:

sudo grep -E '^LOKI_(URL|PUSH_URL|QUERY_URL|USER|CA_CERT)=' /root/loki-credentials.txt

LOKI_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/
LOKI_PUSH_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/push
LOKI_QUERY_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/loki/api/v1/query_range
LOKI_USER=loki
LOKI_CA_CERT=/etc/ssl/loki/loki.crt

The password is the LOKI_PASSWORD line of the same file: sudo grep '^LOKI_PASSWORD=' /root/loki-credentials.txt. Store it in your password manager or secrets store; it is unique to this instance.

The URL uses the instance's public DNS name from EC2 instance metadata (IMDSv2), or its public or private IPv4 address when it has no public name. The certificate names that host plus the instance's public and private IP addresses, so agents can use any of them:

openssl x509 -in /etc/ssl/loki/loki.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ec2-<public-ip-dashed>.compute-1.amazonaws.com, O = cloudimg
X509v3 Subject Alternative Name:
    DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost
notAfter=<date about 27 months after first boot>

Step 5: Check the login

Without the password every path is refused, a wrong password is refused, and port 80 redirects to HTTPS. With the password, /ready answers. The commands below read the password from the credentials file into a shell variable and pass it to curl through a file descriptor, so it never appears in the process list or your shell history:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt)
CA=/etc/ssl/loki/loki.crt
curl -s -o /dev/null -w 'http://  -> %{http_code} %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/ready"
curl -s -o /dev/null -w 'no login -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" "https://$H/ready"
curl -s -o /dev/null -w 'wrong password -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -u loki:not-the-password "https://$H/ready"
curl -s -w ' <- HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/ready"

http://  -> 301 https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ready
no login -> HTTP 401
wrong password -> HTTP 401
ready
 <- HTTP 200

--resolve sends the request to this instance while keeping the real host name, so the certificate check passes. From another machine, drop --resolve and use the URL directly.

Step 6: Push a log line and query it back

Loki's push API takes JSON streams: a set of labels and a list of [timestamp in nanoseconds, line] pairs. Push one line, then read it back with a LogQL query:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt)
CA=/etc/ssl/loki/loki.crt
TS=$(date +%s%N)
curl -s -o /dev/null -w 'push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" \
  -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Content-Type: application/json' \
  -d "{\"streams\":[{\"stream\":{\"job\":\"guide-test\",\"host\":\"$(hostname)\"},\"values\":[[\"$TS\",\"hello from the cloudimg Loki guide\"]]}]}" \
  "https://$H/loki/api/v1/push"
sleep 2
curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -G \
  --data-urlencode 'query={job="guide-test"}' --data-urlencode 'limit=5' \
  "https://$H/loki/api/v1/query_range" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["status"]); [print(s["stream"]["job"], v[1]) for s in d["data"]["result"] for v in s["values"]]'

push -> HTTP 204
success
guide-test hello from the cloudimg Loki guide

List the labels Loki has seen and the values of one label:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt)
CA=/etc/ssl/loki/loki.crt
curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/labels"; echo
curl -s --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") "https://$H/loki/api/v1/label/job/values"; echo

{"status":"success","data":["host","job","service_name"]}
{"status":"success","data":["guide-test"]}

Step 7: Send logs from your servers

Any agent that speaks the Loki push API works. Give it the push URL, the user loki, the password, and the instance's certificate so it can verify the connection. Copy the certificate to each agent host first (it is public, not a secret):

scp -i <your-key.pem> ubuntu@<loki-host>:/etc/ssl/loki/loki.crt ./loki.crt

Grafana Alloy (/etc/alloy/config.alloy on the agent host), shipping the systemd journal:

loki.source.journal "system" {
  forward_to = [loki.write.cloudimg.receiver]
  labels     = { job = "systemd-journal", host = constants.hostname }
}

loki.write "cloudimg" {
  endpoint {
    url = "https://<loki-host>/loki/api/v1/push"
    basic_auth {
      username = "loki"
      password = "<LOKI_PASSWORD>"
    }
    tls_config {
      ca_file = "/etc/alloy/loki.crt"
    }
  }
}

Fluent Bit ([OUTPUT] section):

[OUTPUT]
    Name        loki
    Match       *
    Host        <loki-host>
    Port        443
    tls         On
    tls.verify  On
    tls.ca_file /etc/fluent-bit/loki.crt
    http_user   loki
    http_passwd <LOKI_PASSWORD>
    labels      job=fluent-bit, host=${HOSTNAME}

Keep the password in the agent's own secret store or a root only file rather than in a world readable configuration. Agents send no browser Origin header, so the image's cross site request guard does not affect them.

Step 8: Read your logs in Grafana

Loki has no user interface of its own; Grafana is the usual way to explore it. In Grafana open Connections, Data sources, Add data source, Loki, and set:

URL: https://<loki-host>/ (the LOKI_URL from Step 4)
Authentication: Basic authentication, user loki, password LOKI_PASSWORD
TLS settings: turn on Add self-signed certificate and paste the contents of /etc/ssl/loki/loki.crt as the CA certificate (or install a CA issued certificate, Step 9)

Choose Save & test, then use Explore with a query such as {job="systemd-journal"} |= "error".

Step 9: Use your own DNS name or certificate

The served host name is fixed by configuration, never taken from a request. To serve Loki under your own DNS name, point the name at the instance and run (as root) the helper, which records the name, regenerates the self signed certificate for it, updates the credentials file and reloads nginx:

sudo loki-set-hostname <your-dns-name>

Run the same command after a stop and start if the instance's public DNS name changed (an Elastic IP avoids that), because a request for a name the instance does not serve is dropped.

To use a certificate from your own CA or a public CA instead, replace /etc/ssl/loki/loki.crt (full chain) and /etc/ssl/loki/loki.key (mode 0600, owner root), then run sudo nginx -t && sudo systemctl reload nginx. Alternatively, put the instance behind an Application Load Balancer with an AWS Certificate Manager certificate and let the load balancer forward to port 443.

Step 10: Storage and retention

Loki writes chunks, the write ahead log and the TSDB index under /var/lib/loki on the root volume:

sudo du -sh /var/lib/loki
df -h /var/lib/loki | tail -1 | awk '{print $2" total, "$4" free on "$6}'

84K /var/lib/loki
19G total, 15G free on /

By default Loki keeps logs until you delete them. To expire old logs automatically, add a retention period to /etc/loki/loki.yaml, for example 31 days, and restart Loki with sudo systemctl restart loki:

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

limits_config:
  retention_period: 744h

Grow the root EBS volume (or attach a dedicated volume mounted at /var/lib/loki) as your log volume grows. For larger or multi node deployments, Loki can also store chunks in Amazon S3; see the Loki storage documentation.

Step 11: Security notes

The Loki service is sandboxed and may only talk to loopback, so it cannot reach instance metadata or any other address:

systemctl show loki -p User -p IPAddressAllow -p IPAddressDeny -p ProtectSystem -p NoNewPrivileges

IPAddressAllow=127.0.0.0/8 ::1/128
IPAddressDeny=0.0.0.0/0 ::/0
User=loki
ProtectSystem=strict
NoNewPrivileges=yes

A state changing request carrying a foreign browser Origin is refused, even with the right password, so a web page cannot push or tail through a signed in browser:

H=$(sudo sed -n 's#^LOKI_URL=https://\([^/]*\)/.*#\1#p' /root/loki-credentials.txt)
PW=$(sudo sed -n 's/^LOKI_PASSWORD=//p' /root/loki-credentials.txt)
CA=/etc/ssl/loki/loki.crt
curl -s -o /dev/null -w 'Origin evil.example push -> HTTP %{http_code}\n' --cacert $CA --resolve "$H:443:127.0.0.1" -K <(printf 'user = "loki:%s"\n' "$PW") -H 'Origin: https://evil.example' -H 'Content-Type: application/json' -d '{"streams":[]}' "https://$H/loki/api/v1/push"

Origin evil.example push -> HTTP 403

Other points:

Loki runs single tenant (auth_enabled: false): anyone with the password can push and read every stream. Give each team its own instance if they must not see each other's logs.
nginx logs the real client address of every request; X-Forwarded-For headers sent by clients are ignored.
The AMI requires IMDSv2. Restrict port 443 to the networks your agents and Grafana use, and never open 3100 or 9096.
To rotate the password, generate a new one, write its hash to /etc/nginx/loki.htpasswd as loki:<hash> (openssl passwd -6), update /root/loki-credentials.txt, reload nginx and update your agents.

Support and licensing

Loki is open source under the GNU Affero General Public License v3.0 (AGPL-3.0); the image ships the unmodified upstream binary and there is no per user or per CPU licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

Email: support@cloudimg.co.uk
Website: https://www.cloudimg.co.uk

Loki is developed by Grafana Labs; for Loki features and LogQL, see the upstream Loki documentation.

Grafana Loki is a trademark of Grafana Labs. cloudimg is not affiliated with or endorsed by Grafana Labs; the name is used only to identify the unmodified upstream software.