Observability AWS

GoAlert 0.35.0 on AWS User Guide

| Product: GoAlert 0.35.0 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using GoAlert 0.35.0 on Ubuntu 24.04 from cloudimg on Amazon EC2. It is one version of the cloudimg Hardened Uptime Kuma, Gatus & Uptime and Status Pages AMIs listing on AWS Marketplace; you pick it as the software version when you subscribe and launch. The other monitoring and status page products in the listing are described on the listing's guide hub.

GoAlert is an open source on-call platform originally built at Target. It keeps track of who is on call (rotations and schedules), decides who gets paged next (escalation policies with delays and repeats), and turns signals from your monitoring into alerts that are acknowledged, escalated and closed. Monitoring systems raise alerts through per service integration keys (a generic HTTP API, email, Grafana, Prometheus Alertmanager and Site24x7), and GoAlert notifies people by SMS and voice (Twilio), email (SMTP), Slack and webhooks.

What the cloudimg image adds:

  • HTTPS from the first boot. nginx serves GoAlert on port 443 with a certificate generated on your instance's first boot. Port 80 redirects to HTTPS, and requests for any host name the instance does not serve are dropped.
  • No default login. GoAlert itself ships without users; first boot creates a single admin account with a password unique to your instance, stored only as a bcrypt hash and in a root-only file.
  • A per-instance encryption key. GoAlert encrypts its stored secrets (provider tokens, signing keys) with a data encryption key that first boot generates for your instance alone and keeps in a root-only environment file.
  • A database with no password and no network port. First boot creates the PostgreSQL 16 cluster on your instance. PostgreSQL listens on no TCP port at all; GoAlert reaches it over the local socket as its own system user (peer authentication), so there is no database password to leak.
  • Fail closed. GoAlert and nginx refuse to start until first boot has finished, so the web interface is never reachable without its admin account and certificate.
  • Built from source, vulnerability scanned. The upstream 0.35.0 release binary was built with Go 1.26.6, whose standard library has known vulnerabilities since fixed. This image compiles the 0.35.0 source with Go 1.27.3 and patched Go modules; govulncheck reports no known vulnerability in the shipped binary (see Step 11).
  • Hardened defaults. GoAlert listens on loopback only and runs as an unprivileged user in a systemd sandbox; GoAlert, nginx and PostgreSQL are all blocked from the instance metadata service, and the AMI requires IMDSv2.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: GoAlert 0.35.0 (/usr/local/bin/goalert, compiled from source), PostgreSQL 16, nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows TCP 22 (SSH), TCP 443 (the web interface and the alert APIs) and TCP 80 (redirects to HTTPS) from your own networks and monitoring systems only.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick GoAlert 0.35.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=20,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=goalert}]'

First boot takes about a minute: it creates the database, the admin account, the encryption key and the certificate.

Step 2: Connect to your instance

Version SSH user
GoAlert 0.35.0 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

PostgreSQL, GoAlert and nginx are active, and the first boot and per boot host units have completed:

systemctl is-active postgresql@16-main goalert nginx goalert-firstboot goalert-configure
sudo test -f /var/lib/cloudimg/goalert-firstboot.done && echo FIRSTBOOT_DONE

active
active
active
active
active
FIRSTBOOT_DONE

Only SSH and nginx listen on public addresses. GoAlert listens on loopback, and PostgreSQL has no TCP listener at all:

sudo ss -lntp | awk 'NR>1 {print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx"))
127.0.0.1:8081 users:(("goalert"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"),("nginx"),("nginx"))

Step 4: Get your URL and admin password

sudo cat /root/goalert-credentials.txt

The file is readable by root only and holds the sign in URL, the user name and this instance's password:

# GoAlert - generated on first boot by goalert-firstboot.service
# These credentials are unique to this instance. Store them somewhere safe.

GOALERT_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/
GOALERT_ADMIN_USER=admin
GOALERT_ADMIN_PASSWORD=<GOALERT_ADMIN_PASSWORD>
#
# Open GOALERT_URL in a browser (self-signed certificate minted on this instance's first boot; port 80 redirects
# to https) and sign in with the username and password above. GoAlert only accepts sign-ins made from that URL.
# Serve it under your own DNS name:  sudo goalert-set-hostname <your-dns-name>
# Alert delivery (SMS, voice, email, Slack, webhooks) is configured under Admin -> Config after signing in.
# Full deployment guide: https://www.cloudimg.co.uk/guides/goalert-0-35-0-on-ubuntu-24-04-aws/

The URL uses the instance's public DNS name from EC2 instance metadata. GoAlert accepts a sign in only from the URL it is configured with (it checks the browser's Referer), so always open this URL rather than another name or address. The certificate also names the public and private IP addresses. If you stop and start the instance, the public name changes; at the next boot the image re-reads it from instance metadata and updates the URL, the certificate and /root/goalert-credentials.txt for you. To use your own DNS name instead, see Step 9.

Step 5: Sign in from your browser

Open the GOALERT_URL in your browser. The certificate is self signed, so your browser asks you to confirm it the first time (or install your own certificate, Step 9). Sign in as admin with the password from Step 4:

The GoAlert sign in page with Username and Password fields and a Login button

The Alerts page lists open alerts for the services you are on call for or have marked as favourites:

The GoAlert Alerts page showing one unacknowledged alert from the Web shop service: Checkout latency above 2s

Open an alert to acknowledge, escalate or close it, and to see which escalation step is paging whom and when the next escalation happens:

An alert detail page with Acknowledge, Escalate and Close buttons, the alert details and the escalation policy step that pages the admin user

A service ties an escalation policy to the integration keys your monitoring uses to raise alerts:

The Web shop service page with its escalation policy, the on call users for step 1 and quick links to alerts, heartbeat monitors and integration keys

Schedules combine rotations, rules and temporary overrides into a calendar of who is on call:

The Platform primary schedule in the Europe/London time zone with a month calendar and links to assignments, overrides and shifts

A typical first setup: Users (invite your team, or switch on GitHub or OIDC sign in under Admin), Rotations and Schedules (who is on call when), Escalation Policies (which schedule or person is paged first, and after how many minutes the next step follows), then Services with an integration key for each monitoring system. Each user adds their own contact methods and notification rules under their profile. The Wizard in the menu creates a service, escalation policy and schedule in one go.

Step 6: Raise a test alert from the command line

Without signing in, only the health check answers; port 80 redirects to HTTPS, GoAlert's internal health endpoints are not exposed, and other host names are dropped:

H=$(sudo grep '^GOALERT_URL=' /root/goalert-credentials.txt | sed -E 's#^GOALERT_URL=https://([^/]+)/.*#\1#')
curl -s -o /dev/null -w 'http://     -> %{http_code} %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/alerts"
echo "/healthz   -> $(curl -sk --resolve "$H:443:127.0.0.1" "https://$H/healthz")"
curl -sk -o /dev/null -w '/health    -> HTTP %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/health"
curl -sk -o /dev/null -w 'other Host -> %{http_code} (connection closed)\n' -H 'Host: example.com' https://127.0.0.1/ || true

http://     -> 301 https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/alerts
/healthz   -> ok
/health    -> HTTP 404
other Host -> 000 (connection closed)

The following signs in as admin, creates a Web shop service whose escalation policy pages you, gives it a generic API integration key, and raises an alert through that key exactly as a monitoring system would. The password goes to curl on standard input and the integration key travels in an Authorization: Bearer header, so neither appears in the process list, your shell history or the web server's access log:

sudo bash <<'EOF_GOALERT'
set -euo pipefail
F=/root/goalert-credentials.txt
P=$(grep '^GOALERT_ADMIN_PASSWORD=' "$F" | cut -d= -f2-)
H=$(grep '^GOALERT_URL=' "$F" | sed -E 's#^GOALERT_URL=https://([^/]+)/.*#\1#')
B="https://$H"; R=(--resolve "$H:443:127.0.0.1"); J=$(mktemp); trap 'rm -f "$J"' EXIT
# Sign in as admin: the form body goes to curl on stdin, so the password never appears in the process list
printf 'username=admin&password=%s' "$P" | curl -sk "${R[@]}" -o /dev/null -c "$J" -H "Referer: $B/" --data-binary @- "$B/api/v2/identity/providers/basic"
gql() { curl -sk "${R[@]}" -b "$J" -H "Referer: $B/" -H "Origin: $B" -H 'Content-Type: application/json' --data-binary @- "$B/api/graphql"; }
ME=$(printf '%s' '{"query":"{user{id name role}}"}' | gql | jq -r '.data.user.id')
[ -n "$ME" ] && [ "$ME" != null ] || { echo "sign-in failed"; exit 1; }
# A service whose escalation policy pages you, with a generic API integration key (reused if it already exists)
S=$(printf '%s' '{"query":"{services(input:{search:\"Web shop\"}){nodes{id name integrationKeys{id type}}}}"}' | gql)
SVC=$(jq -c '[.data.services.nodes[] | select(.name == "Web shop")][0] // empty' <<<"$S")
if [ -z "$SVC" ]; then
  Q=$(jq -nc --arg me "$ME" '{query:"mutation($in:CreateServiceInput!){createService(input:$in){id name integrationKeys{id type}}}",variables:{in:{name:"Web shop",newEscalationPolicy:{name:"Web shop on-call",steps:[{delayMinutes:5,targets:[{id:$me,type:"user"}]}]},newIntegrationKeys:[{type:"generic",name:"monitoring"}]}}}')
  SVC=$(printf '%s' "$Q" | gql | jq -c '.data.createService // empty')
fi
SID=$(jq -r '.id // empty' <<<"$SVC")
KEY=$(jq -r '[.integrationKeys[] | select(.type == "generic")][0].id // empty' <<<"$SVC")
[ -n "$SID" ] && [ -n "$KEY" ] || { echo "service creation failed: $SVC"; exit 1; }
echo "service: $(jq -r '.name' <<<"$SVC") (integration key type generic)"
# Raise an alert the way a monitoring system would: the key travels in an Authorization header
C=$(printf 'Authorization: Bearer %s\n' "$KEY" | curl -sk "${R[@]}" -o /dev/null -w '%{http_code}' -H @- -X POST "$B/api/v2/generic/incoming" \
    --data-urlencode 'summary=Checkout latency above 2s' --data-urlencode 'details=p95 2.4 s for 5 minutes on web-1')
echo "generic API: HTTP $C"
[ "$C" = 204 ] || [ "$C" = 200 ] || { echo "alert was not accepted"; exit 1; }
for i in $(seq 1 10); do
  A=$(jq -nc --arg s "$SID" '{query:"query($s:ID!){alerts(input:{filterByServiceID:[$s]}){nodes{alertID status summary}}}",variables:{s:$s}}' | gql)
  [ "$(jq '.data.alerts.nodes | length' <<<"$A")" -ge 1 ] && break; sleep 1
done
L=$(jq -r '.data.alerts.nodes[] | "alert #\(.alertID) \(.status): \(.summary)"' <<<"$A")
case "$L" in *"StatusUnacknowledged: Checkout latency above 2s"*) echo "$L" ;; *) echo "the alert is not open: $A"; exit 1 ;; esac
EOF_GOALERT

service: Web shop (integration key type generic)
generic API: HTTP 204
alert #1 StatusUnacknowledged: Checkout latency above 2s

The alert now shows on the Alerts page and pages whoever the escalation policy targets. From a monitoring system, send the same request to https://<your-goalert-host>/api/v2/generic/incoming with the integration key in an Authorization: Bearer <key> header; copy a service's key from Services > (service) > Integration Keys. GoAlert also accepts the key as a token query parameter, but a header keeps it out of web server logs. Add action=close with the same dedup value to close an alert automatically when the problem clears.

Step 7: Deliver notifications (SMS, voice, email, Slack)

Notification providers are switched off until you configure them under Admin > Config, with your own accounts: Twilio for SMS and voice calls, SMTP for email, Slack for Slack messages and Webhook for outgoing webhooks. Their tokens are stored in the database encrypted with this instance's data encryption key.

The GoAlert Admin Configuration page listing the General, Services, Maintenance and Auth sections and the GitHub, OIDC, Mailgun, Slack, Twilio, SMTP, Webhook and Feedback providers, each Disabled

Twilio and Slack call back into GoAlert (delivery status, SMS replies, button presses), so those features need the instance reachable from them at its configured URL with a certificate they trust: put GoAlert under your own DNS name with a trusted certificate (Step 9) and allow their traffic on port 443. Outgoing webhooks and the generic API work without that.

Step 8: Security checks

The environment file holds the database URL (a local socket, no password) and this instance's data encryption key; it is readable by root only and systemd reads it before starting GoAlert as its own user. The command prints the variable names only:

sudo stat -c '%a %U:%G %n' /etc/goalert/goalert.env /root/goalert-credentials.txt /etc/ssl/goalert/goalert.key
sudo grep -oE '^[A-Z_]+=' /etc/goalert/goalert.env
sudo grep -E '^GOALERT_(DB_URL|LISTEN)=' /etc/goalert/goalert.env

600 root:root /etc/goalert/goalert.env
600 root:root /root/goalert-credentials.txt
600 root:root /etc/ssl/goalert/goalert.key
GOALERT_DB_URL=
GOALERT_DATA_ENCRYPTION_KEY=
GOALERT_LISTEN=
GOALERT_DB_URL=postgres://goalert@/goalert?host=/var/run/postgresql
GOALERT_LISTEN=127.0.0.1:8081

PostgreSQL has no network listener, no role has a password, and local connections are allowed only for the matching operating system user:

sudo -u postgres psql -XtAc "SHOW listen_addresses" | sed 's/^$/(empty: no TCP listener)/'
sudo -u postgres psql -XtAc "SELECT count(*) || ' roles with a password' FROM pg_authid WHERE rolpassword IS NOT NULL"
sudo -u postgres psql -XtAc "SELECT type, database, user_name, auth_method FROM pg_hba_file_rules WHERE type = 'local'"

(empty: no TCP listener)
0 roles with a password
local|{all}|{all}|peer
local|{replication}|{all}|peer

GoAlert runs unprivileged in a sandbox with no capabilities, and GoAlert, nginx and PostgreSQL cannot reach the instance metadata service (GoAlert sends webhooks to addresses its users enter, so this matters):

systemctl show goalert -p User -p ProtectSystem -p NoNewPrivileges -p CapabilityBoundingSet
systemctl show goalert nginx postgresql@16-main -p Id -p IPAddressDeny

CapabilityBoundingSet=
User=goalert
ProtectSystem=strict
NoNewPrivileges=yes
IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
Id=goalert.service

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
Id=nginx.service

IPAddressDeny=169.254.169.254/32 168.63.129.16/32 fd00:ec2::254/128
Id=postgresql@16-main.service

nginx passes GoAlert the configured host name and the real client address from the connection, never values a client sends. It also refuses a state changing request or a WebSocket upgrade whose Origin is another web site (HTTP 403):

grep -E 'proxy_set_header (Host|X-Forwarded-For|X-Real-IP)' /etc/nginx/sites-available/cloudimg-goalert

        proxy_set_header Host $goalert_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;

The certificate was generated on this instance and names its own addresses, and SSH allows key based logins only:

H=$(sudo grep '^GOALERT_URL=' /root/goalert-credentials.txt | sed -E 's#^GOALERT_URL=https://([^/]+)/.*#\1#')
echo | openssl s_client -connect 127.0.0.1:443 -servername "$H" 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName
sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication|x11forwarding) '

subject=CN = ec2-<public-ip-dashed>.<region>.compute.amazonaws.com, O = cloudimg
X509v3 Subject Alternative Name: 
    DNS:ec2-<public-ip-dashed>.<region>.compute.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost
permitrootlogin no
passwordauthentication no
kbdinteractiveauthentication no
x11forwarding no

The session cookie GoAlert sets after sign in is Secure, HttpOnly and SameSite=Strict, and the sign in form is refused when it is posted from any page other than the configured URL.

Step 9: Your own DNS name and certificate

Point a DNS record at the instance, then set the name GoAlert is served as. This mints a new self signed certificate for the name, points GoAlert's public URL and every link it sends at it, updates nginx and /root/goalert-credentials.txt, and keeps the name across reboots and stop/start:

sudo goalert-set-hostname <your-domain>

To use a certificate from your own certificate authority (needed for Twilio and Slack callbacks), copy it over the generated pair and reload nginx (running goalert-set-hostname again replaces it with a self signed one):

sudo install -m 0644 <your-domain>.crt /etc/ssl/goalert/goalert.crt
sudo install -m 0600 <your-domain>.key /etc/ssl/goalert/goalert.key
sudo nginx -t && sudo systemctl reload nginx

Step 10: Backups, password and maintenance

Everything GoAlert knows lives in the goalert PostgreSQL database. Its stored secrets are encrypted with the key in /etc/goalert/goalert.env, so back up that file together with the database, and keep both somewhere safe:

sudo -u postgres pg_dump -Fc goalert | sudo tee <backup-dir>/goalert-$(date +%F).dump >/dev/null
sudo install -m 0600 /etc/goalert/goalert.env <backup-dir>/goalert-$(date +%F).env

To change the admin password, sign in, open Users > admin, choose Edit Access and enter the current and the new password. Afterwards update or delete GOALERT_ADMIN_PASSWORD in /root/goalert-credentials.txt so it does not hold an old password.

Logs go to the journal:

sudo journalctl -u goalert --no-pager -n 3 -o cat

time="2026-10-21T13:10:06Z" level=info msg="producer: Producer job counts" module=river num_completed_jobs=8 num_jobs_running=0 queue=status-manager

GoAlert reads its startup flags from GOALERT_ environment variables (a flag such as --listen becomes GOALERT_LISTEN; goalert --help lists the flags); add them to /etc/goalert/goalert.env and run sudo systemctl restart goalert. Everything else is configured in the web interface under Admin.

Step 11: Server components and build record

Component Path
GoAlert binary /usr/local/bin/goalert
Environment (database URL, encryption key) /etc/goalert/goalert.env (root only)
Public URL /etc/goalert/public-url.env (written by goalert-set-hostname)
Service unit /etc/systemd/system/goalert.service
Database PostgreSQL 16 cluster 16/main, /var/lib/postgresql/16/main, socket /var/run/postgresql
nginx site /etc/nginx/sites-available/cloudimg-goalert
Certificate and key /etc/ssl/goalert/
First boot and per boot host scripts /usr/local/sbin/goalert-firstboot.sh, /usr/local/sbin/goalert-configure.sh
Host name helper /usr/local/sbin/goalert-set-hostname
Credentials /root/goalert-credentials.txt (mode 0600)
Build record, licence, govulncheck reports /usr/share/doc/goalert/ 
/usr/local/bin/goalert version
sed -n '1,4p;/^binary sha256/p;/^govulncheck/,+2p' /usr/share/doc/goalert/BUILD-INFO.txt

Version:   v0.35.0
GitCommit: db9829187a1c1f96a7c57112b74dba54442d7c6b (clean)
BuildDate: 2026-09-15T21:24:20Z
GoVersion: go1.27.3 (gc)
Platform:  linux/amd64
Migration: cm-private (#281)
GoAlert v0.35.0 - cloudimg source build (d-320)
Licence: Apache-2.0 (LICENSE.md in this directory). The resulting go.mod and go.sum are in this directory.

source            https://github.com/target/goalert/archive/refs/tags/v0.35.0.tar.gz
binary sha256     389f2c5db4de3a3c457ee2b9570601f2758ccfa18e92a8686a83caeb5ea68206 (reproducible)
govulncheck       v1.9.0: -mode=binary "No vulnerabilities found"; source mode 0 symbol and 0 package
                  findings; module-level only GO-2026-5932 (golang.org/x/crypto/openpgp, no fixed version, not
                  linked: go list -deps ./cmd/goalert has no x/crypto/openpgp package) (DB updated: 2026-10-21 00:27:29 +0000 UTC)

Step 12: Security notes and recommendations

About this build. The upstream GoAlert 0.35.0 release binary was compiled with Go 1.26.6, and govulncheck reports known Go standard library vulnerabilities reachable in it (TLS, X.509, HTTP and HTML template handling, all fixed in later Go releases). This image compiles the same 0.35.0 source (tag commit db982918) with Go 1.27.3 and updated golang.org/x/net and golang.org/x/crypto modules. govulncheck finds no vulnerability in the shipped binary. The only entry it lists is GO-2026-5932 (golang.org/x/crypto/openpgp, deprecated, no fixed version), which is a package of a required module that GoAlert never imports, so it is not in the binary at all. The web interface is built from upstream's locked dependencies; the browser side npm packages carry advisories that are listed with their severities in /usr/share/doc/goalert/ui-npm-advisories.txt (denial of service and prototype pollution class issues reported for markdown-it, linkify-it, lodash, diff and mdast-util-to-hast, none critical). That code runs only in a signed in user's browser; no JavaScript runtime runs on the server.

GoAlert 0.35.0 adds multi-acknowledge escalation steps, private contact methods, an alerts view per escalation policy and new admin hardening options; every new behaviour is off until you enable it. One of them is worth considering: Admin > Config > Webhook > Block Private Addresses stops outgoing webhooks to private, loopback and link-local addresses (the instance metadata service is already blocked for GoAlert in this image).

Recommendations:

  • Allow ports 22, 80 and 443 in the security group from your own networks and monitoring systems only.
  • Keep the admin password secret, give every person their own user, and switch on GitHub or OIDC sign in under Admin > Config > Auth if you have an identity provider.
  • Use your own DNS name and a certificate your users and providers trust (Step 9).
  • Keep the operating system patched; unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.
  • Back up the database and the environment file together (Step 10), or snapshot the root volume.

Support and licensing

GoAlert is open source under the Apache License 2.0; there is no per user or per CPU licence fee. The AMI price covers cloudimg's source build, hardening, first boot setup and support.

  • Email: support@cloudimg.co.uk
  • Website: https://www.cloudimg.co.uk
  • Support: 24/7, with a 24 hour response SLA