Streaming & Messaging AWS

webhookd 1.22.1 Webhook Server on AWS User Guide

| Product: webhookd 1.22.1 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the webhookd 1.22.1 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe.

webhookd is an open source (MIT) webhook server written in Go. Each executable script in its scripts directory becomes an HTTP endpoint: a request to https://<host>/deploy runs deploy.sh, and the script's output is streamed back to the caller as it is printed. It is used to trigger deployments, builds, backups, chat-ops commands and other small automation from CI systems, Git hosting webhooks, monitoring alerts or curl.

What the cloudimg image adds:

  • HTTPS only, on 443. nginx serves TLS 1.3 and TLS 1.2 (AEAD cipher suites only) on port 443 and passes requests to webhookd, which listens on 127.0.0.1:8080 only. Nothing serves plain HTTP: port 80 is closed, and plain HTTP sent to 443 is refused (Step 11).
  • A certificate made on your instance. At first boot the instance gets a TLS key and an end-entity certificate for its own names and addresses. It can only vouch for this server and can never sign other certificates. You pin it in your clients (Step 5), or replace it with one from your own CA or Let's Encrypt (Step 13).
  • A login on every request, generated for your instance. webhookd itself serves everyone when its password file is missing or unreadable. Here every request needs a login (only /healthz answers without one). First boot generates the login cloudimg with a 128-bit random password, stored only as a bcrypt hash. Before every start a gate refuses a missing, empty or non-bcrypt password file, and after every start a guard stops a webhookd that answers an anonymous or wrong-password request (Step 11).
  • Hooks that webhookd cannot change. The scripts directory /opt/webhookd/scripts and every hook in it are owned by root. webhookd runs as the unprivileged user webhookd, which can run the hooks but cannot write, replace or add one. The directory is mounted read-only with nosymfollow, so a symbolic link is never followed (Step 8).
  • A sandboxed hook user. Hooks run as webhookd with no capabilities, NoNewPrivileges (a hook cannot use sudo), a read-only system, a private /tmp and no access to home directories. Each hook may run for 10 seconds by default.
  • No instance metadata and no route into your network. webhookd and nginx run with the EC2 instance metadata addresses blocked. Hooks may reach the internet (to call an API or fetch a release), but a firewall table refuses every connection from the hook user to your VPC, the private address ranges and link-local addresses, unless you open a specific range (Step 12).
  • Browsers cannot fire your hooks. A request with a browser Origin header or a cross-site Sec-Fetch-Site header is refused, so a web page cannot use a login a browser remembered.
  • Rate and size limits. nginx allows 5 requests per second per client address (bursts of 10) and 10 connections per address, and refuses a body over 1 MB.
  • A pinned supply chain. The webhookd binary comes from the webhookd project's GitHub release v1.22.1 (webhookd-linux-amd64.tgz), checked against its SHA-256 digest, with every shipped file recorded in /usr/share/doc/cloudimg-webhookd/PROVENANCE. webhookd has no telemetry. Ubuntu 24.04 LTS is fully patched at build time.

What is on the image: webhookd 1.22.1 (/usr/local/bin/webhookd; configuration in /etc/webhookd/webhookd.env, logins in /etc/webhookd/htpasswd, hooks in /opt/webhookd/scripts, hook logs in /var/lib/webhookd), nginx from Ubuntu 24.04 as the TLS front, htpasswd (apache2-utils) for logins, the example hook echo.sh, and the cloudimg helpers webhookd-selfcheck, webhookd-config-gate, webhookd-egress-apply, webhookd-set-hostname and webhookd-install-cert.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows:
Port Protocol Source Purpose
22 TCP your admin network SSH
443 TCP the networks your callers (CI, Git hosting, monitoring) connect from Hooks over HTTPS

Nothing else needs to be open. webhookd's own port 8080 is bound to 127.0.0.1 and port 80 is not served, so opening them achieves nothing (Step 11).

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick webhookd 1.22.1 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (a smaller instance is enough for occasional hooks), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=webhookd}]'

The image requires IMDSv2 (session tokens). First boot takes under a minute after the instance is running: a boot service reads the instance's EC2 names and VPC ranges, creates the login and the certificate, loads the hook user's firewall table, then webhookd and nginx start and the guard checks them.

Step 2: Connect to your instance

Version SSH user
webhookd 1.22.1 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

The login banner shows where the login and the certificate are:

  webhookd 1.22.1 on Ubuntu 24.04 (cloudimg)
  Login, endpoint, certificate fingerprint:  sudo cat /root/webhookd-credentials.txt
  Hooks: https://<this host>/<hook> (443 only; curl --cacert /etc/webhookd/tls/cert.pem --netrc-file <file>)
  Scripts: /opt/webhookd/scripts (root-owned)   Logs: journalctl -u webhookd   Verify: sudo webhookd-selfcheck
  User guide: https://www.cloudimg.co.uk/guides/webhookd-1-22-1-on-ubuntu-24-04-aws/

Step 3: Check the services and the listening set

systemctl is-active webhookd-metadata webhookd-egress webhookd-scripts-mount webhookd-firstboot webhookd nginx
sudo ss -tlnp | grep -E ':(22|80|443|8080) '
sudo ss -ulnp | grep -E 'webhookd|nginx' || echo 'no UDP socket from webhookd or nginx'

active
active
active
active
active
active
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=897,fd=3),("systemd",pid=1,fd=134))
LISTEN 0      511          0.0.0.0:443       0.0.0.0:*    users:(("nginx",pid=14105,fd=5),("nginx",pid=14104,fd=5),("nginx",pid=14030,fd=5))
LISTEN 0      4096       127.0.0.1:8080      0.0.0.0:*    users:(("webhookd",pid=14471,fd=3))
LISTEN 0      4096            [::]:22           [::]:*    users:(("sshd",pid=897,fd=4),("systemd",pid=1,fd=135))
LISTEN 0      511             [::]:443          [::]:*    users:(("nginx",pid=14105,fd=6),("nginx",pid=14104,fd=6),("nginx",pid=14030,fd=6))
no UDP socket from webhookd or nginx

webhookd-metadata reads this instance's EC2 names and VPC ranges, webhookd-egress loads the hook user's firewall table, webhookd-scripts-mount mounts the scripts directory read-only, webhookd-firstboot prepares the login and the certificate, webhookd is the webhook server and nginx is the TLS front. Only nginx listens on the network (443, IPv4 and IPv6). webhookd listens on 127.0.0.1:8080, and a firewall rule lets only nginx's user (and root) connect to it.

Step 4: Your endpoint and login

Everything generated for your instance is in /root/webhookd-credentials.txt (readable by root only). View it all with sudo cat /root/webhookd-credentials.txt. The lines that are not secret are:

sudo grep -E '^WEBHOOKD_(HOST|URL|PRIVATE_IP|USER|TLS_CA_FILE|TLS_CERT_SHA256|TLS_PUBKEY_SHA256)=' /root/webhookd-credentials.txt

WEBHOOKD_HOST=54.88.128.13
WEBHOOKD_URL=https://54.88.128.13/
WEBHOOKD_PRIVATE_IP=172.31.89.213
WEBHOOKD_USER=cloudimg
WEBHOOKD_TLS_CA_FILE=/etc/webhookd/tls/cert.pem
WEBHOOKD_TLS_CERT_SHA256=DB:E2:7C:F6:FE:4A:46:AA:3D:F2:2E:AA:A1:E3:0C:FE:CF:49:04:13:19:4C:07:BF:16:D6:76:49:EB:57:54:56
WEBHOOKD_TLS_PUBKEY_SHA256=afa97698c3a2981354ce2326911b5746c9fe53e7ef3e61c78d031285a629a5db

The file also holds WEBHOOKD_PASSWORD, the password of cloudimg. Give each caller (each CI system, Git host or alerting tool) a login of its own (Step 9) rather than sharing this one, so you can remove one caller without touching the others. Keep passwords in files with mode 0600, such as a netrc file, never on a command line (a command line is visible to every user on a machine through ps) or in shell history.

Step 5: Check the certificate and pin it in your client

The server's certificate is self-signed, made on your instance:

sudo openssl x509 -in /etc/webhookd/tls/cert.pem -noout -subject -enddate -ext subjectAltName,basicConstraints,extendedKeyUsage
sudo openssl x509 -in /etc/webhookd/tls/cert.pem -noout -fingerprint -sha256

subject=CN = webhookd, O = cloudimg
notAfter=Feb 10 14:19:26 2029 GMT
X509v3 Subject Alternative Name:
    IP Address:54.88.128.13, DNS:ec2-54-88-128-13.compute-1.amazonaws.com, IP Address:172.31.89.213, DNS:ip-172-31-89-213.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
X509v3 Extended Key Usage:
    TLS Web Server Authentication
sha256 Fingerprint=DB:E2:7C:F6:FE:4A:46:AA:3D:F2:2E:AA:A1:E3:0C:FE:CF:49:04:13:19:4C:07:BF:16:D6:76:49:EB:57:54:56

CA:FALSE means it is an end-entity certificate that cannot sign anything else. The SHA-256 fingerprint is the WEBHOOKD_TLS_CERT_SHA256 value in your credentials file. The certificate covers the instance's public and private IPv4 addresses and EC2 DNS names, localhost and 127.0.0.1. An EC2 stop and start gives the instance a new public IP. At the next boot the certificate is re-issued for the new names with the same key: clients that pin the public key (WEBHOOKD_TLS_PUBKEY_SHA256) keep working, while clients that pin the certificate file need the new copy.

Step 6: Call a hook on the instance

Put the login in a netrc file that only you can read (umask 077 makes the file 0600), then call the example hook echo:

umask 077
sudo awk -F= '/^WEBHOOKD_USER=/{u=$2} /^WEBHOOKD_PASSWORD=/{p=$2} END{printf "machine localhost login %s password %s\n", u, p}' /root/webhookd-credentials.txt > ~/webhookd.netrc
curl -sS -N --cacert /etc/webhookd/tls/cert.pem --netrc-file ~/webhookd.netrc https://localhost/echo
curl -sS -N --cacert /etc/webhookd/tls/cert.pem --netrc-file ~/webhookd.netrc -H 'Content-Type: text/plain' --data-binary 'hello from the guide' https://localhost/echo

hook=echo id=2 user=cloudimg body_bytes=0
step 1/3
step 2/3
step 3/3
done
hook=echo id=3 user=cloudimg body_bytes=20
step 1/3
step 2/3
step 3/3
done

The lines arrive one by one, half a second apart: webhookd streams a hook's output as it is printed (-N stops curl from buffering it). Each run gets a number (id), and webhookd keeps its output in /var/lib/webhookd. The second call sends a text body, which the hook receives as $1.

Requests without the right login, plain HTTP, port 80, webhookd's own port and a path that tries to leave the scripts directory are all refused. /healthz answers without a login so a load balancer can check the instance:

printf 'machine localhost login cloudimg password not-the-password\n' > ~/wrong.netrc
curl -s -o /dev/null -w 'no login: %{http_code}\n' --cacert /etc/webhookd/tls/cert.pem https://localhost/echo || true
curl -s -o /dev/null -w 'wrong password: %{http_code}\n' --cacert /etc/webhookd/tls/cert.pem --netrc-file ~/wrong.netrc https://localhost/echo || true
curl -s -o /dev/null -w 'plain HTTP on 443: %{http_code}\n' http://localhost:443/echo || true
curl -s -m 5 -o /dev/null -w 'port 80: %{http_code} %{errormsg}\n' http://localhost/ || true
curl -s -m 5 -o /dev/null -w '127.0.0.1:8080 as ubuntu: %{http_code} %{errormsg}\n' http://127.0.0.1:8080/healthz || true
curl -s -o /dev/null -w 'traversal: %{http_code}\n' --path-as-is --cacert /etc/webhookd/tls/cert.pem --netrc-file ~/webhookd.netrc 'https://localhost/../../etc/passwd' || true
curl -s -o /dev/null -w 'healthz without a login: %{http_code}\n' --cacert /etc/webhookd/tls/cert.pem https://localhost/healthz
rm -f ~/wrong.netrc

no login: 401
wrong password: 401
plain HTTP on 443: 400
port 80: 000 Failed to connect to localhost port 80 after 0 ms: Couldn't connect to server
127.0.0.1:8080 as ubuntu: 000 Failed to connect to 127.0.0.1 port 8080 after 0 ms: Couldn't connect to server
traversal: 400
healthz without a login: 204

Step 7: Call a hook from another machine

Copy the certificate (it is public; the private key never leaves the instance) to the machine that calls your hooks, and compare its fingerprint with WEBHOOKD_TLS_CERT_SHA256 before you trust it:

client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/webhookd/tls/cert.pem' > webhookd-cert.pem
client-host$ openssl x509 -in webhookd-cert.pem -noout -fingerprint -sha256

Create a netrc file with mode 0600 on that machine holding machine <public-ip> login cloudimg password <the password> (or the login you made for this caller in Step 9), then:

client-host$ curl -sS -N --cacert webhookd-cert.pem --netrc-file webhookd.netrc https://<public-ip>/echo
client-host$ curl -sS -N --cacert webhookd-cert.pem --netrc-file webhookd.netrc 'https://<public-ip>/echo?greeting=hello'

Give your client only this file as its CA file (--cacert for curl, verify= in Python requests, ca in Node.js), with host name checking on. Do not add it to the operating system's trust store and never turn verification off (curl -k): the client should trust this one server, so it refuses any other, including one in the middle that presents a different certificate. A client that uses only its system trust store fails with SSL certificate problem: self-signed certificate (curl exit code 60).

From another machine, a request with no login or a wrong password gets 401, plain HTTP sent to 443 gets 400, port 80 and port 8080 refuse the connection even inside a security group that allows all traffic, and a client that sends more than about 10 requests at once, or more than 5 a second, gets 429.

Most CI systems and Git hosts accept a webhook URL with a login. Where a tool cannot pin a certificate, give the instance a DNS name and a certificate from a public CA (Step 13).

Step 8: Write hooks safely

A hook is an executable file in /opt/webhookd/scripts; its name without .sh is its path (deploy.sh is https://<host>/deploy, ci/build.sh is https://<host>/ci/build). A new hook works at once, with no restart. Install it owned by root, so the webhookd user that runs it cannot change it:

cat > ~/deploy.sh <<'EOF'
#!/bin/sh
# webhookd passes each query parameter as a variable of the same name
branch="${branch:-main}"
case "$branch" in
    *[!A-Za-z0-9._-]*) echo "refused: branch name has unexpected characters"; exit 1 ;;
esac
printf 'deploy requested for branch %s\n' "$branch"
EOF
sudo install -o root -g root -m 0755 ~/deploy.sh /opt/webhookd/scripts/deploy.sh
rm -f ~/deploy.sh
curl -sS --cacert /etc/webhookd/tls/cert.pem --netrc-file ~/webhookd.netrc 'https://localhost/deploy?branch=release-1'
curl -sS --cacert /etc/webhookd/tls/cert.pem --netrc-file ~/webhookd.netrc 'https://localhost/deploy?branch=%24%28id%29' | head -n 1
sudo rm -f /opt/webhookd/scripts/deploy.sh

deploy requested for branch release-1
refused: branch name has unexpected characters

The refused call's output ends with a further line, error: exit status 1, which webhookd adds when a hook exits non-zero (head -n 1 above shows only the hook's own line).

Everything a caller sends reaches the hook as text it controls: every query parameter as an environment variable of the same name in lower case (?branch=x is $branch), a text/plain or JSON request body as $1 (other body types are not passed to the hook), and the caller's login as $x_webauth_user. Treat all of it as untrusted:

  • Quote every variable ("$branch"), never eval it and never build a command line, a file path or SQL from it. Check it against what you expect, as the case above does, and refuse anything else. In the example, $(id) arrives as the literal text $(id) and is refused; it is never run.
  • End every line of output with a newline: webhookd sends output line by line. A hook that exits non-zero ends its output with error: exit status <n>.
  • A hook runs as webhookd, not root. It cannot use sudo, cannot read /root or the TLS key and cannot write outside /var/lib/webhookd. If a hook needs to do something privileged, have it write a request (for example a file in /var/lib/webhookd) that a root service you control checks and acts on. Do not give the webhookd user sudo rights.
  • webhookd ends a hook after WHD_HOOK_TIMEOUT seconds (10 by default) and runs at most WHD_HOOK_WORKERS hooks at a time (2 by default). Raise them in /etc/webhookd/webhookd.env for long jobs (at most 3600 seconds), then sudo systemctl restart webhookd.
  • Edit a hook with sudoedit /opt/webhookd/scripts/<name>.sh or replace it with sudo install as above. A symbolic link in the scripts directory is never followed.

Step 9: Add and remove logins

Logins are in /etc/webhookd/htpasswd (root:webhookd 0640, bcrypt hashes only). Add one with htpasswd, which reads the password from standard input with -i so it never appears on a command line, -B for bcrypt and -C 12 for its cost. webhookd reads the file when it starts, so restart it after a change:

openssl rand -base64 24 > ci-bot.password && chmod 0600 ci-bot.password
sudo htpasswd -i -B -C 12 /etc/webhookd/htpasswd ci-bot < ci-bot.password
sudo systemctl restart webhookd

Give ci-bot.password to that caller over a safe channel and delete your copy. To remove a login:

sudo htpasswd -D /etc/webhookd/htpasswd ci-bot
sudo systemctl restart webhookd

The gate refuses to start webhookd with an empty password file, a non-bcrypt entry (such as {SHA}, MD5 or plain text) or the wrong owner or mode, so keep using htpasswd -B. To change the cloudimg password, add it again with the same command; the credentials file keeps the first-boot password, so update or delete that line yourself.

Step 10: Require signed requests

On top of the login, webhookd can require every request to carry an HTTP signature made with a key you hold. Put the callers' public keys (PEM PUBLIC KEY or CERTIFICATE blocks, never a private key) in a root-owned file directly in /etc/webhookd, name it in /etc/webhookd/webhookd.env and restart:

sudo install -o root -g root -m 0644 <callers-public-keys.pem> /etc/webhookd/truststore.pem
echo 'WHD_TRUSTSTORE_FILE=/etc/webhookd/truststore.pem' | sudo tee -a /etc/webhookd/webhookd.env
sudo systemctl restart webhookd

The gate checks the file before every start (journalctl -u webhookd then shows truststore 1 key(s): HTTP signatures required), and any request without a valid signature is refused with 401, even with the right login. How a caller signs a request (the Signature header and the key id) is described in the webhookd documentation. Remove the line and restart to turn it off.

Step 11: The server refuses to run unsafely, and why 8080 is closed

webhookd has no configuration file: its WHD_* environment variables are its configuration, read from /etc/webhookd/webhookd.env. Upstream, it listens on every address on port 8080 in plain HTTP, serves everyone when its password file is missing and can serve static files and expvar counters. On a cloud instance each of those exposes your hooks, so this image fixes them:

  • Port 8080: webhookd listens on 127.0.0.1:8080 only (WHD_LISTEN_ADDR is pinned), and a firewall rule lets only nginx's user (and root) connect to it, so no other local user or hook can go around nginx's limits. webhookd still asks for the login there. Callers use 443.
  • Port 80: not served. Every maintained HTTP client supports HTTPS.
  • TLS in webhookd (WHD_TLS_*): refused. nginx terminates TLS with the cloudimg settings.
  • Static files, notifications and debug logging (WHD_STATIC_*, WHD_NOTIFICATION_*, WHD_LOG_LEVEL=debug): refused. Debug logging would write every hook's parameters and body to the journal. /varz and /static/ return 404.
  • Headers passed to hooks (WHD_ALLOWED_UPSTREAM_HEADERS): Authorization and Cookie are refused, so a login never reaches a hook's environment or log.

webhookd.service runs webhookd-config-gate before every start and webhookd-guard after it. They refuse any setting not listed in this guide, an unsafe password file, a scripts directory not owned by root or with a link or a writable file in it, and a missing firewall table. nginx has its own gate, run before every start and reload. Check them, and the whole service as a client sees it:

sudo journalctl -u webhookd -b --no-pager -o cat | grep -E 'webhookd-(config-gate|guard): OK' | tail -n 2
sudo webhookd-selfcheck

webhookd-config-gate: OK (unit): 1 user(s), 1 executable hook(s); listen 127.0.0.1:8080; scripts mount nosymfollow; egress vpc
webhookd-guard: OK (start): sockets tcp:127.0.0.1:8080 only; anonymous and wrong-password requests refused (401); scripts mount ro,nosuid,nodev,relatime,nosymfollow; egress vpc
SELFCHECK OK: webhookd 1.22.1 behind nginx 443 (TLS, certificate DB:E2:7C:F6:FE:4A:46:AA...); login cloudimg accepted, no login / wrong password 401; /echo streamed; 127.0.0.1:8080 needs the login; port 80 closed

If a change is refused, webhookd (or nginx) does not start and journalctl -u webhookd (or journalctl -u nginx) shows the reason. If the running webhookd ever listens anywhere else or answers a request without a valid login, the guard stops it. If first boot cannot finish, neither service starts.

Step 12: The instance metadata block and outbound traffic

The EC2 instance metadata service (169.254.169.254) hands out the instance's IAM role credentials to any process that asks, and a hook runs whatever a caller can make it run. webhookd, nginx and webhookd-firstboot run with the metadata addresses blocked by systemd (IPAddressDeny). Only webhookd-metadata, which reads the instance's names and VPC ranges, may reach the metadata service, and it may reach nothing else:

for u in webhookd nginx webhookd-firstboot webhookd-metadata webhookd-egress webhookd-scripts-mount; do printf '%-24s deny=%s allow=%s\n' "$u" "$(systemctl show -p IPAddressDeny --value "$u")" "$(systemctl show -p IPAddressAllow --value "$u")"; done

webhookd                 deny=::ffff:169.254.169.254/128 169.254.169.254/32 fd00:ec2::254/128 168.63.129.16/32 ::ffff:168.63.129.16/128 allow=
nginx                    deny=fd00:ec2::254/128 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 168.63.129.16/32 169.254.169.254/32 allow=
webhookd-firstboot       deny=169.254.169.254/32 ::ffff:168.63.129.16/128 ::ffff:169.254.169.254/128 fd00:ec2::254/128 168.63.129.16/32 allow=
webhookd-metadata        deny=0.0.0.0/0 ::/0 allow=169.254.169.254/32 fd00:ec2::254/128
webhookd-egress          deny=0.0.0.0/0 ::/0 allow=
webhookd-scripts-mount   deny=0.0.0.0/0 ::/0 allow=

On top of that, the nftables table loaded by webhookd-egress refuses every connection from the hook user webhookd to instance metadata, link-local and reserved addresses, this instance's VPC ranges (IPv4 and IPv6, read at every boot), the private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fc00::/7) and carrier-grade NAT (100.64.0.0/10). Hooks may reach the internet. nginx's user may reach 127.0.0.1:8080 only. Other users are unaffected:

sudo webhookd-egress-apply --check
IP=$(sudo sed -n 's/^WEBHOOKD_PRIVATE_IP=//p' /root/webhookd-credentials.txt)
for t in http://169.254.169.254/latest/meta-data/ "http://$IP:22/" https://github.com/; do sudo -u webhookd curl -s -m 5 -o /dev/null -w "webhookd -> $t: %{http_code} %{errormsg}\n" "$t" || true; done
timeout 3 bash -c "exec 3<>/dev/tcp/$IP/22" && echo "ubuntu -> $IP:22: open"

webhookd-egress: CHECK OK (nothing loaded): VPC 172.31.0.0/16; IPv6 VPC none; 0 drop-in file(s) opening nothing
webhookd -> http://169.254.169.254/latest/meta-data/: 000 Failed to connect to 169.254.169.254 port 80 after 0 ms: Couldn't connect to server
webhookd -> http://172.31.89.213:22/: 000 Failed to connect to 172.31.89.213 port 22 after 0 ms: Couldn't connect to server
webhookd -> https://github.com/: 200
ubuntu -> 172.31.89.213:22: open

If a hook must reach something inside your network (a deploy target, an internal API), open exactly that range with a root-owned file in /etc/webhookd/egress-allow.d/ ending in .cidr, one network per line, then reload the table:

printf '# the deploy target the deploy hook calls\n10.0.5.0/24\n' | sudo tee /etc/webhookd/egress-allow.d/10-deploy-target.cidr
sudo systemctl reload webhookd-egress
sudo webhookd-egress-apply --check

A range must lie inside one that is denied (the VPC, private or carrier-grade NAT ranges): the metadata, link-local and reserved addresses can never be opened, and an internet range is refused because it is already open. A refused file leaves the loaded rules unchanged, and journalctl -u webhookd-egress says why. Delete the file and reload to close the range again.

Step 13: Use your own DNS name or a certificate from your CA

Point a DNS name at the instance (an Elastic IP keeps the address across stop and start), then add the name to the certificate:

sudo webhookd-set-hostname <dns-name>

This re-issues the self-signed certificate with the same key to cover the name, updates the endpoint in /root/webhookd-credentials.txt and reloads nginx. Copy the new certificate to callers that pin the certificate file.

To use a certificate from your CA (your own, or a public one such as Let's Encrypt), copy the chain (server certificate first) and its unencrypted PEM key to the instance, then run:

sudo webhookd-install-cert <certificate-chain.pem> <private-key.pem> <dns-name>

Port 443 then serves your certificate, and callers trust your CA instead of pinning this instance; Git hosting and CI services that verify against public CAs then work without extra settings. The boot-time renewal never replaces an installed certificate. sudo webhookd-set-hostname <name> goes back to a new self-signed certificate.

Logs, files and limits

  • webhookd log: journalctl -u webhookd (each hook run, its number and status; never a password). nginx access and error logs: /var/log/nginx/. First boot: journalctl -u webhookd-firstboot.
  • Hook output: one file per run in /var/lib/webhookd (owner webhookd, mode 0640), named after the hook, its number and the time. Remove old ones as you see fit.
  • Configuration: /etc/webhookd/webhookd.env (root 0644, checked by the gate on every start; changes take effect on sudo systemctl restart webhookd). Logins: /etc/webhookd/htpasswd. Certificate and key: /etc/webhookd/tls/. Egress drop-ins: /etc/webhookd/egress-allow.d/.
  • Limits: 1 MB request body, 5 requests a second per client address (bursts of 10), 10 connections per address, 10 second hook timeout by default.
  • When you are done with the instance netrc file from Step 6, remove it with rm -f ~/webhookd.netrc.

Support

cloudimg provides support for the image: launch problems, the first-boot setup, the TLS and login configuration and the cloudimg helpers. Contact support@cloudimg.co.uk. webhookd itself is documented at github.com/ncarlier/webhookd.