Streaming & Messaging AWS

NATS 2 on AWS User Guide

| Product: NATS 2 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the NATS 2 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg streaming and messaging listing on AWS Marketplace; you pick it as the software version when you subscribe.

NATS is an open source (Apache-2.0) messaging system from the Cloud Native Computing Foundation. Applications publish messages to subjects (such as app.orders.created) and NATS delivers them to every client subscribed to a matching subject or wildcard (app.orders.*, app.>). It also does request/reply, queue groups for load balancing, and, with JetStream, persistent streams with replay, acknowledgements, consumers and key/value and object stores. Clients exist for Go, Python, JavaScript/TypeScript, Java, .NET, Rust, C and more, plus the nats command-line tool.

What the cloudimg image adds:

  • TLS only. Clients reach the server on 4222 over TLS 1.3 or 1.2 with modern AEAD cipher suites. A plain-text client is refused. Nothing else listens on the network (Step 10).
  • A certificate made on your instance. At first boot the server gets a TLS key and an end-entity certificate for this instance's own names and addresses. It can only vouch for this server and can never sign other certificates. You pin it in your clients (Step 5), or replace it with one from your own CA (Step 12).
  • No anonymous access and no default password. Every client logs in. First boot generates the application login cloudimg-app with a password unique to your instance, stored only as a bcrypt hash. It may use the app.> subjects and JetStream, and is denied the $SYS system account.
  • Every login is its own tenant. nats-user adds a login with its password read from standard input (never a command line), in its own NATS account with its own JetStream limits. One login cannot see, read, purge or delete another login's subjects, replies or streams (Step 8).
  • JetStream with limits. Streams are stored in /var/lib/nats/jetstream, with the server limited to 256 MB of memory and 2 GB of file storage, split between the accounts (Step 9).
  • A server that refuses to run unsafely. Before every start and every reload a configuration gate refuses an unsafe change. After every start and reload a guard checks the running server and stops it if anything is wrong. If first boot cannot finish, the server never serves (Step 11).
  • No telemetry, no outbound network and no access to the instance metadata service. NATS has no telemetry, and the server account may only answer connections that clients open to it (Step 13).
  • A pinned supply chain. NATS Server 2.15.2 is the upstream linux-amd64 release, its SHA-256 checksum pinned and checked against the release's checksum file. Ubuntu 24.04 LTS is fully patched at build time.

What is on the image: NATS Server 2.15.2 (/usr/local/bin/nats-server; configuration in /etc/nats/nats-server.conf, logins in /etc/nats/accounts.conf, the certificate in /etc/nats/tls/, JetStream data in /var/lib/nats/jetstream), and the cloudimg helpers nats-probe (a NATS test client), nats-selfcheck, nats-user, nats-set-hostname, nats-install-cert, nats-config-gate and nats-guard.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
  • A security group that allows:
Port Protocol Source Purpose
22 TCP your admin network SSH
4222 TCP the networks your NATS clients connect from NATS client connections over TLS

Nothing else needs to be open. The monitoring port 8222 listens on the instance's loopback address only, and the cluster (6222), gateway (7222), leafnode (7422), WebSocket and MQTT listeners are not configured, so opening them achieves nothing.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick NATS 2 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (or larger for many thousands of connections, high message rates or large JetStream streams), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=nats-server}]'

The image requires IMDSv2 (session tokens). First boot takes under a minute after the instance is running: the boot service reads the instance's EC2 names, creates the login and the certificate, then the server starts and the guard checks it.

Step 2: Connect to your instance

Version SSH user
NATS 2 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

The login banner shows where the login and the certificate are.

Step 3: Check the services and the listening set

systemctl is-active nats-metadata-names nats-firstboot nats-egress-guard nats
sudo ss -tlnp | grep -E ':(22|4222|6222|7222|7422|8080|8222|1883) '
sudo ss -ulnp | grep nats-server || echo 'no UDP socket'

active
active
active
active
LISTEN 0      4096         0.0.0.0:22        0.0.0.0:*    users:(("sshd",pid=1187,fd=3),("systemd",pid=1,fd=57))
LISTEN 0      4096       127.0.0.1:8222      0.0.0.0:*    users:(("nats-server",pid=1085,fd=3))
LISTEN 0      4096            [::]:22           [::]:*    users:(("sshd",pid=1187,fd=4),("systemd",pid=1,fd=58))
LISTEN 0      4096               *:4222            *:*    users:(("nats-server",pid=1085,fd=6))
no UDP socket

nats-metadata-names reads this instance's EC2 names for the certificate, nats-firstboot prepares the login and the certificate, nats-egress-guard blocks outbound connections from the server account (Step 13), and nats is the server: clients on 4222 on every address, monitoring on 127.0.0.1:8222 only, and nothing else.

Step 4: Your endpoint and login

Everything generated for your instance is in /root/nats-credentials.txt (readable by root only). View it all with sudo cat /root/nats-credentials.txt. The lines that are not secret are:

sudo grep -E '^NATS_(HOST|URL|PORT|PRIVATE_IP|USER|ACCOUNT|SUBJECT_PREFIX|TLS_CA_FILE|TLS_CERT_SHA256|TLS_PUBKEY_SHA256)=' /root/nats-credentials.txt

NATS_HOST=44.201.76.105
NATS_URL=tls://44.201.76.105:4222
NATS_PORT=4222
NATS_PRIVATE_IP=172.31.91.58
NATS_ACCOUNT=APP
NATS_USER=cloudimg-app
NATS_SUBJECT_PREFIX=app.
NATS_TLS_CA_FILE=/etc/nats/tls/nats.crt
NATS_TLS_CERT_SHA256=AA:F9:92:3C:A7:57:9C:DD:9F:FA:A2:88:A0:2D:F0:9F:6C:BF:F6:0B:C2:8E:47:5A:A8:48:12:9D:FC:14:69:3C
NATS_TLS_PUBKEY_SHA256=189c6dff6b814b73a7accf8407beb1e3c82e40d83f9be6c8eca11eabf667532e

The file also holds NATS_PASSWORD. Keep passwords in files with mode 0600, not in shell history or on command lines (a command line is visible to every user on a machine through ps), and never write them into a nats://user:password@host URL in a configuration file.

Step 5: Check the certificate and pin it in your client

The server's certificate is self-signed, made on your instance:

sudo openssl x509 -in /etc/nats/tls/nats.crt -noout -subject -enddate -ext subjectAltName,basicConstraints
sudo openssl x509 -in /etc/nats/tls/nats.crt -noout -fingerprint -sha256

subject=CN = nats-server, O = cloudimg
notAfter=Feb 10 08:12:39 2029 GMT
X509v3 Subject Alternative Name:
    IP Address:44.201.76.105, DNS:ec2-44-201-76-105.compute-1.amazonaws.com, IP Address:172.31.91.58, DNS:ip-172-31-91-58.ec2.internal, IP Address:127.0.0.1, DNS:localhost
X509v3 Basic Constraints: critical
    CA:FALSE
sha256 Fingerprint=AA:F9:92:3C:A7:57:9C:DD:9F:FA:A2:88:A0:2D:F0:9F:6C:BF:F6:0B:C2:8E:47:5A:A8:48:12:9D:FC:14:69:3C

CA:FALSE means it is an end-entity certificate that cannot sign anything else. The SHA-256 fingerprint is the NATS_TLS_CERT_SHA256 value in your credentials file. Copy the certificate (it is public; the private key never leaves the instance) to each client machine:

client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/nats/tls/nats.crt' > nats.crt
client-host$ openssl x509 -in nats.crt -noout -fingerprint -sha256

Compare the fingerprint with NATS_TLS_CERT_SHA256 before you trust the file. Then give your client only this file as its CA (--tlsca for the nats CLI, ssl.create_default_context(cafile=...) in nats.py), with host name checking on. Do not add it to the operating system's trust store: the client should trust this one server, not every program on the machine. Your client will then refuse any other server, including one in the middle that presents a different certificate.

The certificate covers the instance's public and private IPv4 addresses and EC2 DNS names, localhost and 127.0.0.1. An EC2 stop and start gives the instance a new public IP. At the next boot the certificate is re-issued for the new names with the same key: clients that pin the public key (NATS_TLS_PUBKEY_SHA256) keep working, while clients that pin the certificate file need the new copy.

Step 6: Test on the instance

nats-selfcheck logs in as cloudimg-app over TLS with the password from the credentials file, verifies the pinned certificate, does a publish/subscribe round trip and a JetStream round trip (the test stream is deleted afterwards), and proves the refusals:

sudo nats-selfcheck

PASS TLS login as cloudimg-app on 127.0.0.1:4222 (TLSv1.3 TLS_AES_128_GCM_SHA256; certificate verified for 44.201.76.105)
PASS the served certificate is the pinned one (SHA-256 AA:F9:92:3C:A7:57:9C:DD:9F:FA:A2:88:A0:2D:F0:9F:6C:BF:F6:0B:C2:8E:47:5A:A8:48:12:9D:FC:14:69:3C)
PASS the server requires TLS and a login (INFO)
PASS publish/subscribe round trip on app.cloudimg.check.*
PASS JetStream: stream CLOUDIMG_CHECK_DB82261D45 (file storage), 3 acknowledged publishes, pull consumer, deleted
PASS publish outside app.> refused (DENIED Permissions Violation for Publish to "cloudimg-outside.e746be")
PASS subscribe to $SYS.> refused (DENIED Permissions Violation for Subscription to "$SYS.>")
PASS publish to $SYS.REQ.SERVER.PING refused (DENIED Permissions Violation for Publish to "$SYS.REQ.SERVER.PING")
PASS anonymous refused (Authorization Violation)
PASS a wrong password refused (Authorization Violation)
PASS an unknown user refused (Authorization Violation)
PASS a plain-text (non-TLS) connection refused: closed (EOFError)
SELFCHECK OK

nats-probe is the small client behind it. It reads passwords from a file (or the credentials file with --creds) and trusts only the server's certificate. It prints what the server did (OPEN, REFUSED or DENIED and the server's reason). A client without a login and a plain-text client are refused, and a $SYS subscription is denied (the || true keeps a script going whatever the outcome):

sudo nats-probe connect --creds /root/nats-credentials.txt --connect-addr 127.0.0.1
sudo nats-probe connect --anonymous --connect-addr 127.0.0.1 || true
sudo nats-probe connect --plain --creds /root/nats-credentials.txt --connect-addr 127.0.0.1 || true
sudo nats-probe deny --creds /root/nats-credentials.txt --connect-addr 127.0.0.1 --op sub --subject '$SYS.>' || true

OPEN ok TLSv1.3 TLS_AES_128_GCM_SHA256 user=cloudimg-app
REFUSED Authorization Violation
REFUSED closed (tls_required=true): connection closed by the server
DENIED Permissions Violation for Subscription to "$SYS.>"

--connect-addr 127.0.0.1 dials the loopback address while the certificate is still checked for the public name in the credentials file.

Step 7: Connect a client from another machine

On the client machine, install nats.py (python3 -m venv natsenv && natsenv/bin/pip install nats-py) and copy the certificate as in Step 5. Put the password in a file that only you can read:

client-host$ umask 077
client-host$ ssh -i <your-key.pem> ubuntu@<public-ip> "sudo grep '^NATS_PASSWORD=' /root/nats-credentials.txt | cut -d= -f2-" > app.password

This program connects over TLS (trusting only nats.crt), publishes and receives a message, does a request/reply, and creates, fills, reads and deletes a JetStream stream:

# nats_demo.py - publish, request/reply and JetStream through your cloudimg NATS server over TLS (nats-py 2.x).
# Usage: python3 nats_demo.py <server-host> <nats.crt> <user> <password-file>
# The password is read from a file, never the command line; TLS trusts ONLY the server's own certificate.
import asyncio
import ssl
import sys

import nats

host, cafile, user, pwfile = sys.argv[1:5]
with open(pwfile) as f:
    password = f.read().strip()


async def main():
    tls = ssl.create_default_context(cafile=cafile)         # verify against this one certificate, host name checked
    nc = await nats.connect("tls://%s:4222" % host, tls=tls, tls_hostname=host, user=user, password=password)
    got = asyncio.get_running_loop().create_future()

    async def hello(m):                                    # nats-py callbacks are coroutines
        if not got.done():
            got.set_result(m.data.decode())
    await nc.subscribe("app.demo.hello", cb=hello)
    await nc.publish("app.demo.hello", b"hello from nats.py")
    print("received:", await asyncio.wait_for(got, 10))

    async def echo(m):
        await m.respond(b"pong: " + m.data)
    await nc.subscribe("app.demo.echo", cb=echo)
    print("reply:", (await nc.request("app.demo.echo", b"ping", timeout=5)).data.decode())

    js = nc.jetstream()
    await js.add_stream(name="DEMO", subjects=["app.demo.js.>"])
    for i in range(3):
        await js.publish("app.demo.js.%d" % i, b"event %d" % i)
    sub = await js.pull_subscribe("app.demo.js.>", durable="demo")
    msgs = await sub.fetch(3, timeout=5)
    for m in msgs:
        await m.ack()
    print("jetstream: read", len(msgs), "messages from stream DEMO")
    await js.delete_stream("DEMO")
    await nc.close()

asyncio.run(main())

client-host$ natsenv/bin/python nats_demo.py <server-ip> nats.crt cloudimg-app app.password
received: hello from nats.py
reply: pong: ping
jetstream: read 3 messages from stream DEMO

Python's asyncio may also print returning true from eof_received() has no effect when using ssl when the connection closes; it is harmless.

Use a name the certificate covers as <server-ip>: the public IP from outside AWS, the private IP from inside the VPC (a security group rule that names another security group matches private addresses only), an EC2 DNS name, or your own DNS name once you have added it (Step 12). The nats CLI works the same way: --server tls://<server-ip>:4222 --tlsca nats.crt --user cloudimg-app, with the password kept off its command line (in a saved context file with mode 0600, for example).

With this login, publishing outside app.> is refused with a Permissions Violation, and so is anything under $SYS.

Step 8: Add logins, each in its own account

Give each application or tenant its own login rather than sharing one. nats-user reads the new password from standard input, stores only its bcrypt hash, puts the login in its own NATS account (U_<NAME>), re-renders /etc/nats/accounts.conf, checks it with the configuration gate and reloads the server (connected clients stay connected):

umask 077
openssl rand -hex 16 > ~/orders.password
openssl rand -hex 16 > ~/billing.password
sudo nats-user add orders < ~/orders.password
sudo nats-user add billing --publish 'billing.>' --subscribe 'billing.>' --no-jetstream < ~/billing.password
sudo nats-user list

added orders in its own account U_ORDERS: publish app.>,$JS.API.>,$JS.ACK.>,$JS.FC.>; subscribe app.>,_INBOX.>; replies allowed; denied $SYS.>; JetStream on (its own streams only)
nats-user: nats-server reloaded
added billing in its own account U_BILLING: publish billing.>; subscribe billing.>,_INBOX.>; replies allowed; denied $SYS.>; JetStream off
nats-user: nats-server reloaded
cloudimg-app account=APP jetstream=yes publish=app.>,$JS.API.>,$JS.ACK.>,$JS.FC.> subscribe=app.>,_INBOX.> (deny $SYS.>)
orders account=U_ORDERS jetstream=yes publish=app.>,$JS.API.>,$JS.ACK.>,$JS.FC.> subscribe=app.>,_INBOX.> (deny $SYS.>)
billing account=U_BILLING jetstream=no publish=billing.> subscribe=billing.>,_INBOX.> (deny $SYS.>)

By default a new login may publish and subscribe to app.> and use JetStream in its own account; --publish and --subscribe (repeat them as often as you need) replace those subjects, and --no-jetstream gives the account no JetStream at all. Every login is denied $SYS.>. Replies need no grant: a login answers the requests it receives (allow_responses), and its _INBOX.> subscription only ever sees its own account's replies.

Because each login is a separate account, two logins never see each other's messages even on the same subject names, and one cannot read, purge, delete or attach a consumer to the other's streams: to that login the other's streams do not exist. nats-probe isolation proves it with two logins (it creates a test stream as the first, which it deletes again):

openssl rand -hex 16 > ~/tenant-b.password
sudo nats-user add tenant-b < ~/tenant-b.password
sudo nats-probe isolation --creds /root/nats-credentials.txt --connect-addr 127.0.0.1 --a-user orders --a-password-file ~/orders.password --b-user tenant-b --b-password-file ~/tenant-b.password

added tenant-b in its own account U_TENANT_B: publish app.>,$JS.API.>,$JS.ACK.>,$JS.FC.>; subscribe app.>,_INBOX.>; replies allowed; denied $SYS.>; JetStream on (its own streams only)
nats-user: nats-server reloaded
PASS orders created stream A_s1 on app.iso.f96225e2.> (OK)
PASS orders published 1 message into A_s1 (seq 1)
PASS tenant-b STREAM.INFO on orders's A_s1 refused: 404 10059 stream not found
PASS tenant-b STREAM.MSG.GET on orders's A_s1 refused: 404 10059 stream not found
PASS tenant-b DIRECT.GET on orders's A_s1 refused: no direct-get responder (NATS/1.0 503)
PASS tenant-b STREAM.PURGE on orders's A_s1 refused: 404 10059 stream not found
PASS tenant-b STREAM.DELETE on orders's A_s1 refused: 404 10059 stream not found
PASS tenant-b CONSUMER.CREATE on orders's A_s1 refused: 404 10059 stream not found
PASS tenant-b STREAM.NAMES does not show A_s1 (sees 0 stream(s): none)
PASS tenant-b STREAM.LIST does not show A_s1 (sees 0 stream(s): none)
PASS orders's A_s1 still holds its 1 message after tenant-b's attempts
PASS tenant-b subscribed to _INBOX.>, app.> and app.iso.f96225e2.> (its own account's namespace)
PASS orders's request on app.iso-echo.5a980728 got its reply on _INBOX.* (allow_responses; no _INBOX publish grant)
PASS orders publishing to an inbox it was not asked on is refused (DENIED Permissions Violation for Publish to "_INBOX.8dea6db0e93d")
PASS positive control: tenant-b's own app.self publish reached its own app.> subscription
PASS tenant-b's _INBOX.>, app.>, app.iso.f96225e2.> and exact-inbox subscriptions saw 0 of orders's request, reply and publish
PASS tenant-b's own JetStream works in its own account (it lists B_s1; orders lists A_s1)
PASS orders's own publish/subscribe round trip on app.cloudimg.check.* still works
PASS orders deleted A_s1
ISOLATION OK

Applications that must exchange messages share one login (one account), for example a producer and its consumers. Logins in separate accounts are separate tenants by design; this image adds no exports or imports between accounts, and the configuration gate refuses a hand-edited accounts file that adds them.

sudo nats-user passwd <name> changes a password (from standard input), and sudo nats-user remove <name> removes a login and its account. Removing a login does not delete its JetStream data: its streams stay in /var/lib/nats/jetstream/U_<NAME>, and a login you add later with the same name gets the same account and sees them. Delete a login's streams (as that login, for example with nats stream rm) before you remove it, and do not give a removed login's name to someone else. Hand the password files to your applications' secret store, then delete the local copies. To remove the example logins again:

sudo nats-user remove orders
sudo nats-user remove billing
sudo nats-user remove tenant-b
shred -u ~/orders.password ~/billing.password ~/tenant-b.password

removed orders
nats-user: nats-server reloaded
removed billing
nats-user: nats-server reloaded
removed tenant-b
nats-user: nats-server reloaded

Passwords must be 16 to 128 printable characters without spaces, and the published default passwords are refused. Do not edit /etc/nats/accounts.conf by hand: it is rendered from /etc/nats/cloudimg-users.json, and the gate refuses plain-text passwords, anonymous logins, an account holding two logins, and exports or imports.

Step 9: JetStream storage and limits

JetStream stores streams in /var/lib/nats/jetstream on the root EBS volume. The server may use at most 256 MB of memory and 2 GB of file storage for streams, split between the accounts:

Account Memory File Streams Consumers
APP (cloudimg-app) 128 MB 1 GB 100 1000
each login added with nats-user (at most 8 with JetStream) 16 MB 128 MB 20 200

A ninth JetStream login is refused (add it with --no-jetstream), so the accounts can never claim more than the server has. Give each stream its own max_bytes or max_age so it cannot fill its account's share. Check what is in use with the loopback monitoring endpoint:

curl -s http://127.0.0.1:8222/jsz | python3 -m json.tool | grep -E '"(max_memory|max_storage|store_dir|streams|consumers|messages|bytes)"'

        "max_memory": 268435456,
        "max_storage": 2147483648,
        "store_dir": "/var/lib/nats/jetstream",
    "streams": 0,
    "consumers": 0,
    "messages": 0,
    "bytes": 0,

The server's limits are max_memory_store and max_file_store in the jetstream block of /etc/nats/nats-server.conf. You can raise them (grow the EBS volume first), but never lower them below the accounts' total of 256 MB and 2 GB: NATS does not enable JetStream when the accounts' limits add up to more than the server's. The per-account shares stay as shown. Take EBS snapshots for backups.

Step 10: Why 8222 and the cluster ports are closed

NATS is often run with its monitoring endpoint on every address and a cluster port open, which tells anyone on the network about every connection, subscription and account, and lets another server join your cluster. This image sets every listener in NATS's own configuration rather than relying on a firewall:

  • 4222 (clients): TLS required, a login required.
  • 8222 (HTTP monitoring): on 127.0.0.1 only. Read it on the instance (curl -s http://127.0.0.1:8222/varz) or through an SSH tunnel (ssh -L 8222:127.0.0.1:8222 ubuntu@<public-ip>, then http://127.0.0.1:8222/ on your machine).
  • 6222 (cluster routes), 7222 (gateways), 7422 (leafnodes), WebSocket and MQTT: not configured. This is a single server; the configuration gate refuses a listener for any of them, and the guard stops a server that runs with one.

From another machine a connection to 8222, 6222, 7222, 7422, 8080 or 1883 is refused even if your security group allows it, and a plain-text NATS client on 4222 is refused.

Step 11: The server refuses to run unsafely

nats.service runs nats-config-gate before every start and every reload, and nats-guard after both. Check them at any time:

sudo nats-config-gate --unit && echo gate-ok
sudo nats-guard --check

nats-config-gate: OK (unit)
gate-ok
nats-guard: OK (check): TLS and a login required, anonymous refused, listeners 0.0.0.0:4222 127.0.0.1:8222, 1 login(s) each in its own account (cloudimg-app:APP), JetStream in /var/lib/nats/jetstream, uid nats egress replies only

If you edit /etc/nats/nats-server.conf or /etc/nats/accounts.conf and add a listener (cluster, gateway, leafnode, WebSocket, MQTT, profiling, monitoring on every address), allow plain text, turn TLS off, add an anonymous or a plain-text password login, give a login $SYS access, put two logins in one account, add exports or imports, or make the configuration or key world-readable, the gate refuses the change. On a start the server does not start; on systemctl reload nats the running server keeps its previous settings. journalctl -u nats shows the reason. If the running server ever listens anywhere but 4222 and 127.0.0.1:8222, accepts an anonymous login or runs without the outbound block, the guard stops it.

Step 12: Use your own DNS name or a certificate from your CA

Point a DNS name at the instance (an Elastic IP keeps the address across stop and start), then add the name to the certificate:

sudo nats-set-hostname <dns-name>

This re-issues the self-signed certificate with the same key to cover the name, updates the endpoint in /root/nats-credentials.txt and reloads the server. Copy the new nats.crt to clients that pin the certificate file.

To use a certificate from your CA (your own, or a public one such as Let's Encrypt), copy the chain (server certificate first) and its unencrypted PEM key to the instance, then run:

sudo nats-install-cert <certificate-chain.pem> <private-key.pem> <dns-name>

Port 4222 then serves your certificate, and clients trust your CA instead of pinning this instance. The boot-time renewal never replaces an installed certificate. sudo nats-set-hostname <name> goes back to a new self-signed certificate.

Step 13: The instance metadata and outbound block

The EC2 instance metadata service (169.254.169.254) hands out the instance's IAM role credentials to any process that asks. A network service must not be able to reach it, nor use the instance as a stepping stone into your VPC. Two layers keep the server in:

  • nats and nats-firstboot run with the metadata addresses blocked by systemd (IPAddressDeny). Only nats-metadata-names, which reads the instance's names for the certificate, may reach the metadata service, and it may reach nothing else.
  • The server account (nats) has no outbound network at all. nats-egress-guard loads an nftables table that lets that account answer the connections clients open to it and refuses everything else, over IPv4 and IPv6: your VPC's addresses (whatever its ranges), loopback, the metadata service, the reserved ranges and the internet. NATS here needs no outbound connection: there is no cluster, gateway or leafnode, and no telemetry.
for u in nats nats-firstboot nats-metadata-names; do printf '%-20s deny=%s allow=%s\n' "$u" "$(systemctl show -p IPAddressDeny --value "$u")" "$(systemctl show -p IPAddressAllow --value "$u")"; done
sudo nft list table inet cloudimg_nats_egress | grep -o 'comment "[a-z0-9-]*"'
sudo setpriv --reuid=nats --regid=nats --clear-groups -- curl -s -m 5 -o /dev/null -w '%{http_code}\n' https://nats.io/ || true

nats                 deny=::ffff:169.254.169.254/128 fd00:ec2::254/128 ::ffff:168.63.129.16/128 169.254.169.254/32 168.63.129.16/32 allow=
nats-firstboot       deny=fd00:ec2::254/128 168.63.129.16/32 ::ffff:168.63.129.16/128 169.254.169.254/32 ::ffff:169.254.169.254/128 allow=
nats-metadata-names  deny=0.0.0.0/0 ::/0 allow=fd00:ec2::254/128 169.254.169.254/32
comment "cloudimg-nats-backstop4"
comment "cloudimg-nats-mcast4"
comment "cloudimg-nats-mcast6"
comment "cloudimg-nats-replies"
comment "cloudimg-nats-deny-tcp"
comment "cloudimg-nats-deny"
000

The last command tries to reach the internet as the nats user and prints 000: no connection was made (the DNS lookup is refused first). If you add a cluster, gateway or leafnode yourself, allow its peers in /usr/local/share/cloudimg/nats/nats-egress.nft (and in the gate) first.

Logs, files and limits

  • Server log: journalctl -u nats (connections, authorization failures and refusals; never a password); first boot: journalctl -u nats-firstboot.
  • Configuration: /etc/nats/nats-server.conf (checked by the gate on every start and reload; the shipped copy is in /usr/local/share/cloudimg/nats/). Logins: /etc/nats/accounts.conf (root:nats 0640, rendered from /etc/nats/cloudimg-users.json, root only). Certificate and key: /etc/nats/tls/.
  • JetStream data: /var/lib/nats/jetstream. Messages per payload: at most 1 MB (max_payload); connections: at most 10,000.
  • The server runs as the unprivileged nats user in a systemd sandbox (no capabilities, a read-only system except the JetStream store, a system-call filter, core dumps off).

Support

cloudimg provides support for the image: launch problems, the first-boot setup, the TLS configuration and the cloudimg helpers. Contact support@cloudimg.co.uk. NATS itself is documented at docs.nats.io.