rsyslog 8.2312.0 on AWS User Guide

Overview

This guide covers launching and using the rsyslog 8.2312.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg log management listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

rsyslog is the syslog daemon that ships as the system logger on most Linux distributions. This image configures it as a central log server. Instead of only writing the instance's own logs, it listens for syslog from your other machines and files everything it receives under the host and the program that sent it. Logs that live only on the machine that produced them are the logs you cannot read when that machine is the problem; a collector gives you one place to search, one retention policy and a copy of the evidence that survives the instance.

Port	Protocol	What it is
514	UDP	Classic syslog, spoken by switches, firewalls, routers and older Unix hosts. Lossy and spoofable, but universal. Restricted by sender address.
514	TCP	Syslog over a stream. No loss from a full buffer, but no application level acknowledgement. Restricted by sender address.
2514	RELP over TLS	The one to use where you can. Every message is acknowledged by the collector, so nothing is lost across a restart, and the session is encrypted and authenticated in both directions.
22	TCP	SSH.

Everything the collector receives is written to /var/log/remote/<sending-host>/<program>.log.

Security by design

This image is not an open relay. A syslog receiver reachable from the internet is found and abused within hours: UDP syslog is trivially spoofable, the protocol has no authentication, and a few megabits of junk aimed at port 514 fills a disk. So:

• Closed by default. The listing security group opens SSH only. You open the receivers to your own VPC or VPN ranges when you are ready (Step 6).
• The plaintext receivers accept private addresses only. UDP and TCP 514 take messages from loopback and the three RFC1918 private ranges only, controlled by one commented file you narrow to your own subnet.
• The encrypted receiver demands a client certificate. On its first boot your instance mints its own certificate authority, a server certificate and one client certificate. That authority exists on no other machine, so the only hosts that can deliver over RELP are the ones you issue certificates to.
• Rate limited and bounded. Both plaintext receivers are rate limited per source, and the queue is bounded in memory and on disk, shedding informational and debug messages before it can fill the volume.
• Fail closed. rsyslog does not start until first boot has created the instance's certificates, so a listener never runs without its encryption material or with material from the image.
• Nothing cryptographic is baked into the image. No two customers share a trust root.
• IMDSv2 only, and rsyslog fetches nothing from the network: it only receives.
• Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

Why the distro build. rsyslog comes from the Ubuntu 24.04 archive (with rsyslog-relp and rsyslog-gnutls), not from the upstream package repository or source. A log collector has to keep taking security fixes for years with nobody logged into it, and the distro build is the one Ubuntu's unattended upgrades patch, with support through 2029.

What is on the image: rsyslog 8.2312.0 configured as a central collector, a per host, per program log tree, a per instance certificate authority generated on first boot, a retention policy driven by its own hourly timer, two self test commands, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
• A security group that allows TCP 22 (SSH) from your own networks. Add the receivers (UDP 514, TCP 514, TCP 2514) only from the VPC or VPN ranges your log sources live on, never from 0.0.0.0/0.
• Decide which network your log sources live on. You narrow the collector to it in Step 6.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick rsyslog 8.2312.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (a collector is I/O bound: for hundreds of hosts give it more disk rather than more cores), your key pair and a security group as above. The AMI is IMDSv2 only.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --block-device-mappings 'DeviceName=/dev/sda1,Ebs={VolumeSize=50,VolumeType=gp3}' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=log-collector}]'

Received logs live on the root volume under /var/log/remote; size it for your retention (Step 9), or attach a larger volume and mount it there.

Step 2: Connect to your instance

Version	SSH user
rsyslog 8.2312.0 on Ubuntu 24.04	ubuntu

ssh -i <your-key.pem> ubuntu@<public-ip>

The login banner shows the receivers, where the logs land and where this guide lives.

Step 3: Check the collector

First boot mints the instance's certificate authority and keys, then releases rsyslog:

systemctl is-active rsyslog.service
systemctl is-active rsyslog-firstboot.service
rsyslogd -v | head -1

active
active
rsyslogd  8.2312.0 (aka 2023.12) compiled with:

rsyslog-firstboot.service is a one shot unit that stays active after it succeeds. Three receivers listen: UDP 514, TCP 514 and TCP 2514. SSH is the only other public listener; the DNS resolver and the time client stay on loopback:

sudo ss -Hlntu | awk '{print $1, $5}' | sort -u

tcp 0.0.0.0:22
tcp 0.0.0.0:2514
tcp 0.0.0.0:514
tcp 127.0.0.53%lo:53
tcp 127.0.0.54:53
tcp [::]:22
tcp [::]:2514
tcp [::]:514
udp 0.0.0.0:514
udp 127.0.0.1:323
udp 127.0.0.53%lo:53
udp 127.0.0.54:53
udp <private-ip>%ens5:68
udp [::1]:323
udp [::]:514

If rsyslog.service is not active, check the first boot unit first. The collector is deliberately gated on it, so a first boot that failed shows up as a collector that never started rather than one listening without encryption material:

sudo systemctl status rsyslog-firstboot.service --no-pager | head -20

Step 4: Read the connection details

The address, ports, certificate paths and fingerprints your log sources need are in a root only file written on first boot:

sudo grep -E '^(COLLECTOR_PRIVATE_IP|SYSLOG_UDP_PORT|SYSLOG_TCP_PORT|RELP_TLS_PORT|RELP_CA_CERT|RELP_CLIENT_CERT|RELP_CLIENT_KEY|CA_FINGERPRINT_SHA256)=' /root/rsyslog-credentials.txt

COLLECTOR_PRIVATE_IP=<collector-ip>
SYSLOG_UDP_PORT=514
SYSLOG_TCP_PORT=514
RELP_TLS_PORT=2514
RELP_CA_CERT=/etc/rsyslog.d/tls/ca.crt
RELP_CLIENT_CERT=/etc/rsyslog.d/tls/client.crt
RELP_CLIENT_KEY=/etc/rsyslog.d/tls/client.key
CA_FINGERPRINT_SHA256=<ca-fingerprint>

The fingerprints are unique to this instance. Send to the private address: it does not change when the instance stops and starts. The server certificate names the instance's private and public addresses:

sudo openssl x509 -in /etc/rsyslog.d/tls/server.crt -noout -subject -issuer -ext subjectAltName

subject=O = cloudimg, CN = cloudimg-rsyslog-collector
issuer=O = cloudimg, OU = rsyslog Central Log Server, CN = cloudimg-rsyslog-ca-<random-id>
X509v3 Subject Alternative Name:
    IP Address:<private-ip>, IP Address:127.0.0.1, DNS:ip-<private-ip-dashed>, DNS:localhost, IP Address:<public-ip>

The credentials file and the three private keys are not readable by other users:

sudo stat -c '%n  mode=%a  owner=%U:%G' /root/rsyslog-credentials.txt /etc/rsyslog.d/tls/ca.key /etc/rsyslog.d/tls/server.key /etc/rsyslog.d/tls/client.key

/root/rsyslog-credentials.txt  mode=600  owner=root:root
/etc/rsyslog.d/tls/ca.key  mode=640  owner=root:syslog
/etc/rsyslog.d/tls/server.key  mode=640  owner=root:syslog
/etc/rsyslog.d/tls/client.key  mode=640  owner=root:syslog

The certificate authority's private key, /etc/rsyslog.d/tls/ca.key, issues every client certificate. Treat it as a secret and back it up somewhere private (Backing up, below).

Step 5: Send your first message

The quickest proof is to have the collector receive a message over the network from itself. logger is part of every Ubuntu host:

logger --server 127.0.0.1 --port 514 --tcp \
  --tag guidetest --priority local4.warning \
  "cloudimg guide test message"
sleep 3
sudo find /var/log/remote -type f -name 'guidetest.log' -exec tail -n 1 {} \;

2026-10-20T21:55:09.456770+00:00 ip-<private-ip-dashed> 127.0.0.1 local4.warning guidetest cloudimg guide test message

Reading left to right: an RFC3339 timestamp, the hostname the sender claimed, the address the message actually came from, the facility and severity, the program tag and the message. Syslog lets a sender put anything in the hostname field, so the real source address on every line is what makes a message attributable.

One directory per sending host, one file per program inside it. The tree is owned by syslog:adm, mode 0750, so it is not world readable:

sudo find /var/log/remote -type f | sort
sudo stat -c '%n %a %U:%G' /var/log/remote

/var/log/remote/ip-<private-ip-dashed>/guidetest.log
/var/log/remote 750 syslog:adm

A remote host controls both the hostname and the program tag it sends, and both are used to build a path. Each is sanitised first, so a sender that puts ../../../../tmp/x in its hostname cannot write outside the tree. In practice rsyslog's parser rejects a malformed hostname and files the message under the sender's IP address instead.

Step 6: Open the collector to your own network

Security group. Add inbound rules for the receivers you will use, scoped to the address range your log sources live on, for example your VPC CIDR:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol udp --port 514 --cidr <your-vpc-cidr>
aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 514 --cidr <your-vpc-cidr>
aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 2514 --cidr <your-vpc-cidr>

Sender policy. The plaintext receivers also check the sender's address in the collector itself. This is the one file to edit:

sudo grep '^\$AllowedSender' /etc/rsyslog.d/10-cloudimg-allowed-senders.conf

$AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
$AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16

As shipped it permits loopback and the three RFC1918 private ranges. Narrow it to the network your log sources actually live on, for example:

$AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16
$AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16

then sudo systemctl restart rsyslog. Never add 0.0.0.0/0. No plaintext syslog receiver is safe to expose to the internet. To receive from outside your private network, use RELP on 2514, which is encrypted and authenticated. The file has no entry for RELP on purpose: its access control is the client certificate, not the source address.

Step 7: Point a Linux host at the collector

On each host whose logs you want, add one file to its rsyslog configuration. For hosts inside your private network, create /etc/rsyslog.d/90-forward-to-collector.conf:

# Forward everything to the cloudimg central log server.
# The disk-assisted queue buffers rather than drops messages
# if the collector is briefly unreachable.
*.* action(type="omfwd"
           target="COLLECTOR_ADDRESS" port="514" protocol="tcp"
           queue.type="LinkedList"
           queue.filename="fwd-collector"
           queue.maxDiskSpace="256m"
           queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

Replace COLLECTOR_ADDRESS with the collector's private address (COLLECTOR_PRIVATE_IP from Step 4), then run sudo systemctl restart rsyslog on the sending host. Within seconds a directory named after that host appears under /var/log/remote on the collector.

Step 8: Encrypted, authenticated delivery with RELP

RELP is what makes rsyslog worth choosing for central collection: the sender holds each message until the collector acknowledges it, so a collector restart or a network blip does not silently lose the lines in flight. On this image RELP is TLS only and requires a client certificate. Copy three files from the collector to the sending host (ca.crt, client.crt and client.key from /etc/rsyslog.d/tls/, for example with scp from your workstation), install rsyslog-relp and rsyslog-gnutls there, put the files in /etc/rsyslog.d/tls/ owned root:syslog mode 0640, and create /etc/rsyslog.d/90-forward-relp.conf:

module(load="omrelp")
*.* action(type="omrelp"
           target="COLLECTOR_ADDRESS" port="2514"
           tls="on"
           tls.caCert="/etc/rsyslog.d/tls/ca.crt"
           tls.myCert="/etc/rsyslog.d/tls/client.crt"
           tls.myPrivKey="/etc/rsyslog.d/tls/client.key"
           tls.authMode="name"
           tls.permittedPeer=["cloudimg-rsyslog-collector"]
           queue.type="LinkedList"
           queue.filename="relp-collector"
           queue.maxDiskSpace="256m"
           queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

Both ends now authenticate each other: the sender checks the collector's certificate against the authority, and the collector checks the sender's. A certificate signed by anything other than this instance's own authority is refused at the handshake, including one that carries the exact permitted name. rsyslog also offers a certvalid mode that sounds stricter; on the librelp version Ubuntu 24.04 ships it does not enforce the client certificate at all, so this image uses name mode, which was measured rejecting both kinds of forgery.

More log sources. Every source can share the one client certificate, or you can issue one each so a single compromised host can be revoked on its own. Issue from the collector, keeping the common name cloudimg-relp-client (or add the new name to tls.permittedPeer in /etc/rsyslog.d/30-cloudimg-collector-inputs.conf):

sudo bash -c 'cd /etc/rsyslog.d/tls && umask 077 &&
  openssl ecparam -name prime256v1 -genkey -noout -out host2.key &&
  openssl req -new -key host2.key -subj "/O=cloudimg/CN=cloudimg-relp-client" -out host2.csr &&
  printf "basicConstraints=CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=clientAuth\n" > host2.ext &&
  openssl x509 -req -in host2.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
    -sha256 -days 825 -extfile host2.ext -out host2.crt &&
  rm -f host2.csr host2.ext'

Copy ca.crt, host2.crt and host2.key to the new source and delete host2.key from the collector.

Step 9: Verify the whole thing, at any time

The image carries its own end to end check. It stands up simulated remote log sources on loopback addresses, sends a message from a different source address over each of the three transports, and asserts each one landed in the right file with the right facility, severity and timestamp. It then proves the refusals (a sender outside the permitted networks, a self signed client certificate and a certificate with the permitted name from another authority) and that retention really rotates and compresses:

sudo /usr/local/sbin/rsyslog-roundtrip-check.sh

ROUNDTRIP_OK udp=10.99.99.11->simhost-udp-<pid> tcp=10.99.99.12->simhost-tcp-<pid> relp_tls=10.99.99.13->simhost-relp-<pid> denied_sender=dropped rogue_cert=refused rotation=rotated+compressed

To see what the instance actually exposes, enumerated from the live socket table rather than the configuration:

sudo /usr/local/sbin/rsyslog-surface-audit.sh | tail -n 14

== the shipped sender policy must not be an open relay ==
  $AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
  $AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
  restricted - ok

== rate limiting and the bounded queue must be configured ==
  input rate limiting configured - ok
  bounded disk-assisted queue configured - ok
  queue discard guard configured - ok

== RELP must be TLS with mutual authentication ==
  relp listener is TLS - ok
  relp requires a certificate issued by this VM's authority - ok

SURFACE_AUDIT_OK expected=22/tcp 514/udp 514/tcp 2514/tcp

Retention and disk safety

A collector without retention fills its disk and takes itself down. This one ships with a policy:

grep -vE '^\s*(#|$)' /etc/cloudimg/logrotate-rsyslog-remote.conf
systemctl list-timers cloudimg-rsyslog-logrotate.timer --no-pager

/var/log/remote/*/*.log {
    hourly
    rotate 14
    maxsize 128M
    maxage 30
    missingok
    notifempty
    compress
    delaycompress
    copytruncate
    create 0640 syslog adm
    su syslog adm
    sharedscripts
}
NEXT                        LEFT LAST PASSED UNIT                             ACTIVATES
<next-run>                   <n>min -        - cloudimg-rsyslog-logrotate.timer cloudimg-rsyslog-logrotate.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

As shipped: rotate when a file passes 128 MB or once an hour, whichever comes first; keep 14 generations; compress everything older than the current one; discard anything over 30 days old. The policy is not in /etc/logrotate.d on purpose: it has its own hourly timer and state file, so the daily system logrotate run does not rotate the same files out of step. To change retention, edit that file; it is ordinary logrotate syntax.

Security notes

• The receivers are closed in the security group until you open them, and the plaintext ones accept private addresses only even when open.
• RELP on 2514 accepts only clients holding a certificate from this instance's own authority (mutual TLS, tls.authMode="name"); the round-trip check re-proves that on demand.
• rsyslog runs confined by its AppArmor profile, in enforce mode; /etc/apparmor.d/rsyslog.d/cloudimg-collector adds only the collector's own paths to it.
• Before it opens a listener, rsyslog runs a guard (/usr/local/sbin/rsyslog-secret-guard.sh) that refuses to start if the instance's certificates or keys, or its shipped configuration, match any of the test keys and sample credentials published in rsyslog's own source tree and packages.
• The image is IMDSv2 only, and rsyslog fetches nothing from the network.
• SSH accepts key pairs only, with root login and X11 forwarding switched off, pinned in /etc/ssh/sshd_config.d/00-cloudimg.conf (it sorts first, so later drop-ins cannot override it). The effective values:

sudo sshd -T | grep -E '^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication|x11forwarding) '

permitrootlogin no
passwordauthentication no
kbdinteractiveauthentication no
x11forwarding no

• Unattended upgrades apply Ubuntu security updates, including rsyslog's.

Server components

Component	Version	Purpose
rsyslog (rsyslog, rsyslog-relp, rsyslog-gnutls)	8.2312.0 (Ubuntu 24.04 package)	Syslog collector, RELP and TLS
librelp	1.11.0 (Ubuntu)	RELP protocol library
logrotate	Ubuntu 24.04	Retention
Ubuntu	24.04 LTS	Operating system

Filesystem layout

Path	Purpose
/var/log/remote/<host>/<program>.log	Received logs
/etc/rsyslog.d/10-cloudimg-allowed-senders.conf	Sender policy for the plaintext receivers
/etc/rsyslog.d/20-cloudimg-collector-ruleset.conf	Per host, per program filing and the bounded queue
/etc/rsyslog.d/30-cloudimg-collector-inputs.conf	The three receivers
/etc/rsyslog.d/tls	The instance's certificate authority, server and client certificates and keys
/etc/cloudimg/logrotate-rsyslog-remote.conf	Retention policy
/usr/local/sbin/rsyslog-roundtrip-check.sh	End to end self test
/usr/local/sbin/rsyslog-surface-audit.sh	Exposed surface audit
/root/rsyslog-credentials.txt	Address, ports, certificate paths and fingerprints, root only

Backing up

The certificate authority and the configuration are what make the collector yours:

sudo tar czf rsyslog-collector-backup.tar.gz /etc/rsyslog.d /etc/cloudimg /root/rsyslog-credentials.txt

Store the archive somewhere private: it holds the authority's private key. Restore by extracting it over the same paths and restarting rsyslog. Back up /var/log/remote itself with EBS snapshots or by shipping it to Amazon S3.

Troubleshooting

Nothing is arriving from a remote host. Check the security group allows the port from the sender's address, then that the sender policy includes its range, and look for the refusal:

sudo journalctl -u rsyslog -n 30 --no-pager | grep -i 'disallowed sender' \
  || echo "no disallowed-sender entries in the last 30 lines"

A UDP message from disallowed sender discarded line means the collector received it and dropped it on policy. No line at all means the packet never arrived: check the security group, the network ACL and the sending host's configuration.

RELP connections are refused. The collector logs the reason:

sudo journalctl -u rsyslog -n 40 --no-pager | grep -i 'authentication error' \
  || echo "no RELP authentication errors in the last 40 lines"

certificate validation failed means the client certificate was not issued by this instance's authority. Copy ca.crt, client.crt and client.key from this collector to the sending host again; a certificate from another deployment will not work, which is the point.

Do not delete a live log file. rsyslog keeps the per host files open, so removing one sends the next messages to a deleted file. Truncate instead (sudo truncate -s 0 /var/log/remote/<host>/<program>.log); the retention policy uses copytruncate for the same reason.

The collector is not running after a reboot. Check the first boot unit; the collector is gated on it by design:

systemctl is-active rsyslog-firstboot.service
ls -l /var/lib/cloudimg/

Local logs stopped. This instance is a log server and also logs itself. Confirm both halves:

logger --tag localcheck "local capture check"
sleep 2
sudo grep -c localcheck /var/log/syslog

Support and licensing

rsyslog is open source (GPL-3.0-or-later for the daemon, LGPL-3.0-or-later for its runtime and libraries, with some Apache-2.0 components); the image ships Ubuntu's unmodified packages. There is no per user or per CPU licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

Please include the output of sudo /usr/local/sbin/rsyslog-surface-audit.sh and sudo /usr/local/sbin/rsyslog-roundtrip-check.sh with any report about reception.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk

All product and company names are trademarks or registered trademarks of their respective holders. cloudimg is not affiliated with or endorsed by the rsyslog project or Adiscon GmbH; the name is used only to identify the unmodified software.