Observability Azure

rsyslog 8.2312.0 on Ubuntu 24.04 on Azure User Guide

Last updated: 2026-07-26 | Product: rsyslog 8.2312.0 on Ubuntu 24.04 on Azure

Overview

This guide covers the deployment and use of rsyslog 8.2312.0 on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images.

rsyslog is the syslog daemon that ships as the system logger on most Linux distributions. This image configures it as something rather more useful: a central log server. Instead of only writing this machine's own logs, it listens for syslog from your other machines, and files everything it receives under the host and the program that sent it.

Centralising logs is one of those things that is obvious in hindsight. Logs that live only on the machine that produced them are the logs you cannot read when that machine is the problem, and they are gone entirely when the instance is replaced. A collector gives you one place to grep, one retention policy, and a copy of the evidence that survives the box.

The shape of the deployment

Port	Protocol	What it is
`514`	UDP	Classic syslog. What switches, firewalls, routers and older Unix hosts speak. Lossy and spoofable, but universal. Restricted by sender address.
`514`	TCP	Syslog over a stream. No message loss from a full buffer, but no application level acknowledgement either, so a collector restart can still lose messages in flight. Restricted by sender address.
`2514`	RELP over TLS	The one to use where you can. Every message is acknowledged by the collector at the application layer, so nothing is lost across a restart or a broken connection, and the session is encrypted and authenticated in both directions.
`22`	TCP	SSH.

Everything the collector receives is written to:

/var/log/remote/<sending-host>/<program>.log

Security by design

This is not an open relay, and that is a deliberate design decision rather than an omission. A syslog receiver reachable from the public internet is found and abused within hours: UDP syslog is trivially spoofable, there is no authentication in the protocol, and a few megabits of junk aimed at port 514 will fill a disk. So:

The plaintext receivers accept traffic only from loopback and the three RFC1918 private ranges. One clearly commented file controls this, and this guide shows you how to narrow it further to your own subnet, which you should.
The encrypted receiver demands a client certificate. On its first boot, your VM mints its own certificate authority, a server certificate, and one client certificate. That authority exists on no other machine in the world, so the only hosts that can deliver over RELP are the ones you issue certificates to.
Both plaintext receivers are rate limited. A single host stuck in a logging loop is the ordinary way a collector falls over.
The queue is bounded in memory and on disk, and sheds informational and debug messages before it can fill the volume rather than blocking the receiver.
The collector will not start at all until first boot has created this VM's certificates. There is no window in which a listener is up with no encryption material, or with material inherited from the image.
Nothing cryptographic is baked into the image. No two customers share a trust root.

A note on the version, and why it is the distro build

This image installs rsyslog from the Ubuntu 24.04 archive rather than from the upstream project's own package repository or from source. A log collector is infrastructure that has to keep taking security fixes for years with nobody logged into it, and the distro build is the one wired into Ubuntu's unattended upgrades security pocket, with support through 2029. Upstream's own newest release line moves faster; this appliance deliberately trades that for a receiver that keeps patching itself.

What is included:

rsyslog 8.2312.0 from the Ubuntu 24.04 archive, with rsyslog-relp and rsyslog-gnutls, configured as a central collector
A per host, per program log tree under /var/log/remote, with the path components sanitised so a hostile sender cannot write outside it
A per VM certificate authority, server certificate and client certificate, generated on first boot
A retention policy that rotates on size, compresses, and keeps 14 generations, driven by its own hourly timer
Two self test tools, so you can prove reception and inspect the exposed surface at any time
A paired deployment guide and 24/7 support from cloudimg

Before you start

You will need an Azure subscription, permission to create a virtual machine, and an SSH key pair. Nothing needs to be installed locally beyond an SSH client.

Decide up front which network your log sources live on. You will narrow the collector to it in a few minutes, and it is much easier to do that once than to remember later.

Deploy the virtual machine

Deploy from the Azure Marketplace listing, choosing:

Size: Standard_B2s is the recommended starting point. A collector is I/O bound rather than CPU bound; if you are receiving from hundreds of hosts, give it more disk rather than more cores.
Authentication: SSH public key, with the admin username azureuser.
Disk: the default operating system disk holds the received logs at /var/log/remote. For a busy fleet, attach a larger disk and mount it there.

Network Security Group rules

By default the Marketplace deployment opens only SSH. That is the correct starting point: the collector is restricted internally as well, but there is no reason to expose a receiver before you have decided who is allowed to reach it.

Add rules for the receivers you intend to use, scoped to the source address range your log sources live on and never to Any:

# Replace 10.20.0.0/16 with the network your log sources are on.
az network nsg rule create --resource-group <your-resource-group> \
  --nsg-name <your-nsg> --name allow-syslog-udp --priority 1001 \
  --source-address-prefixes 10.20.0.0/16 --destination-port-ranges 514 \
  --protocol Udp --access Allow

Connect and read this VM's credentials

ssh azureuser@<collector-ip>

Your VM generated its own certificate authority and fingerprints on its first boot, and recorded them here. The file is readable only by root.

sudo cat /root/rsyslog-credentials.txt

You will see the collector's address, the three ports, the paths to the certificate authority and the client certificate and key, and the SHA-256 fingerprints of all three certificates. The fingerprints are unique to this virtual machine. Nothing in that file exists on any other cloudimg deployment.

The private key of the authority lives at /etc/rsyslog.d/tls/ca.key. Treat it as a secret, and back it up somewhere you can reach if this VM is ever lost, because it is what all of your log sources' certificates are issued from.

Confirm the collector is running

systemctl is-active rsyslog.service
systemctl is-active rsyslog-firstboot.service
ss -lntu | grep -E ':(514|2514)'

rsyslog.service should be active, the first boot unit should be active (it is a one-shot that stays active after succeeding), and you should see three receivers: UDP 514, TCP 514 and TCP 2514.

If rsyslog.service is not active, check the first boot unit first. The collector is deliberately gated on it, so a first boot that failed shows up as a collector that never started rather than as a collector listening without encryption material:

sudo systemctl status rsyslog-firstboot.service --no-pager | head -20

Send your first message

The quickest proof is to have the collector receive a message over the network from itself. logger is part of the base system on every Ubuntu host.

logger --server 127.0.0.1 --port 514 --tcp \
  --tag guidetest --priority local4.warning \
  "cloudimg guide test message"

Then find it. The collector files messages by the hostname in the message and the program that sent them, so this one lands under this VM's own hostname:

sleep 3
sudo find /var/log/remote -type f -name 'guidetest.log'
sudo find /var/log/remote -type f -name 'guidetest.log' -exec tail -n 1 {} \;

The line you get back looks like this:

2026-07-26T12:36:16.035147+00:00 collector-host 127.0.0.1 local4.warning guidetest cloudimg guide test message

Reading left to right: an RFC3339 timestamp, the hostname the sender claimed, the address the message actually came from, the facility and severity, the program tag, and the message. The claimed hostname and the real source address are both recorded on every line on purpose. Syslog lets a sender put whatever it likes in the hostname field, so the source address is what makes a message attributable.

Where your logs land

sudo find /var/log/remote -type f | sort

One directory per sending host, one file per program inside it. The tree is owned by syslog:adm and is mode 0750, so it is not world readable.

A remote host controls both the hostname and the program tag in the messages it sends, and both are used to build a path. Each is sanitised before it is used, so a sender that puts ../../../../tmp/x in its hostname cannot write outside the tree. In practice rsyslog's parser rejects a malformed hostname outright and files the message under the sender's IP address instead, which is the behaviour you want.

Open the collector to your own network

This is the file to edit, and it is the only one:

sudo cat /etc/rsyslog.d/10-cloudimg-allowed-senders.conf

As shipped it permits loopback and the three RFC1918 private ranges. Narrow it to the network your log sources actually live on. Replace the two directives with your own range:

$AllowedSender UDP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16
$AllowedSender TCP, 127.0.0.1/32, [::1]/128, 10.20.0.0/16

Then restart the collector:

sudo systemctl restart rsyslog

Do not add 0.0.0.0/0. There is no configuration of a plaintext syslog receiver that is safe to expose to the public internet. If you need to receive logs from outside your own network, use the RELP receiver on 2514, which is encrypted and authenticated.

There is deliberately no entry in that file for RELP. Its access control is the client certificate, not the source address.

Point a Linux host at the collector

On each host whose logs you want, add one file to its rsyslog configuration.

Plaintext, for hosts inside your private network — create /etc/rsyslog.d/90-forward-to-collector.conf:

# Forward everything to the cloudimg central log server.
# The disk-assisted queue means the sending host buffers rather than
# dropping messages if the collector is briefly unreachable.
*.* action(type="omfwd"
           target="COLLECTOR_ADDRESS" port="514" protocol="tcp"
           queue.type="LinkedList"
           queue.filename="fwd-collector"
           queue.maxDiskSpace="256m"
           queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

Replace COLLECTOR_ADDRESS with your collector's private address, then sudo systemctl restart rsyslog on the sending host. Within seconds a directory named after that host appears under /var/log/remote on the collector.

Encrypted, authenticated delivery with RELP

RELP is what makes rsyslog worth choosing for central collection. The sender holds each message until the collector acknowledges it, so a collector restart, a network blip or a full buffer does not silently eat the log lines that were in flight.

On this appliance RELP is TLS only and requires a client certificate. Copy three files from the collector to the sending host:

sudo scp /etc/rsyslog.d/tls/ca.crt /etc/rsyslog.d/tls/client.crt \
         /etc/rsyslog.d/tls/client.key user@sending-host:/etc/rsyslog.d/tls/

On the sending host, install rsyslog-relp and rsyslog-gnutls, put the three files in /etc/rsyslog.d/tls/ owned root:syslog mode 0640, and create /etc/rsyslog.d/90-forward-relp.conf:

module(load="omrelp")
*.* action(type="omrelp"
           target="COLLECTOR_ADDRESS" port="2514"
           tls="on"
           tls.caCert="/etc/rsyslog.d/tls/ca.crt"
           tls.myCert="/etc/rsyslog.d/tls/client.crt"
           tls.myPrivKey="/etc/rsyslog.d/tls/client.key"
           tls.authMode="name"
           tls.permittedPeer=["cloudimg-rsyslog-collector"]
           queue.type="LinkedList"
           queue.filename="relp-collector"
           queue.maxDiskSpace="256m"
           queue.saveOnShutdown="on"
           action.resumeRetryCount="-1")

Both ends now authenticate each other: the sender checks the collector's certificate against the authority, and the collector checks the sender's.

A certificate signed by anything other than this VM's own authority is refused at the handshake, including one that carries the exact permitted name. That is worth stating plainly because rsyslog offers a certvalid authentication mode that sounds stricter than the mode used here and, on the librelp version Ubuntu 24.04 ships, does not actually enforce the client certificate at all. This image uses name mode, which was measured rejecting both kinds of forgery.

Issuing certificates for more log sources

Every log source can share the one client certificate, or you can issue one each so that a single compromised host can be revoked on its own. Issue from the collector:

sudo openssl req -newkey ec:<(openssl ecparam -name prime256v1) -nodes \
  -keyout /etc/rsyslog.d/tls/host2.key -subj "/O=cloudimg/CN=cloudimg-relp-client" \
  -out /tmp/host2.csr

Keep the common name as cloudimg-relp-client unless you also add the new name to tls.permittedPeer in /etc/rsyslog.d/30-cloudimg-collector-inputs.conf.

Verify the whole thing, at any time

The appliance carries its own end to end check. It stands up simulated remote log sources on loopback addresses, sends a message from a different source address over each of the three transports, and asserts each one landed in the correct file with the right facility, severity and timestamp. It then proves the refusals: a sender outside the permitted networks, and two forged certificates.

sudo /usr/local/sbin/rsyslog-roundtrip-check.sh

A healthy collector prints a single ROUNDTRIP_OK line. Anything else names the specific check that failed.

To see what this machine actually exposes, enumerated from the live socket table rather than from what the configuration claims:

sudo /usr/local/sbin/rsyslog-surface-audit.sh

Retention and disk safety

A collector without retention fills its disk and takes itself down. This one ships with a policy rather than a suggestion:

cat /etc/cloudimg/logrotate-rsyslog-remote.conf
systemctl list-timers cloudimg-rsyslog-logrotate.timer --no-pager

As shipped: rotate when a file passes 128 MB or once an hour, whichever comes first; keep 14 generations; compress everything older than the current one; discard anything over 30 days old.

Two details worth knowing:

The policy is not in /etc/logrotate.d, on purpose. It has its own hourly timer and its own state file. If it were in the usual place, the daily system logrotate run would also process it against a different state file, and the two runs would rotate each other's files out of step.
Hourly, not daily. maxsize only takes effect when logrotate is actually invoked, and a busy collector can write many gigabytes between two daily runs.

To change retention, edit that file. It is ordinary logrotate syntax.

Troubleshooting

Nothing is arriving from a remote host. Work outwards. On the collector, check the sender is permitted (/etc/rsyslog.d/10-cloudimg-allowed-senders.conf must include its address range) and look for the refusal:

sudo journalctl -u rsyslog -n 30 --no-pager | grep -i 'disallowed sender' \
  || echo "no disallowed-sender entries in the last 30 lines"

A UDP message from disallowed sender discarded line means the collector received it and dropped it on policy, which is the sender policy working. No line at all means the packet never arrived, so the problem is the Network Security Group rule or the sending host's own configuration.

RELP connections are refused. The collector logs the reason:

sudo journalctl -u rsyslog -n 40 --no-pager | grep -i 'authentication error' \
  || echo "no RELP authentication errors in the last 40 lines"

certificate validation failed means the client certificate was not issued by this VM's authority. Copy ca.crt, client.crt and client.key from this collector to the sending host again; a certificate from a different cloudimg deployment will not work, which is the point.

Do not delete a live log file. rsyslog keeps the per host output files open for performance, so removing one sends the next messages to a deleted file and they are lost silently until the handle is recycled. Truncate instead:

sudo truncate -s 0 /var/log/remote/some-host/some-program.log

This is also why the retention policy uses copytruncate.

The collector is not running after a reboot. Check the first boot unit; the collector is gated on it by design.

systemctl is-active rsyslog-firstboot.service
ls -l /var/lib/cloudimg/

Local logs stopped. This machine is a log server and also logs itself. Confirm both halves:

logger --tag localcheck "local capture check"
sleep 2
sudo grep -c localcheck /var/log/syslog

Support

Every cloudimg image comes with 24/7 support. Contact us at support@cloudimg.co.uk or visit www.cloudimg.co.uk.

Please include the output of sudo /usr/local/sbin/rsyslog-surface-audit.sh and sudo /usr/local/sbin/rsyslog-roundtrip-check.sh with any report about reception; between them they identify almost every configuration problem immediately.