Observability AWS

Fluentd 1.19.4 on AWS User Guide

| Product: Fluentd 1.19.4 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Fluentd 1.19.4 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg log management listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Fluentd is the open source data collector from the CNCF Fluent project. It gathers logs from agents, files, syslog and the systemd journal, parses, filters and routes them, and delivers them to destinations such as Amazon S3, OpenSearch, Elasticsearch, Kafka or another Fluentd. This image ships fluent-package 6.0.5, the supported long term support (LTS) distribution of Fluentd, which bundles Fluentd 1.19.4 with its own Ruby runtime and a set of common plugins.

A stock Fluentd install only runs a demo pipeline and, once you add a network input, accepts records from anyone who can reach the port. The cloudimg image turns it into a secure forward aggregator that your Fluentd and Fluent Bit agents send to:

  • TLS from the first boot. The forward input on TCP 24224 accepts TLS 1.2 or 1.3 only, with a certificate generated on your instance's first boot. Plaintext clients are refused.
  • A shared key unique to your instance. Every agent must also pass the secure forward handshake with a shared key that is generated on first boot and written to the root only file /root/fluentd-credentials.txt. A wrong key is refused. No key or certificate is baked into the image.
  • Fail closed. Fluentd refuses to start until first boot has created the certificate and the key, so the input is never open without them.
  • Unprivileged and sandboxed. Fluentd runs as the package's _fluentd system user with no capabilities, a read only file system apart from its log directories, and the systemd sandbox switched on.
  • Monitoring on loopback only. The monitor_agent HTTP API listens on 127.0.0.1:24220.
  • Closed by default. The listing security group opens SSH only; you open TCP 24224 to your own networks when you are ready.
  • IAM role ready. Fluentd uses the instance's IAM role via IMDSv2; no other service on the image can reach IMDS. Attach a role and the bundled Amazon S3 output works without keys on disk.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Fluentd 1.19.4 in fluent-package 6.0.5 LTS (Apache-2.0) from the official fluentd.cdn.cncf.io repository in /opt/fluent, the aggregator pipeline in /etc/fluent, a self test command, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
  • A security group that allows TCP 22 (SSH) from your own networks. Add TCP 24224 (the forward input) only from the VPC or VPN ranges your agents send from, never from 0.0.0.0/0. Fluentd has no web interface, so no other port is needed.
  • To ship to Amazon S3, an IAM role (instance profile) with the permissions your output needs (for example s3:PutObject on your bucket).

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Fluentd 1.19.4 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair, a security group as above and, if you will use the S3 output, your IAM role. Under Advanced details, set Metadata version to V2 only (token required) and Metadata response hop limit to 1.

The AMI is IMDSv2 only. Set the hop limit to 1 at launch (an AMI cannot carry it), so a container on the instance can never borrow the instance role. With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --iam-instance-profile Name=<your-instance-profile> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled,HttpPutResponseHopLimit=1 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fluentd}]'

Step 2: Connect to your instance

Version SSH user
Fluentd 1.19.4 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the service

First boot generates the certificate and the shared key, then Fluentd starts as the _fluentd user:

systemctl is-active fluentd.service
sudo test -f /var/lib/cloudimg/fluentd-firstboot.done && echo FIRSTBOOT_DONE
/opt/fluent/bin/fluentd --version
dpkg-query -W -f='fluent-package ${Version}\n' fluent-package

active
FIRSTBOOT_DONE
fluentd 1.19.4
fluent-package 6.0.5-1

The monitor_agent API answers on loopback and lists the running plugins: the secure forward input, the file output for received records, and the two housekeeping inputs fluent-package adds (one warns about obsolete plugins, the other logs when a newer fluent-package LTS release is available):

curl -s http://127.0.0.1:24220/api/plugins.json | /opt/fluent/bin/ruby -rjson -e 'JSON.parse(STDIN.read)["plugins"].each { |p| puts "#{p["plugin_category"]} #{p["type"]}" }'

input monitor_agent
input forward
input obsolete_plugins
input fluent_package_update_notifier
output file
output null

Only SSH and the forward input listen on public addresses; the monitoring API and the local DNS resolver stay on loopback:

sudo ss -Hltn | awk '{print $4}' | sort -u

0.0.0.0:22
0.0.0.0:24224
127.0.0.1:24220
127.0.0.53%lo:53
127.0.0.54:53
[::]:22

The forward input also answers the UDP heartbeat on 24224, which carries no data. You only need to open TCP 24224 for agents.

Step 4: Get the connection details

The address, port, host name and certificate path your agents need are in the root only credentials file:

sudo grep -E '^FLUENTD_(FORWARD_HOST|FORWARD_PORT|SERVER_HOSTNAME|CA_CERT)=' /root/fluentd-credentials.txt

FLUENTD_FORWARD_HOST=<private-ip>
FLUENTD_FORWARD_PORT=24224
FLUENTD_SERVER_HOSTNAME=ip-<private-ip-dashed>.ec2.internal
FLUENTD_CA_CERT=/etc/fluent/tls/forward.crt

The shared key is the FLUENTD_SHARED_KEY line of the same file. Read it when you configure an agent, sudo grep '^FLUENTD_SHARED_KEY=' /root/fluentd-credentials.txt, and keep it in your secrets store; it is unique to this instance.

The certificate names the instance's private and public addresses and EC2 host names. Agents should connect to the private DNS name (FLUENTD_SERVER_HOSTNAME): it is in the certificate, it resolves to the private address inside the VPC, and it does not change when the instance stops and starts.

sudo openssl x509 -in /etc/fluent/tls/forward.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ip-<private-ip-dashed>.ec2.internal, O = cloudimg
X509v3 Subject Alternative Name:
    IP Address:<private-ip>, IP Address:<public-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:127.0.0.1, DNS:localhost
notAfter=<about 27 months after first boot>

The credentials file, the input configuration that holds the key and the private key are not readable by other users:

sudo stat -c '%n  mode=%a  owner=%U:%G' /root/fluentd-credentials.txt /etc/fluent/forward-secure.conf /etc/fluent/tls/forward.key

/root/fluentd-credentials.txt  mode=600  owner=root:root
/etc/fluent/forward-secure.conf  mode=640  owner=root:_fluentd
/etc/fluent/tls/forward.key  mode=640  owner=root:_fluentd

Step 5: Prove the input on the instance

fluentd-forward-test sends records to the local forward input with a throwaway Fluentd client and checks that one arrived. Run it with TLS and the right key, then with a wrong key and without TLS; only the first is accepted (the refused runs take about 20 seconds each):

sudo fluentd-forward-test
sudo fluentd-forward-test --wrong-key || echo "refused as expected"
sudo fluentd-forward-test --no-tls || echo "refused as expected"

ACCEPTED cloudimg-forward-test-<timestamp>
NOT-ACCEPTED cloudimg-forward-test-<timestamp>-wrong-key
refused as expected
NOT-ACCEPTED cloudimg-forward-test-<timestamp>-no-tls
refused as expected

Fluentd logs why it refused the wrong key:

sudo grep -a 'Shared key mismatch' /var/log/fluent/fluentd.log | tail -1

<date> <time> +0000 [warn]: #0 Shared key mismatch address="127.0.0.1" hostname="forward-test-client"

Records the aggregator receives are written to /var/log/fluentd/records.log (time, tag, record) until you add your own outputs (Step 8):

sudo tail -n 1 /var/log/fluentd/records.log

<date>T<time>+00:00 cloudimg.forward.test   {"message":"cloudimg-forward-test-<timestamp>"}

Step 6: Open the port to your agents

Add an inbound rule for TCP 24224 from the address range your agents send from, for example your VPC CIDR:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 24224 --cidr <your-vpc-cidr>

Copy the instance's certificate to each agent host so the agent can verify it is talking to your aggregator:

ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/fluent/tls/forward.crt' > forward.crt

Step 7: Point your agents at the aggregator

Fluentd agent (the forward output is built in). self_hostname names the agent; the shared key is the one from Step 4:

<match **>
  @type forward
  transport tls
  tls_ca_cert_path /etc/fluent/forward.crt
  tls_verify_hostname true
  <security>
    self_hostname <agent-name>
    shared_key <FLUENTD_SHARED_KEY>
  </security>
  <server>
    host ip-<private-ip-dashed>.ec2.internal
    port 24224
  </server>
</match>

If the agent must connect by IP address instead, add name <private-ip> inside <server>: when host is an IP address, Fluentd verifies the certificate against name, and without it the TLS connection fails.

Fluent Bit agent (classic configuration):

[OUTPUT]
    Name          forward
    Match         *
    Host          ip-<private-ip-dashed>.ec2.internal
    Port          24224
    tls           on
    tls.verify    on
    tls.ca_file   /etc/fluent-bit/forward.crt
    Shared_Key    <FLUENTD_SHARED_KEY>
    Self_Hostname <agent-name>

Restart the agent; its records appear in /var/log/fluentd/records.log on the aggregator with the agent's tags.

Step 8: Ship records on to Amazon S3 or another destination

The pipeline is /etc/fluent/fluentd.conf. Its final <match **> writes everything to /var/log/fluentd/records.log; replace it or put your own <match> blocks above it. The forward input itself lives in forward-secure.conf, which first boot generated; leave it as it is.

Fluentd uses the instance's IAM role via IMDSv2; no other service on the image can reach IMDS. With a role attached, the bundled S3 output (fluent-plugin-s3) needs no access keys:

<match **>
  @type s3
  s3_region <region>
  s3_bucket <your-bucket>
  path fluentd/%Y/%m/%d/
  <buffer tag,time>
    @type file
    path /var/log/fluent/buffer/s3
    timekey 300
    timekey_wait 60
    chunk_limit_size 64m
  </buffer>
</match>

fluent-package also bundles outputs for OpenSearch, Elasticsearch, Kafka, OpenTelemetry and Prometheus. Other destinations, for example CloudWatch Logs, are added as plugins: sudo /opt/fluent/bin/fluent-gem install fluent-plugin-cloudwatch-logs. Plugins from RubyGems run with Fluentd's privileges, so install only the ones you trust.

Edit the file with sudo nano /etc/fluent/fluentd.conf, check it, and restart the service:

sudo -u _fluentd /opt/fluent/bin/fluentd -c /etc/fluent/fluentd.conf --dry-run
sudo systemctl restart fluentd.service

Buffers on disk must live under /var/log/fluent, /var/log/fluentd or /var/lib/fluent: the sandbox keeps the rest of the file system read only for Fluentd. Watch for output errors in the Fluentd log: sudo tail -f /var/log/fluent/fluentd.log.

Step 9: Use your own DNS name or certificate

To present a certificate from your own certificate authority (for example one that names your DNS record), replace /etc/fluent/tls/forward.crt and /etc/fluent/tls/forward.key, keep their ownership and modes, then restart:

sudo install -o root -g _fluentd -m 0640 your.key /etc/fluent/tls/forward.key
sudo install -o root -g root -m 0644 your.crt /etc/fluent/tls/forward.crt
sudo systemctl restart fluentd.service

Give your agents the new CA certificate instead of forward.crt.

To rotate the shared key, put a new value on the shared_key line of /etc/fluent/forward-secure.conf (generate one with openssl rand -hex 20), restart the service and update every agent. The credentials file keeps the first boot value, so update your own records.

Step 10: Security notes

The service runs with no capabilities, cannot gain privileges and sees the file system read only:

systemctl show -p User -p NoNewPrivileges -p ProtectSystem -p CapabilityBoundingSet fluentd.service

CapabilityBoundingSet=
User=_fluentd
ProtectSystem=strict
NoNewPrivileges=yes

Fluentd is the one service allowed to reach the instance metadata service, because its S3 output signs requests with the instance role, so it carries no metadata deny rule:

systemctl show -p IPAddressDeny fluentd.service

IPAddressDeny=

More:

  • The forward input accepts TLS 1.2 and 1.3 only; older protocols and plaintext connections are refused.
  • The key is never on a command line, in the unit's environment or in the monitoring API; Fluentd reads it from forward-secure.conf, which only root and the _fluentd group can read, and masks it in its own log.
  • Fluentd logs a warning at start that private_key_passphrase is recommended. The private key is protected by its file permissions instead (0640, root and the _fluentd group only); a passphrase would have to sit next to it in the configuration.
  • Keep TCP 24224 restricted to your agents' networks in the security group. Use the private DNS name; there is no need to expose the aggregator to the internet.
  • The instance is IMDSv2 only. Launch it with a metadata hop limit of 1 (Step 1) so containers cannot reach the role.
  • If you add an input that reads another agent protocol (syslog, HTTP, OpenTelemetry), bind it to 127.0.0.1 or protect it with TLS before opening its port.

Server components

Component Version Purpose
Fluentd 1.19.4 Log collector and router, forward aggregator
fluent-package 6.0.5 LTS Fluentd distribution with its own Ruby 3.4 runtime and bundled plugins
OpenSSL 3.0.13 (Ubuntu) TLS for the forward input
Ubuntu 24.04 LTS Operating system

Filesystem layout

Path Purpose
/opt/fluent/bin/fluentd The Fluentd binary (fluent-package)
/etc/fluent/fluentd.conf The pipeline: monitoring API and outputs
/etc/fluent/forward-secure.conf The forward input with the shared key, generated on first boot
/etc/fluent/tls The instance's certificate and private key
/var/log/fluentd/records.log Records received until you add outputs (rotated daily, 7 kept)
/var/log/fluent/fluentd.log Fluentd's own log
/usr/local/sbin/fluentd-forward-test The self test command
/root/fluentd-credentials.txt Address, host name, port, shared key and certificate path, root only

Managing the service

systemctl show -p Id -p ActiveState -p SubState fluentd.service

Id=fluentd.service
ActiveState=active
SubState=running

Restart after a configuration change with sudo systemctl restart fluentd.service. The cloudimg settings (sandbox, first boot gate) live in the drop-in /etc/systemd/system/fluentd.service.d/10-cloudimg.conf.

Backing up

The configuration, the certificate and the key are what make the aggregator yours:

sudo tar czf fluentd-backup.tar.gz /etc/fluent /root/fluentd-credentials.txt

Store the archive somewhere private: it holds the shared key and the private key. Restore by extracting it over the same paths and restarting the service.

Troubleshooting

The aggregator logs Shared key mismatch. The agent's key differs from the shared_key line in /etc/fluent/forward-secure.conf. Copy the value again from the credentials file.

An agent times out connecting. Check that the security group allows TCP 24224 from the agent's address, that the agent uses TLS (transport tls or tls on), and that it trusts forward.crt.

A Fluentd agent fails with IPAddr::AddressFamilyError or a certificate verification error. It connects by an IP address without name, or by a name that is not in the certificate. Use the private DNS name, add name <private-ip> to <server>, or install your own certificate (Step 9).

Fluentd did not start. It waits for first boot. Check the first boot log:

sudo journalctl -u fluentd-firstboot.service --no-pager -n 20

Support and licensing

Fluentd and fluent-package are open source under the Apache License 2.0; the image ships the unmodified upstream fluent-package 6.0.5 package. There is no per user or per CPU licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

  • Email: support@cloudimg.co.uk
  • Website: https://www.cloudimg.co.uk

All product and company names are trademarks or registered trademarks of their respective holders. cloudimg is not affiliated with or endorsed by the Fluentd project, the Fluent project or the Cloud Native Computing Foundation; the name is used only to identify the unmodified upstream software.