Observability AWS

Fluent Bit 5.1.4 on AWS User Guide

| Product: Fluent Bit 5.1.4 on Ubuntu 24.04 on AWS

Overview

This guide covers launching and using the Fluent Bit 5.1.4 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg log management listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Fluent Bit is the lightweight, high performance log and metrics processor from the CNCF Fluent project. It collects records from files, the systemd journal, syslog and other agents, parses and filters them, and ships them to destinations such as Amazon S3, CloudWatch Logs, Kinesis Data Firehose, OpenSearch, Loki or Kafka.

A stock Fluent Bit install collects nothing useful and, once you add a network input, accepts records from anyone who can reach the port. The cloudimg image turns it into a secure forward aggregator that your Fluent Bit and Fluentd agents send to:

  • TLS from the first boot. The forward input on TCP 24224 accepts TLS 1.2 or 1.3 only, with a certificate generated on your instance's first boot. Plaintext clients are refused.
  • A shared key unique to your instance. Every agent must also pass the secure forward handshake with a shared key that is generated on first boot and written to the root only file /root/fluent-bit-credentials.txt. A wrong key is refused. No key or certificate is baked into the image.
  • Fail closed. Fluent Bit refuses to start until first boot has created the certificate and the key, so the input is never open without them.
  • Unprivileged and sandboxed. Fluent Bit runs as the fluentbit system user with no capabilities, a read only file system apart from its log and state directories, and the systemd sandbox switched on.
  • Monitoring on loopback only. The HTTP monitoring API (health, metrics, uptime) listens on 127.0.0.1:2020.
  • Closed by default. The listing security group opens SSH only; you open TCP 24224 to your own networks when you are ready.
  • IAM role ready. Fluent Bit uses the instance's IAM role via IMDSv2; no other service on the image can reach IMDS. Attach a role and its S3, CloudWatch Logs and Kinesis Data Firehose outputs work without keys on disk.
  • Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Fluent Bit 5.1.4 (Apache-2.0) from the official packages.fluentbit.io repository in /opt/fluent-bit, the aggregator pipeline in /etc/fluent-bit, a self test command, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

  • An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
  • A security group that allows TCP 22 (SSH) from your own networks. Add TCP 24224 (the forward input) only from the VPC or VPN ranges your agents send from, never from 0.0.0.0/0. Fluent Bit has no web interface, so no other port is needed.
  • To ship to AWS services, an IAM role (instance profile) with the permissions your outputs need (for example logs:CreateLogStream and logs:PutLogEvents, or s3:PutObject).

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Fluent Bit 5.1.4 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair, a security group as above and, if you will use the AWS outputs, your IAM role. Under Advanced details, set Metadata version to V2 only (token required) and Metadata response hop limit to 1.

The AMI is IMDSv2 only. Set the hop limit to 1 at launch (an AMI cannot carry it), so a container on the instance can never borrow the instance role. With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --iam-instance-profile Name=<your-instance-profile> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled,HttpPutResponseHopLimit=1 \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fluent-bit}]'

Step 2: Connect to your instance

Version SSH user
Fluent Bit 5.1.4 on Ubuntu 24.04 ubuntu 
ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the service

First boot generates the certificate and the shared key, then Fluent Bit starts as the fluentbit user:

systemctl is-active fluent-bit.service
sudo test -f /var/lib/cloudimg/fluent-bit-firstboot.done && echo FIRSTBOOT_DONE
/opt/fluent-bit/bin/fluent-bit --version | head -1
ps -o user=,args= -C fluent-bit

active
FIRSTBOOT_DONE
Fluent Bit v5.1.4
fluentb+ /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf

The health endpoint of the monitoring API answers on loopback:

curl -s http://127.0.0.1:2020/api/v1/health; echo

ok

Only SSH and the forward input listen on public addresses; the monitoring API and the local DNS resolver stay on loopback:

sudo ss -Hltn | awk '{print $4}' | sort -u

0.0.0.0:22
0.0.0.0:24224
127.0.0.1:2020
127.0.0.53%lo:53
127.0.0.54:53
[::]:22

Step 4: Get the connection details

The address, port and certificate path your agents need are in the root only credentials file:

sudo grep -E '^FLUENT_BIT_(FORWARD_HOST|FORWARD_PORT|SERVER_HOSTNAME|CA_CERT)=' /root/fluent-bit-credentials.txt

FLUENT_BIT_FORWARD_HOST=<private-ip>
FLUENT_BIT_FORWARD_PORT=24224
FLUENT_BIT_SERVER_HOSTNAME=ip-<private-ip-dashed>.ec2.internal
FLUENT_BIT_CA_CERT=/etc/fluent-bit/tls/forward.crt

The shared key is the FLUENT_BIT_SHARED_KEY line of the same file. Read it when you configure an agent, sudo grep '^FLUENT_BIT_SHARED_KEY=' /root/fluent-bit-credentials.txt, and keep it in your secrets store; it is unique to this instance.

The certificate names the instance's private and public addresses and EC2 host names. Agents should send to the private address: it does not change when the instance stops and starts.

sudo openssl x509 -in /etc/fluent-bit/tls/forward.crt -noout -subject -ext subjectAltName -enddate

subject=CN = ip-<private-ip-dashed>.ec2.internal, O = cloudimg
X509v3 Subject Alternative Name:
    IP Address:<private-ip>, IP Address:<public-ip>, DNS:ip-<private-ip-dashed>.ec2.internal, DNS:ec2-<public-ip-dashed>.compute-1.amazonaws.com, IP Address:127.0.0.1, DNS:localhost
notAfter=<about 27 months after first boot>

The credentials file, the input configuration that holds the key and the private key are not readable by other users:

sudo stat -c '%n  mode=%a  owner=%U:%G' /root/fluent-bit-credentials.txt /etc/fluent-bit/forward-secure.conf /etc/fluent-bit/tls/forward.key

/root/fluent-bit-credentials.txt  mode=600  owner=root:root
/etc/fluent-bit/forward-secure.conf  mode=640  owner=root:fluentbit
/etc/fluent-bit/tls/forward.key  mode=640  owner=root:fluentbit

Step 5: Prove the input on the instance

fluent-bit-forward-test sends one record to the local forward input with a throwaway Fluent Bit client and checks that it arrived. Run it with TLS and the right key, then with a wrong key and without TLS; only the first is accepted:

sudo fluent-bit-forward-test
sudo fluent-bit-forward-test --wrong-key || echo "refused as expected"
sudo fluent-bit-forward-test --no-tls || echo "refused as expected"

ACCEPTED cloudimg-forward-test-<timestamp>
NOT-ACCEPTED cloudimg-forward-test-<timestamp>-wrong-key
refused as expected
NOT-ACCEPTED cloudimg-forward-test-<timestamp>-no-tls
refused as expected

The forward input logs why it refused the wrong key:

sudo journalctl -u fluent-bit.service --no-pager -o cat | grep -a 'shared_key mismatch' | tail -1

[<date> <time>] [error] [input:forward:forward.0] shared_key mismatch

Records the aggregator receives are written to /var/log/fluent-bit/records.log until you add your own outputs (Step 8):

sudo tail -n 1 /var/log/fluent-bit/records.log

cloudimg.forward.test: [<epoch>.<ns>, {"message":"cloudimg-forward-test-<timestamp>"}]

Step 6: Open the port to your agents

Add an inbound rule for TCP 24224 from the address range your agents send from, for example your VPC CIDR:

aws ec2 authorize-security-group-ingress --group-id <your-security-group> \
  --protocol tcp --port 24224 --cidr <your-vpc-cidr>

Copy the instance's certificate to each agent host so the agent can verify it is talking to your aggregator:

ssh -i <your-key.pem> ubuntu@<public-ip> 'sudo cat /etc/fluent-bit/tls/forward.crt' > forward.crt

Step 7: Point your agents at the aggregator

Fluent Bit agent (classic configuration). Self_Hostname names the agent; the shared key is the one from Step 4:

[OUTPUT]
    Name          forward
    Match         *
    Host          <private-ip>
    Port          24224
    tls           on
    tls.verify    on
    tls.ca_file   /etc/fluent-bit/forward.crt
    Shared_Key    <FLUENT_BIT_SHARED_KEY>
    Self_Hostname <agent-name>

If the agent connects by a name that is not in the certificate (for example your own DNS record), add tls.vhost ip-<private-ip-dashed>.ec2.internal or issue a certificate for that name (Step 9).

Fluentd agent (fluent-plugin-forward is built in):

<match **>
  @type forward
  transport tls
  tls_cert_path /etc/fluent/forward.crt
  tls_verify_hostname true
  <security>
    self_hostname <agent-name>
    shared_key <FLUENT_BIT_SHARED_KEY>
  </security>
  <server>
    host <private-ip>
    port 24224
  </server>
</match>

Restart the agent; its records appear in /var/log/fluent-bit/records.log on the aggregator with the agent's tags.

Step 8: Ship records on to AWS or another destination

The pipeline is /etc/fluent-bit/fluent-bit.conf. Its [OUTPUT] writes everything to /var/log/fluent-bit/records.log; replace it or add outputs next to it. The forward input itself lives in forward-secure.conf, which first boot generated; leave it as it is.

Fluent Bit uses the instance's IAM role via IMDSv2; no other service on the image can reach IMDS. With a role attached, the AWS outputs need no access keys. CloudWatch Logs:

[OUTPUT]
    Name              cloudwatch_logs
    Match             *
    region            <region>
    log_group_name    /cloudimg/fluent-bit
    log_stream_prefix agent-
    auto_create_group On

Amazon S3:

[OUTPUT]
    Name            s3
    Match           *
    region          <region>
    bucket          <your-bucket>
    total_file_size 50M
    upload_timeout  5m
    s3_key_format   /fluent-bit/%Y/%m/%d/$TAG-%H%M%S

Edit the file with sudo nano /etc/fluent-bit/fluent-bit.conf, check it, and restart the service:

sudo -u fluentbit /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
sudo systemctl restart fluent-bit.service

Watch for output errors in the journal: sudo journalctl -u fluent-bit.service -f.

Step 9: Use your own DNS name or certificate

To present a certificate from your own certificate authority (for example one that names your DNS record), replace /etc/fluent-bit/tls/forward.crt and /etc/fluent-bit/tls/forward.key, keep their ownership and modes, then restart:

sudo install -o root -g fluentbit -m 0640 your.key /etc/fluent-bit/tls/forward.key
sudo install -o root -g root -m 0644 your.crt /etc/fluent-bit/tls/forward.crt
sudo systemctl restart fluent-bit.service

Give your agents the new CA certificate instead of forward.crt.

To rotate the shared key, put a new value on the Shared_Key line of /etc/fluent-bit/forward-secure.conf (generate one with openssl rand -hex 20), restart the service and update every agent. The credentials file keeps the first boot value, so update your own records.

Step 10: Security notes

The service runs with no capabilities, cannot gain privileges and sees the file system read only:

systemctl show -p User -p NoNewPrivileges -p ProtectSystem -p CapabilityBoundingSet fluent-bit.service

CapabilityBoundingSet=
User=fluentbit
ProtectSystem=strict
NoNewPrivileges=yes

Fluent Bit is the one service allowed to reach the instance metadata service, because its AWS outputs sign requests with the instance role, so it carries no metadata deny rule:

systemctl show -p IPAddressDeny fluent-bit.service

IPAddressDeny=

More:

  • The forward input accepts TLS 1.2 and 1.3 only; TLS 1.1 and plaintext connections are refused.
  • The key is never on a command line or in the unit's environment; Fluent Bit reads it from forward-secure.conf, which only root and the fluentbit group can read.
  • Keep TCP 24224 restricted to your agents' networks in the security group. Use the private address; there is no need to expose the aggregator to the internet.
  • The instance is IMDSv2 only. Launch it with a metadata hop limit of 1 (Step 1) so containers cannot reach the role.
  • If you add an input that reads another agent protocol (syslog, HTTP, OpenTelemetry), bind it to 127.0.0.1 or protect it with TLS before opening its port.

Server components

Component Version Purpose
Fluent Bit 5.1.4 Log and metrics processor, forward aggregator
OpenSSL 3.0.13 (Ubuntu) TLS for the forward input
Ubuntu 24.04 LTS Operating system

Filesystem layout

Path Purpose
/opt/fluent-bit/bin/fluent-bit The Fluent Bit binary
/etc/fluent-bit/fluent-bit.conf The pipeline: service settings and outputs
/etc/fluent-bit/forward-secure.conf The forward input with the shared key, generated on first boot
/etc/fluent-bit/tls The instance's certificate and private key
/var/log/fluent-bit/records.log Records received until you add outputs (rotated daily, 7 kept)
/var/lib/fluent-bit Fluent Bit state directory
/usr/local/sbin/fluent-bit-forward-test The self test command
/root/fluent-bit-credentials.txt Address, port, shared key and certificate path, root only

Managing the service

systemctl show -p Id -p ActiveState -p SubState fluent-bit.service

Id=fluent-bit.service
ActiveState=active
SubState=running

Restart after a configuration change with sudo systemctl restart fluent-bit.service. The cloudimg settings (user, sandbox, first boot gate) live in the drop-in /etc/systemd/system/fluent-bit.service.d/10-cloudimg.conf.

Backing up

The configuration, the certificate and the key are what make the aggregator yours:

sudo tar czf fluent-bit-backup.tar.gz /etc/fluent-bit /root/fluent-bit-credentials.txt

Store the archive somewhere private: it holds the shared key and the private key. Restore by extracting it over the same paths and restarting the service.

Troubleshooting

An agent reports shared_key mismatch or the aggregator logs it. The agent's key differs from the Shared_Key line in /etc/fluent-bit/forward-secure.conf. Copy the value again from the credentials file.

An agent times out connecting. Check that the security group allows TCP 24224 from the agent's address, that the agent has tls on, and that it trusts forward.crt.

An agent reports a certificate verification error. It is connecting by a name or address that is not in the certificate. Use the private address, set tls.vhost to the name in the certificate, or install your own certificate (Step 9).

Fluent Bit did not start. It waits for first boot. Check the first boot log:

sudo journalctl -u fluent-bit-firstboot.service --no-pager -n 20

Support and licensing

Fluent Bit is open source under the Apache License 2.0; the image ships the unmodified upstream 5.1.4 package. There is no per user or per CPU licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

  • Email: support@cloudimg.co.uk
  • Website: https://www.cloudimg.co.uk

All product and company names are trademarks or registered trademarks of their respective holders. cloudimg is not affiliated with or endorsed by the Fluent project or the Cloud Native Computing Foundation; the name is used only to identify the unmodified upstream software.