Fluent Bit on AWS User Guide

Overview

This image runs Fluent Bit, the fast, lightweight and highly scalable open source log processor and forwarder. Fluent Bit collects logs, metrics and traces from many sources, parses and filters them through a configurable pipeline, then routes the records to dozens of destinations including Elasticsearch, OpenSearch, Loki, Kafka, S3, CloudWatch, Splunk and any HTTP endpoint. The image is delivered as a ready-to-use telemetry pipeline appliance: fluent-bit.service is enabled and starts on boot running a self-contained sample pipeline, so the moment an engineer connects over SSH the service is already processing records.

This is a headless log-pipeline service. There is no web interface and no credentials to manage. Fluent Bit runs as a long-lived daemon driven by a simple configuration directory at /etc/fluent-bit/. The built-in HTTP monitoring server is enabled but bound to loopback only (127.0.0.1:2020), so the health, metrics and uptime endpoints answer on the instance without ever being exposed to the network. The instance security group opens port 22 only; you open whatever ingest or forward ports your own pipeline needs.

The filesystem storage buffer and the sample file output both live on a dedicated, independently resizable EBS data volume mounted at /var/lib/fluent-bit. The on-disk pipeline buffer that protects against bursts and brief output outages, and any records the sample pipeline persists, sit on durable storage rather than the operating system disk.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Fluent Bit. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.small or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a few seconds after the instance state becomes Running and the status checks pass; fluent-bit.service is already running its sample pipeline by the time you can connect.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Fluent Bit Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens port 22 from your management network.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.small \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fluent-bit}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

A welcome banner prints the most useful commands, and sudo cat /root/fluent-bit-info.txt shows the full getting-started notes: what is running, the monitoring API endpoints, where the pipeline configuration lives, and the common input, filter and output plugins.

Step 4: Verify the Service

Fluent Bit runs as a system service that is enabled and started on boot. Confirm the daemon is active:

systemctl is-active fluent-bit

That prints active. For the full unit status, including the main PID, memory use and the command line the daemon is running, use:

systemctl status fluent-bit --no-pager

Confirm the installed binary and version, which is pinned to Fluent Bit 5.0.7 on this image:

fluent-bit --version

You should see Fluent Bit v5.0.7 reported. The fluent-bit binary is on the system path; the package installs it under /opt/fluent-bit/bin and the image links it onto /usr/local/bin so it resolves from any login shell.

Step 5: The Monitoring API

Fluent Bit ships with a built-in HTTP monitoring server. On this image it is enabled and bound to loopback only (127.0.0.1:2020), so the endpoints answer on the instance but are never exposed to the network. Check the health endpoint, which returns ok with HTTP 200 while the pipeline is healthy:

curl -s http://127.0.0.1:2020/api/v1/health

The metrics endpoint returns per-plugin JSON with the record and byte counters for every input and output in the running pipeline. Pretty-print it to watch the sample pipeline flushing records:

curl -s http://127.0.0.1:2020/api/v1/metrics | python3 -m json.tool

You will see the dummy input's records counter and the cpu input's counter climbing, and the file and stdout outputs' proc_records rising with errors at 0. That proves the pipeline is actually running and flushing, not merely loaded. The uptime endpoint reports how long the service has been running:

curl -s http://127.0.0.1:2020/api/v1/uptime

The monitoring server emits Prometheus-format metrics at /api/v1/metrics/prometheus for scraping. To expose the API beyond loopback, change HTTP_Listen in the [SERVICE] block of the configuration and open the port in the instance security group yourself; the read-only API stays private by default.

Step 6: Edit the Pipeline

Fluent Bit reads its pipeline from a simple configuration directory at /etc/fluent-bit/. The shipped configuration is a self-contained sample that needs no external dependencies:

ls /etc/fluent-bit/

The main pipeline is in fluent-bit.conf, parsers are defined in parsers.conf, and external plugins are listed in plugins.conf. The sample pipeline pairs a dummy input (a steady synthetic record stream) and a cpu input (real host telemetry) with a file output that writes JSON lines to the data volume and a stdout output visible in the journal:

sed -n '13,57p' /etc/fluent-bit/fluent-bit.conf

To build your own pipeline, replace the sample inputs and outputs with your own. Common input plugins collect from your sources: tail for log files, systemd for the journal, forward and syslog and tcp and http for network ingest, and cpu and mem for host metrics. Chain filter plugins to process records: parser, grep, modify, record_modifier and lua. Fan out to output plugins to ship the records: es/opensearch, loki, kafka, s3, cloudwatch, splunk, http, forward, file and stdout.

After editing the configuration, validate it with a dry run before reloading, then restart the service to apply your changes. These customer-facing commands edit and reload your pipeline:

sudo fluent-bit --dry-run --config /etc/fluent-bit/fluent-bit.conf
sudo systemctl restart fluent-bit

Open whatever ingest or forward ports your pipeline needs in the instance security group. The records the sample file output writes appear as JSON lines on the data volume, alongside the configuration that produced them:

Step 7: The Data Volume

The filesystem storage buffer (storage.path) and the sample file output both live on a dedicated EBS volume mounted at /var/lib/fluent-bit. This keeps the on-disk pipeline buffer and any persisted records off the operating system disk and lets you resize or snapshot the volume independently. Confirm the mount with:

df -h /var/lib/fluent-bit

The [SERVICE] block sets storage.path to this volume and enables filesystem storage, so when an output is briefly unavailable or a burst arrives, Fluent Bit buffers the records to disk here rather than dropping them or growing memory unbounded. To grow the volume, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device. Set storage.type filesystem on your own inputs to give them the same disk-backed buffering.

Support

This image is published and supported by cloudimg. Support covers pipeline design, input and output plugin configuration, parsers and filters, multiline log handling, buffering and backpressure, routing to your observability backend, the monitoring API, and upgrade planning. Contact cloudimg through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.