Fluentd on AWS User Guide

Overview

This image runs Fluentd, the open source unified logging layer: a flexible, plugin-driven log collector that unifies log collection, parsing, buffering and routing across your whole stack. Fluentd ingests logs from many sources, structures them as JSON, runs them through a configurable pipeline of filters, then routes the records to hundreds of destinations including Elasticsearch, OpenSearch, Loki, Kafka, S3, CloudWatch, MongoDB and any HTTP endpoint. The image is delivered as a ready-to-use log-pipeline appliance: fluentd.service is enabled and starts on boot running a self-contained sample pipeline, so the moment an engineer connects over SSH the daemon is already processing records.

This is a headless log-pipeline service. There is no web interface and no credentials to manage. Fluentd runs as a long-lived daemon installed from the official fluent-package distribution and driven by a simple configuration file at /etc/fluent/fluentd.conf. The built-in monitor_agent HTTP endpoint is enabled but bound to loopback only (127.0.0.1:24220), so the plugins and configuration endpoints answer on the instance without ever being exposed to the network. The instance security group opens port 22 only; you open whatever ingest or forward ports your own pipeline needs.

The file output buffer and the sample persisted log both live on a dedicated, independently resizable EBS data volume mounted at /var/lib/fluentd. The on-disk pipeline buffer that protects against bursts and brief destination outages, and any records the sample pipeline persists, sit on durable storage rather than the operating system disk.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Fluentd. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of t3.small or larger. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes a few seconds after the instance state becomes Running and the status checks pass; fluentd.service is already running its sample pipeline by the time you can connect.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Fluentd Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens port 22 from your management network.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type t3.small \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=fluentd}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

A welcome banner prints the most useful commands, and sudo cat /root/fluentd-info.txt shows the full getting-started notes: what is running, the monitor_agent endpoints, where the pipeline configuration lives, the buffer path, the common input, filter and output plugins, and how to install plugins with fluent-gem.

Step 4: Verify the Service

Fluentd runs as a system service that is enabled and started on boot. Confirm the daemon is active:

systemctl is-active fluentd

That prints active. For the full unit status, including the main PID, memory use and the command line the daemon is running, use:

systemctl status fluentd --no-pager

Confirm the installed binary and the Fluentd core version, which is pinned to the fluent-package 6.0.3 LTS distribution on this image:

fluentd --version

You should see fluentd 1.19.2 reported. The fluentd binary is on the system path; the package installs it under /opt/fluent/bin and the image links it onto /usr/local/bin so it resolves from any login shell. The plugin installer fluent-gem is linked the same way.

Step 5: The Monitoring Endpoint

Fluentd ships with a built-in monitor_agent HTTP endpoint. On this image it is enabled and bound to loopback only (127.0.0.1:24220), so the endpoints answer on the instance but are never exposed to the network. The plugins endpoint returns per-plugin JSON with the live buffer, retry and emitted-record counters for every input and output in the running pipeline:

curl -s http://127.0.0.1:24220/api/plugins.json | python3 -m json.tool

You will see the dummy input's emit_records counter and the file and stdout outputs' counters climbing across calls, with retry_count at 0 and buffer_queue_length at 0. That proves the pipeline is actually running and flushing records, not merely loaded. The configuration endpoint returns the running configuration as JSON:

curl -s http://127.0.0.1:24220/api/config.json | python3 -m json.tool

To expose the monitoring endpoint beyond loopback, change the bind address in the monitor_agent source of the configuration and open the port in the instance security group yourself; the read-only endpoint stays private by default.

Step 6: Edit the Pipeline

Fluentd reads its pipeline from a simple configuration file at /etc/fluent/fluentd.conf. The shipped configuration is a self-contained sample that needs no external dependencies:

ls /etc/fluent/

The shipped sample pairs a monitor_agent source (the loopback metrics endpoint), a dummy source (a steady synthetic record stream) and a forward source (for local testing) with a copy match that fans the records out to a buffered file output on the data volume and a stdout output visible in the journal. Read the sources and matches:

sed -n '14,55p' /etc/fluent/fluentd.conf

To build your own pipeline, replace the sample sources and matches with your own. Common source plugins collect from your inputs: tail for log files, systemd for the journal, forward, syslog, tcp and http for network ingest, and exec for command output. Chain filter plugins to process records: parser, grep, record_transformer and geoip. Fan out to output plugins to ship the records: elasticsearch/opensearch, loki, kafka, s3, cloudwatch, mongo, http, forward, file and stdout.

A vast ecosystem of input, filter and output plugins installs in seconds with the bundled fluent-gem tool. To install a plugin, run sudo /opt/fluent/bin/fluent-gem install fluent-plugin-<name> and add its source, filter or match block to the configuration.

After editing the configuration, validate it with a dry run before reloading, then restart the service to apply your changes. These customer-facing commands edit and reload your pipeline:

sudo /opt/fluent/bin/fluentd --dry-run -c /etc/fluent/fluentd.conf
sudo systemctl restart fluentd

The dry run parses the configuration and reports finished dry run mode when it is valid, then exits without starting the daemon. Open whatever ingest or forward ports your pipeline needs in the instance security group. The records the sample file output writes appear as JSON lines on the data volume:

Step 7: The Data Volume

The file output buffer (<buffer> path) and the sample file output both live on a dedicated EBS volume mounted at /var/lib/fluentd. This keeps the on-disk pipeline buffer and any persisted records off the operating system disk and lets you resize or snapshot the volume independently. Confirm the mount and watch the records flow with:

df -h /var/lib/fluentd
sudo tail -n 4 /var/lib/fluentd/sample-pipeline.*.log

The sample match writes its records to /var/lib/fluentd/sample-pipeline.<date>.log and buffers them under /var/lib/fluentd/buffer/sample, so when an output is briefly unavailable or a burst arrives, Fluentd buffers the records to disk here rather than dropping them or growing memory unbounded. To grow the volume, expand the EBS volume in the AWS console, then grow the filesystem on the instance with sudo resize2fs on the underlying device. Point your own outputs' <buffer> blocks at this volume to give them the same disk-backed buffering.

Support

This image is published and supported by cloudimg. Support covers pipeline design, input and output plugin configuration, parsers and filters, multiline log handling, buffering and backpressure, routing to your observability backend, the monitoring endpoint, and upgrade planning. Contact cloudimg through the support channel listed on the AWS Marketplace listing.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.