Fluentd on Ubuntu 24.04 on Azure User Guide

Overview

Fluentd is the open-source unified logging layer from the Fluent project (a CNCF graduated project). It collects data from many sources, structures it as JSON, filters and buffers it, and routes it to dozens of destinations through a rich plugin ecosystem. The cloudimg image installs Fluentd from the official Fluentd fluent-package LTS apt repository as /opt/fluent/bin/fluentd, ships a clear ready-to-run sample pipeline, and enables the built-in monitor_agent HTTP endpoint so you can inspect health and per-plugin metrics from the moment the VM boots. Backed by 24/7 cloudimg support.

What is included:

• Fluentd 1.16 (fluent-package 5 LTS, Apache-2.0) from the official Fluentd apt repository
• The package fluentd.service systemd unit, enabled and running on boot
• A sample pipeline at /etc/fluent/fluentd.conf: a dummy input, a record_transformer filter, and a copy output fanning out to a rotating file under /var/log/fluentd/ and to stdout
• The built-in monitor_agent HTTP endpoint on 127.0.0.1:24220 (/api/plugins.json, /api/config.json)
• 24/7 cloudimg support

This is a headless service product: the monitor_agent HTTP endpoint binds to 127.0.0.1 only, so no inbound application port is needed beyond SSH.

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet + subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is a good starting point. NSG inbound: allow 22/tcp from your management network. No inbound application ports are required.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Fluentd by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22). Then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name fluentd \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm Fluentd is installed and running

Check the version and that the service is active:

fluentd --version
systemctl is-active fluentd

fluentd --version reports fluent-package 5.0.x fluentd 1.16.x, and systemctl is-active fluentd prints active.

Step 5 - Review the sample pipeline

The ready-to-run pipeline lives at /etc/fluent/fluentd.conf. It enables the monitor_agent HTTP endpoint, generates a sample record every second with the dummy input (a guaranteed source of records), stamps each record with the hostname and a pipeline tag using record_transformer, and fans the matched stream out to a file and to stdout with the copy output:

cat /etc/fluent/fluentd.conf

Step 6 - Inspect live plugin metrics

Fluentd's built-in monitor_agent exposes /api/plugins.json, which returns 200 JSON listing every configured plugin and its per-plugin counters. Query it from the VM and pretty-print with jq:

curl -s http://127.0.0.1:24220/api/plugins.json | jq '.plugins[] | {type, plugin_id}'

The output lists the monitor_agent source, the dummy input, the record_transformer filter, and the copy, file and stdout output plugins.

Step 7 - Watch records flow through the pipeline

The file output writes every record to /var/log/fluentd/records.<date>.log. Tail it to see the live stream - each line is a JSON record stamped with your hostname and the cloudimg-sample pipeline tag:

tail -n 5 /var/log/fluentd/records.*.log

Step 8 - Customise the pipeline

Edit /etc/fluent/fluentd.conf to point Fluentd at your own sources and destinations, then restart the service. A few common building blocks:

# Tail application log files
<source>
  @type tail
  path /var/log/myapp/*.log
  pos_file /var/log/fluentd/myapp.pos
  tag myapp
  <parse>
    @type none
  </parse>
</source>

# Forward to Elasticsearch / OpenSearch
<match myapp.**>
  @type elasticsearch
  host my-search-host
  port 9200
</match>

# Forward to Loki, Kafka, S3, Azure, HTTP, and many more

After editing, apply your changes:

sudo systemctl restart fluentd

Fluentd ships a large plugin ecosystem (tail, syslog, http, forward, Elasticsearch, Loki, Kafka, S3, Azure, and more). See the Fluentd documentation for the full plugin catalogue and install extra plugins with sudo /opt/fluent/bin/fluent-gem install <plugin>.

Maintenance

• Service: managed by systemd - systemctl status fluentd, journalctl -u fluentd.
• Configuration: /etc/fluent/fluentd.conf; restart the service after edits.
• Monitoring: the monitor_agent HTTP endpoint on 127.0.0.1:24220 exposes /api/plugins.json and /api/config.json for health and per-plugin metrics.
• Plugins: install Ruby gem plugins with sudo /opt/fluent/bin/fluent-gem install <plugin>.
• Security patches: unattended-upgrades remains enabled so the OS continues to receive security updates automatically; the fluent-package package stays updatable via apt.

Support

cloudimg provides 24/7 expert support for this image. Contact support@cloudimg.co.uk.

Fluentd is a trademark of the Fluent project, hosted by the Cloud Native Computing Foundation. cloudimg is not affiliated with or endorsed by the Fluent project or the CNCF.