Observability Azure

OpenSearch Log Stack with Fluent Bit on Ubuntu 24.04 on Azure User Guide

| Product: OpenSearch Log Stack with Fluent Bit on Ubuntu 24.04 LTS on Azure

Overview

This image is a working log pipeline on one virtual machine. Fluent Bit collects records from the systemd journal and from any log file you drop into a watched directory, tags each one with the host it came from, and ships it over an encrypted local connection into OpenSearch, which stores them as daily indices. OpenSearch Dashboards is the web interface where you search, filter and chart everything that has been collected.

The three components are already wired to each other and proven to deliver end to end, so there is no pipeline to design before your first query returns. All three are Apache 2.0 licensed. Backed by 24/7 cloudimg support.

What is included:

  • OpenSearch 3.8.0 with the Security plugin, listening on loopback only
  • OpenSearch Dashboards 3.8.0, the single externally reachable service, served over this machine's own TLS certificate
  • Fluent Bit 5.1.0, collecting the systemd journal plus /var/log/cloudimg-logs/*.log
  • A cloudimg-logs-* index pattern created on first boot, so Discover is populated the moment you log in
  • A per VM administrator password, three separate service accounts and a private certificate authority, all generated on the first boot of every machine
  • An end to end pipeline probe you can run at any time
  • Ubuntu's security maintained OpenJDK 21 rather than a vendored Java runtime
  • 24/7 cloudimg support

Key facts:

Item Value
Platform Microsoft Azure
Default SSH user azureuser
Dashboards URL https://<vm-ip>:5601/
OpenSearch REST API https://127.0.0.1:9200 (loopback only)
Credentials file /root/opensearch-fluent-bit-credentials.txt
Log drop in directory /var/log/cloudimg-logs/
Recommended VM size Standard_B2ms (2 vCPU, 8 GB)

No published credential ships in this image. OpenSearch normally bootstraps with the widely documented admin/admin login and a set of demo TLS certificates whose private keys are published upstream. That bootstrap is disabled during the build, so none of it is ever written to disk. Your machine generates its own credentials and its own certificate authority the first time it boots.

Prerequisites

  1. An Azure subscription with permission to create virtual machines
  2. A resource group and a virtual network in the region you are deploying into
  3. A network security group allowing inbound TCP 22 from your management address, and inbound TCP 5601 from the networks that should reach the dashboard
  4. An SSH key pair
  5. The Azure CLI version 2.50 or later if you use the command line path in Step 2

Recommended VM size: Standard_B2ms (2 vCPU, 8 GB). OpenSearch is a JVM service, Dashboards is a Node service and Fluent Bit runs alongside both, so 8 GB is the honest minimum for comfortable operation. The image sizes the OpenSearch heap from the memory it actually finds at boot, so a larger machine is used sensibly without any configuration change.

Step 1: Deploy the Virtual Machine from the Azure Portal

  1. In the Azure Portal choose Create a resource and search the Marketplace for OpenSearch Log Stack with Fluent Bit on Ubuntu 24.04 LTS.
  2. On the Basics tab pick your subscription, resource group and region. Set the virtual machine name and choose size Standard_B2ms or larger.
  3. Set the authentication type to SSH public key, set the username to azureuser and paste your public key.
  4. On the Networking tab select your virtual network and subnet, and attach a network security group that allows inbound TCP 22 from your management address and inbound TCP 5601 from the networks that need the dashboard. Leave every other port closed.
  5. Review and create.

Only port 5601 needs to be reachable. The OpenSearch REST API on 9200, its transport port 9300 and the Fluent Bit monitoring port 2020 are all bound to the loopback interface inside the machine and cannot be reached from the network at all.

Step 2: Deploy the Virtual Machine from the Azure CLI

RG="logging-prod"
LOCATION="eastus"
VM_NAME="cloudimg-logstack"

az group create --name "$RG" --location "$LOCATION"

az network vnet create --resource-group "$RG" --name logstack-vnet \
  --address-prefix 10.10.0.0/16 --subnet-name logstack-subnet \
  --subnet-prefix 10.10.1.0/24 --location "$LOCATION"

az network nsg create --resource-group "$RG" --name logstack-nsg --location "$LOCATION"

az network nsg rule create --resource-group "$RG" --nsg-name logstack-nsg \
  --name allow-ssh --priority 1000 --destination-port-ranges 22 \
  --access Allow --protocol Tcp --source-address-prefixes "<your-management-ip>/32"

az network nsg rule create --resource-group "$RG" --nsg-name logstack-nsg \
  --name allow-dashboards --priority 1010 --destination-port-ranges 5601 \
  --access Allow --protocol Tcp --source-address-prefixes "<your-management-ip>/32"

az vm create --resource-group "$RG" --name "$VM_NAME" \
  --image "cloudimg:opensearch-fluent-bit:default:latest" \
  --size Standard_B2ms --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values ~/.ssh/id_rsa.pub \
  --vnet-name logstack-vnet --subnet logstack-subnet --nsg logstack-nsg \
  --public-ip-sku Standard --location "$LOCATION"

Step 3: Connect via SSH

ssh azureuser@<vm-ip>

Find the address with:

az vm show --resource-group "$RG" --name "$VM_NAME" --show-details --query publicIps -o tsv

Step 4: Wait for First Boot to Finish

The first boot generates this machine's certificate authority, its passwords and its Dashboards index pattern, then starts the three services. OpenSearch loads roughly twenty plugins on a cold start, so allow several minutes on a 2 vCPU machine. The sentinel file appears when first boot is complete:

if [ -f /var/lib/cloudimg/opensearch-fluent-bit-firstboot.done ]; then
  echo "first boot complete"
else
  echo "first boot still running"
fi

Check the three services:

systemctl is-active opensearch opensearch-dashboards fluent-bit

Expected output:

active
active
active

The three services running with their pinned versions, showing that only port 5601 is exposed while 9200, 9300 and 2020 stay on loopback

Step 5: Retrieve Your Credentials

Every password on this machine was generated on its first boot and exists nowhere else. Read them with sudo:

sudo cat /root/opensearch-fluent-bit-credentials.txt

The file is mode 0600 and owned by root. It contains the dashboard URL, the admin password you log in with, and the two service account passwords used internally by Dashboards and Fluent Bit.

You can confirm for yourself that the published demo login does not work:

sudo curl -s -o /dev/null -w 'HTTP %{http_code}\n' \
  --cacert /etc/cloudimg/opensearch-root-ca.pem \
  -u 'admin:admin' https://127.0.0.1:9200/

Expected output:

HTTP 401

The credentials file is root only, the certificate authority is unique to this machine, and the published admin password is rejected with HTTP 401

Step 6: Open OpenSearch Dashboards

Browse to https://<vm-ip>:5601/. The certificate was generated on this machine, so your browser will warn that it is not signed by a public authority. Section "Security Recommendations" explains how to replace it with your own certificate.

Log in with the username admin and the admin_password value from the credentials file.

The OpenSearch Dashboards login page served over this machine's own TLS certificate

Dashboards opens on Discover with the cloudimg-logs-* index pattern already selected, because first boot created it for you. The histogram shows record volume over time and the table below lists the individual records already collected from this machine.

Discover showing collected log records with the volume histogram drawn and the parsed fields listed down the left

Type a query into the search bar to filter. Every record collected by this appliance carries an appliance field, so this query matches everything the stack has ingested:

appliance: "cloudimg-opensearch-log-stack"

A DQL query running against the collected logs and returning matching records

The index pattern lists every field OpenSearch has mapped, including the journald metadata and the fields Fluent Bit adds:

The cloudimg-logs index pattern with its mapped fields and the @timestamp time field

Step 7: Ship Your Own Logs

Anything written to a .log file under /var/log/cloudimg-logs/ is picked up automatically. Fluent Bit rescans that directory every five seconds, so a file created after boot is collected without restarting anything.

echo "$(date -Is) payment-service order failed id=42" | sudo tee -a /var/log/cloudimg-logs/app.log

Wait a few seconds, then search for it in Discover, or from the command line in Step 8.

To ship logs from other machines, point any Fluent Bit, Fluentd or syslog forwarder at this host and write into the same index prefix, or mount or sync the remote log files into /var/log/cloudimg-logs/.

Step 8: Prove the Pipeline End to End

The image ships a probe that emits a uniquely tagged line and then retrieves that exact line back out of OpenSearch. It is the fastest way to confirm collection, indexing and search are all working:

sudo /usr/local/sbin/opensearch-fluent-bit-e2e.sh

Expected output:

E2E: emitted probe token CLOUDIMGE2E7fdf3040b36adaab3a985421 into /var/log/cloudimg-logs/cloudimg-e2e-probe.log
E2E PASS: token CLOUDIMGE2E7fdf3040b36adaab3a985421 emitted by Fluent Bit and retrieved from OpenSearch after 1 poll(s)
{"log":"2026-08-07T12:58:10+00:00 cloudimg end-to-end pipeline probe token=CLOUDIMGE2E7fdf3040b36adaab3a985421","hostname":"ubuntu-opensearch-fluent-bit-build-08071020-6c95e9","appliance":"cloudimg-opensearch-log-stack","@timestamp":"2026-08-07T12:58:10.003Z"}

The token is different on every run. If the probe cannot retrieve its own line it exits non zero and says so, rather than reporting success.

The end to end probe emitting a tagged line and retrieving that exact record back from OpenSearch

Step 9: Search from the REST API

The OpenSearch API is on loopback, so run these on the machine itself, or forward the port over SSH with ssh -L 9200:127.0.0.1:9200 azureuser@<vm-ip>.

List the daily indices:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/cloudimg/opensearch-root-ca.pem -u "admin:$PW" \
  'https://127.0.0.1:9200/_cat/indices/cloudimg-logs-*?v&h=health,index,docs.count,store.size'

Search for a term:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/cloudimg/opensearch-root-ca.pem -u "admin:$PW" \
  -H 'Content-Type: application/json' \
  'https://127.0.0.1:9200/cloudimg-logs-*/_search?size=1' \
  -d '{"query":{"match":{"log":"payment-service"}}}'

Listing the daily log indices and searching them through the OpenSearch REST API

Check cluster health:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/cloudimg/opensearch-root-ca.pem -u "admin:$PW" \
  'https://127.0.0.1:9200/_cluster/health?pretty'

A single node cluster reports green because the shipped index template creates one shard with no replica.

Server Components

Component Version Purpose
OpenSearch 3.8.0 Search and analytics engine storing the log records
OpenSearch Dashboards 3.8.0 Web interface for search, filtering and visualisation
OpenSearch Security plugin bundled TLS and authentication on the REST and transport layers
Fluent Bit 5.1.0 Log collector and forwarder
OpenJDK 21 Java runtime, from Ubuntu, security maintained
Ubuntu 24.04 LTS Base operating system

Filesystem Layout

Path Purpose
/usr/share/opensearch OpenSearch installation
/etc/opensearch/opensearch.yml Cluster and security configuration
/etc/opensearch/certs/ This machine's certificate authority, node and admin certificates
/etc/opensearch/opensearch-security/ Account database, roles and role mappings
/var/lib/opensearch Index data
/var/log/opensearch/ OpenSearch logs
/etc/opensearch-dashboards/opensearch_dashboards.yml Dashboards configuration
/etc/fluent-bit/fluent-bit.conf The collection pipeline
/etc/fluent-bit/opensearch.env Fluent Bit's OpenSearch credential, root only
/var/log/cloudimg-logs/ Drop .log files here to have them collected
/etc/cloudimg/opensearch-root-ca.pem Certificate authority, readable for verifying connections
/root/opensearch-fluent-bit-credentials.txt Per machine credentials, mode 0600

Managing the Services

sudo systemctl status opensearch
sudo systemctl status opensearch-dashboards
sudo systemctl status fluent-bit

Restart the stack in dependency order:

sudo systemctl restart opensearch
sudo systemctl restart opensearch-dashboards
sudo systemctl restart fluent-bit

OpenSearch takes several minutes to become available on a 2 vCPU machine because of the number of plugins it loads. Dashboards reports HTTP 503 with "server is not ready yet" until OpenSearch is answering, which is expected during that window.

Scripts and Log Files

Path Purpose
/usr/local/sbin/opensearch-fluent-bit-e2e.sh End to end pipeline probe
/usr/local/sbin/opensearch-fluent-bit-bootstrap.sh Generates this machine's certificates and passwords
/usr/local/sbin/opensearch-fluent-bit-credguard.sh Refuses to start OpenSearch if a published credential is in effect
/usr/local/sbin/opensearch-fluent-bit-seed-dashboards.sh Creates the cloudimg-logs-* index pattern
/var/log/cloudimg-firstboot.log First boot log
/var/log/opensearch/cloudimg-logs.log OpenSearch server log
/stage/scripts/initial_boot_update.sh Standard cloudimg first boot update hook

On Startup

opensearch-fluent-bit-firstboot.service runs once on the first boot of each machine. It generates a private certificate authority, a node certificate and an admin client certificate, mints three passwords, writes them into the account database, applies that configuration to the cluster, starts the three services and creates the Dashboards index pattern. It then writes /var/lib/cloudimg/opensearch-fluent-bit-firstboot.done and disables itself, so later reboots are ordinary starts.

The three product services will not start until first boot has published /var/lib/cloudimg/opensearch-fluent-bit-bootstrap.ready, which guarantees no service can ever run against credentials that have not been generated for this machine.

Troubleshooting

Dashboards shows "server is not ready yet". OpenSearch has not finished starting. Watch sudo journalctl -u opensearch -f and wait, then reload. A cold start takes several minutes on 2 vCPU.

The dashboard does not load at all. Confirm the network security group allows inbound TCP 5601 from your address and that the service is listening:

sudo ss -ltnp | grep 5601

No records in Discover. Confirm Fluent Bit is running and delivering:

sudo systemctl status fluent-bit --no-pager | head -5
sudo curl -s http://127.0.0.1:2020/api/v1/metrics | head -c 400

Then run the end to end probe from Step 8, which will say exactly where delivery stops.

Your own log file is not collected. It must end in .log and live directly in /var/log/cloudimg-logs/. Confirm Fluent Bit can read it:

ls -l /var/log/cloudimg-logs/

Forgotten admin password. It is only in /root/opensearch-fluent-bit-credentials.txt on the machine. Read it with sudo. It is not recoverable from the image.

Checking disk usage as logs accumulate. Log indices grow over time:

df -h /
sudo du -sh /var/lib/opensearch

Delete indices you no longer need, replacing the date to match:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
sudo curl -s --cacert /etc/cloudimg/opensearch-root-ca.pem -u "admin:$PW" \
  'https://127.0.0.1:9200/_cat/indices/cloudimg-logs-*?v&h=index,docs.count,store.size'

Security Recommendations

  1. Keep 9200 and 2020 on loopback. The image binds them there deliberately. Exposing the search API to a network removes the boundary that makes Dashboards the single, authenticated front door.
  2. Restrict inbound 5601 to the networks that genuinely need the dashboard, and inbound 22 to your management addresses.
  3. Replace the TLS certificate for production. The certificate authority generated on first boot is private to this machine, which is why browsers warn about it. To use a certificate from your own authority, replace /etc/opensearch-dashboards/certs/node.pem and node-key.pem, keeping the file ownership and mode intact, and restart Dashboards.
  4. Rotate the administrator password if it has been shared. Passwords live in the account database at /etc/opensearch/opensearch-security/internal_users.yml as bcrypt hashes and are applied with the security plugin's securityadmin.sh tool.
  5. Create per person accounts rather than sharing admin. OpenSearch Security supports roles and index level permissions, so read only users can be given access to the log indices alone.
  6. Take backups with the OpenSearch snapshot API if the log history matters to you. The image ships no snapshot repository, so you choose the destination.
  7. Keep the machine patched. Unattended security upgrades are enabled, and the OpenSearch and Fluent Bit package repositories remain configured so both keep receiving updates.

Support

cloudimg provides 24/7 support for this image. Contact support through www.cloudimg.co.uk with your Azure subscription id and the virtual machine name.

For questions about the software itself, the upstream projects are OpenSearch and Fluent Bit, both Apache 2.0 licensed.