OpenSearch 3.9.0 + Fluent Bit 5.1.4 on AWS User Guide

Overview

This guide covers launching and using the OpenSearch 3.9.0 + Fluent Bit 5.1.4 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg log management listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

OpenSearch is an open source search and analytics engine, OpenSearch Dashboards is its web interface, and Fluent Bit is a lightweight log collector. In this image the three arrive wired together as one log pipeline: Fluent Bit collects this instance's systemd journal, any .log file you drop into /var/log/cloudimg-logs/ and records forwarded by your other machines, and ships them over TLS into OpenSearch as daily cloudimg-logs-* indices. When you first sign in to Dashboards, Discover already lists your instance's logs.

journald (host logs) ---------------------\
/var/log/cloudimg-logs/*.log --------------+--> Fluent Bit --(TLS, per-instance account)--> OpenSearch 127.0.0.1:9200
your other machines: forward 24224 --------/                                                   |
   (TLS + per-instance shared key)                       nginx 443 (per-instance certificate) --> OpenSearch Dashboards 127.0.0.1:5601

The cloudimg image adds:

• No demo security configuration, ever. OpenSearch normally bootstraps with the widely published admin/admin login and demo certificates whose private keys are public. The demo installer never runs in this image, the package's template account database (with its published password hashes) is removed, and a start-up guard refuses to start OpenSearch if any published password or certificate is in effect.
• Credentials and certificates unique to your instance. First boot creates a private certificate authority, the node, admin, web and forward certificates, the admin password, two service account passwords, the forward shared key, the Dashboards session-cookie key and the OpenSearch keystore. The image ships an empty data directory, so the security index is initialised on your instance from those values. No two instances share a secret.
• HTTPS from the first boot. nginx is the only web entry point and serves Dashboards on port 443 with a certificate issued by your instance's own authority. Port 80 only redirects to HTTPS.
• Everything internal on loopback. The OpenSearch REST API (127.0.0.1:9200), its transport port (127.0.0.1:9300), Dashboards (127.0.0.1:5601) and the Fluent Bit monitor (127.0.0.1:2020) cannot be reached from the network.
• Authenticated remote ingest. The Fluent Bit forward input on TCP 24224 accepts only TLS connections that present your instance's shared key. Keep it closed in the security group until you open it to your own networks.
• Fail closed. OpenSearch, Dashboards, Fluent Bit and nginx refuse to start until first boot has created this instance's secrets, and first boot fails loudly rather than finishing with a credential that does not work.
• Hardened web front end. Requests for a host name the instance does not serve are dropped, cross site requests and WebSocket upgrades are refused, session cookies are marked Secure, and nginx logs the real client address rather than any X-Forwarded-For header a client sends.
• No instance metadata for the web services. OpenSearch, Dashboards and nginx are blocked from the instance metadata service, so a notification channel or webhook cannot be turned against the instance's credentials.
• Key only SSH. Password logins, root logins and X11 forwarding are switched off.
• Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on and the signed OpenSearch and Fluent Bit package repositories kept configured.

What is on the image: OpenSearch 3.9.0 and OpenSearch Dashboards 3.9.0 from the official OpenSearch Project packages, Fluent Bit 5.1.4 from the official Fluent Bit package (all Apache License 2.0, unmodified, signature and checksum verified at build time); Ubuntu's security maintained OpenJDK 21 in place of the Java runtime bundled in the OpenSearch package; nginx; Ubuntu 24.04 LTS; cloud-init and the AWS Systems Manager agent. The Security Analytics (SIEM rules) plugin is not included.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH) and TCP 443 (Dashboards) from your own networks only, and TCP 80 if you want plain HTTP requests redirected. Open TCP 24224 only if other machines will forward logs, and only from their private addresses. Never open 9200, 9300, 5601 or 2020; those ports listen on loopback.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick OpenSearch 3.9.0 + Fluent Bit 5.1.4 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large or larger (OpenSearch is a JVM service and Dashboards a Node service; the image sizes the OpenSearch heap from the memory it finds at boot), your key pair and a security group as above. Increase the root volume on the Configure storage step if you expect to keep a lot of log history. Under Advanced details, keep Metadata version at V2 only.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=opensearch-logs}]'

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

The login message shows the Dashboards address and where to find the credentials.

Step 3: Check the services

First boot creates the certificates, passwords and keystore, starts OpenSearch on an empty data directory, applies this instance's security configuration, installs the single node index template, starts Dashboards, Fluent Bit and nginx, and creates the Dashboards index pattern. It takes a few minutes on a 2 vCPU instance. When it has finished:

systemctl is-active opensearch opensearch-dashboards fluent-bit nginx
sudo test -f /var/lib/cloudimg/opensearch-fluent-bit-firstboot.done && echo FIRSTBOOT_DONE
cat /etc/cloudimg/opensearch-fluent-bit-versions.txt

active
active
active
active
FIRSTBOOT_DONE
opensearch=3.9.0
opensearch-dashboards=3.9.0
fluent-bit=5.1.4

If FIRSTBOOT_DONE is not printed yet, follow progress with sudo journalctl -u opensearch-fluent-bit-firstboot -f.

Only SSH, nginx and the authenticated forward input listen on public addresses; OpenSearch, Dashboards and the Fluent Bit monitor listen on loopback:

sudo ss -Hlntp | awk '{print $4, $6}' | grep -v '127.0.0.5' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort

0.0.0.0:22 users:(("sshd"),("systemd"))
0.0.0.0:24224 users:(("fluent-bit"))
0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
127.0.0.1:2020 users:(("fluent-bit"))
127.0.0.1:5601 users:(("node"))
[::]:22 users:(("sshd"),("systemd"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::]:80 users:(("nginx"),("nginx"),("nginx"))
[::ffff:127.0.0.1]:9200 users:(("java"))
[::ffff:127.0.0.1]:9300 users:(("java"))

SSH accepts keys only: passwords, root logins and X11 forwarding are switched off in /etc/ssh/sshd_config.d/00-cloudimg.conf, which sorts before every other SSH setting. Check the effective values:

sudo sshd -T | grep -E '^(passwordauthentication|kbdinteractiveauthentication|permitrootlogin|x11forwarding|pubkeyauthentication) '

permitrootlogin no
pubkeyauthentication yes
passwordauthentication no
kbdinteractiveauthentication no
x11forwarding no

Step 4: Get your address and credentials

Everything first boot generated is in the root only file /root/opensearch-fluent-bit-credentials.txt (mode 0600). List the entries it holds without printing any secret:

sudo grep -oE '^[a-z_]+=' /root/opensearch-fluent-bit-credentials.txt
sudo stat -c '%a %U:%G' /root/opensearch-fluent-bit-credentials.txt
sudo awk -F= '/^dashboards_url=/{print $2}' /root/opensearch-fluent-bit-credentials.txt

dashboards_url=
admin_username=
admin_password=
opensearch_url=
opensearch_ca=
forward_host=
forward_port=
forward_shared_key=
forward_ca=
dashboards_server_password=
fluentbit_writer_password=
600 root:root
https://<your-public-dns-name>/

Show the admin password when you are ready to sign in:

sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt

The certificate authority is unique to this instance, and the published demo login is refused:

openssl x509 -in /etc/cloudimg/opensearch-root-ca.pem -noout -subject
curl -s -o /dev/null -w 'HTTP %{http_code}\n' --cacert /etc/cloudimg/opensearch-root-ca.pem -u 'admin:admin' https://127.0.0.1:9200/

subject=CN = cloudimg-opensearch-ca-05b72e4e3ee00e5c, OU = cloudimg-appliance, O = cloudimg
HTTP 401

The 16 hex characters after cloudimg-opensearch-ca- are random and different on every instance.

Step 5: Sign in to OpenSearch Dashboards

Browse to the dashboards_url from the credentials file (https://<your-public-dns-name>/). The certificate is issued by your instance's own authority, so your browser warns that it is not signed by a public authority; Step 10 shows how to trust it or use your own name. Sign in as admin with the admin_password.

Open Discover. The cloudimg-logs-* index pattern is already the default, so the histogram and the record list show what Fluent Bit has collected from this instance since first boot.

Every record collected on the instance carries an appliance field and the instance's hostname, so you can narrow a search with Dashboards Query Language, for example:

appliance:"cloudimg-opensearch-log-stack" and log:"payment-service"

Under Dashboards Management, Index patterns, the cloudimg-logs-* pattern lists every field OpenSearch has mapped, with @timestamp as the time field:

Step 6: Ship your application logs

Anything written to a .log file directly under /var/log/cloudimg-logs/ is collected; Fluent Bit rescans the directory every five seconds, so new files are picked up without a restart:

echo "$(date -Is) payment-service order failed id=42 reason=card_declined" | sudo tee -a /var/log/cloudimg-logs/app.log

Search for it in Discover, or from the command line as in Step 8.

Step 7: Prove the pipeline end to end

The image ships a probe that writes a uniquely tagged line into the watched directory and then retrieves that exact line back out of OpenSearch:

sudo /usr/local/sbin/opensearch-fluent-bit-e2e.sh

E2E: emitted probe token CLOUDIMGE2E32a40df594d8237e665a8386 into /var/log/cloudimg-logs/cloudimg-e2e-probe.log
E2E PASS: token CLOUDIMGE2E32a40df594d8237e665a8386 emitted by Fluent Bit and retrieved from OpenSearch after 2 poll(s)
{"log":"2026-10-21T06:49:00+00:00 cloudimg end-to-end pipeline probe token=CLOUDIMGE2E32a40df594d8237e665a8386","hostname":"ip-172-31-87-177","appliance":"cloudimg-opensearch-log-stack","@timestamp":"2026-10-21T06:49:00.501Z"}

The token differs on every run. If the probe cannot read its own line back it exits non zero and says where delivery stopped.

Step 8: Use the OpenSearch REST API

The REST API listens on loopback only, so run these on the instance, or forward the port first with ssh -L 9200:127.0.0.1:9200 ubuntu@<public-ip>. The examples read the password into a shell variable and hand it to curl on standard input, so it never appears on a command line:

PW=$(sudo awk -F= '/^admin_password=/{print $2}' /root/opensearch-fluent-bit-credentials.txt)
printf 'user = "admin:%s"\n' "$PW" | curl -s -K - --cacert /etc/cloudimg/opensearch-root-ca.pem \
  'https://127.0.0.1:9200/_cluster/health' | jq -r '"status=\(.status) nodes=\(.number_of_nodes)"'
printf 'user = "admin:%s"\n' "$PW" | curl -s -K - --cacert /etc/cloudimg/opensearch-root-ca.pem \
  'https://127.0.0.1:9200/_cat/indices/cloudimg-logs-*?h=health,index,pri,rep'
printf 'user = "admin:%s"\n' "$PW" | curl -s -K - --cacert /etc/cloudimg/opensearch-root-ca.pem \
  -H 'Content-Type: application/json' 'https://127.0.0.1:9200/cloudimg-logs-*/_count' \
  -d '{"query":{"match_phrase":{"log":"payment-service"}}}' | jq '.count'

status=green nodes=1
green cloudimg-logs-2026.10.21 1 0
5

The cluster is green because first boot installs an index template that gives the daily cloudimg-logs-* indices one shard and no replica, which is right for a single node.

Step 9: Forward logs from your other machines

Remote Fluent Bit (or Fluentd) agents send to TCP 24224 with the forward protocol over TLS and must present this instance's shared key. A sender without TLS or without the key is refused.

1. In the instance's security group, allow TCP 24224 from the private addresses of the sending machines only.
2. Copy the certificate authority /etc/cloudimg/opensearch-root-ca.pem to each sender (it is public material) and read forward_shared_key from the credentials file.
3. Add an output like this to each sender's Fluent Bit configuration, using the instance's private address or DNS name:

[OUTPUT]
    Name          forward
    Match         *
    Host          <this-instance-private-dns-name>
    Port          24224
    Shared_Key    <forward_shared_key>
    Self_Hostname <sender-name>
    tls           On
    tls.verify    On
    tls.ca_file   /etc/fluent-bit/cloudimg-opensearch-ca.pem

The forward certificate is issued for the names in Step 10, so connect with one of those names (or set tls.vhost). Forwarded records are indexed into the same cloudimg-logs-* indices and keep the sender's own fields.

Step 10: Serve Dashboards under your own name

First boot serves the instance's public DNS name, public IP and private IP. To use your own DNS name, point it at the instance and run:

sudo opensearch-set-hostname logs.example.com

This issues new web and forward certificates for that name from the instance's own authority, updates nginx and the credentials file, and reloads the services. To stop browser warnings, import /etc/cloudimg/opensearch-root-ca.pem into your browser or operating system trust store, or replace /etc/ssl/opensearch-web/web.crt and web.key with a certificate from your own authority and run sudo systemctl reload nginx.

Server components

Files and services

Restart the stack in dependency order with sudo systemctl restart opensearch opensearch-dashboards fluent-bit. Dashboards answers HTTP 503 "server is not ready yet" for a short while after OpenSearch restarts.

Troubleshooting

Dashboards does not load. Check that the security group allows TCP 443 from your address, that first boot has finished (Step 3) and that nginx is running with systemctl status nginx --no-pager.

No records in Discover. Run the probe in Step 7; it reports where delivery stops. Then check Fluent Bit with sudo systemctl status fluent-bit --no-pager and curl -s http://127.0.0.1:2020/api/v1/health.

A remote agent cannot connect. Confirm TCP 24224 is open from the sender's private address, that the sender uses TLS with tls.ca_file pointing at this instance's CA, and that its Shared_Key matches forward_shared_key.

Disk usage grows. Log indices accumulate daily. Check with df -h / and delete old indices you no longer need through Dashboards (Index Management) or the REST API.

Security recommendations

1. Keep 9200, 9300, 5601 and 2020 closed; they are bound to loopback by design.
2. Restrict TCP 443 and 22 to your own networks, and open 24224 only to the machines that send logs.
3. Create named users with the roles they need in Security rather than sharing admin.
4. Rotate the admin password if it is ever shared: change it under Security, Internal users, then update /root/opensearch-fluent-bit-credentials.txt.
5. Take snapshots with the OpenSearch snapshot API if the log history matters to you.
6. Keep the authentication configuration as shipped unless you need more. OpenSearch 3.9.0 and Dashboards 3.9.0 bundle a few third party libraries with published advisories; cloudimg replaced every one that has a fixed release in the same line (Jackson, jsoup, lz4-java, handlebars, axios, follow-redirects, brace-expansion, source-map-js). The few that remain are only reached by features this image leaves switched off: the Jackson copy built into the SAML authenticator, and the node-forge PKCS#12 keystore loader in Dashboards. If you enable SAML or a PKCS#12 keystore, update to the next OpenSearch release first.

Support

cloudimg provides 24/7 technical support for this image by email and live chat: support@cloudimg.co.uk. For questions about the software itself, see the upstream OpenSearch and Fluent Bit projects.