ELK Stack 9 on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers deploying and using ELK Stack 9 on Ubuntu 24.04 LTS from cloudimg on Microsoft Azure.

The ELK Stack is three products from Elastic that work together for log analytics and search: Elasticsearch stores and searches the data, Logstash receives and processes events (here from Beats agents such as Filebeat) and writes them to Elasticsearch, and Kibana is the web interface for searching, exploring and building dashboards. This image runs all three on one virtual machine as a single-node cluster, installed from Elastic's official package repository (artifacts.elastic.co/packages/9.x/apt). Image version 1.0.20260502 carries Elasticsearch, Logstash and Kibana 9.3.4.

Every command and every output in this guide was run on a virtual machine deployed from the public Marketplace image (image version 1.0.20260502), as the azureuser account a customer gets.

What cloudimg adds to the upstream packages:

• A password for each VM. Elasticsearch security is on. At first boot, elk-firstboot.service sets the elastic superuser and the internal kibana_system user to strong passwords generated on that VM, and writes the elastic password to a root-only file. The password used while the image was built is refused afterwards.
• Elasticsearch on localhost only. The REST API (9200) and the transport port (9300) listen on 127.0.0.1, and every request needs a login.
• Kibana asks for a login. Kibana listens on port 5601 and redirects every page to its sign-in screen. Keep 5601 closed in your network security group and reach Kibana through an SSH tunnel (Step 9).
• A ready Beats pipeline. Logstash listens for Beats agents on port 5044 and writes each agent's events to a daily index in the local Elasticsearch (filebeat-2026.11.02, for example).
• Memory sizing at first boot. The first-boot service sets the Elasticsearch and Logstash Java heap sizes from the memory of the VM you chose.
• Pinned to 9.x. The three packages are held with apt-mark hold, so routine upgrades never move them. You upgrade them within 9.x when you choose (Step 13).

What's included

• Elasticsearch, Logstash and Kibana 9.3.4 (image version 1.0.20260502), from Elastic's 9.x APT repository, which stays configured on the VM
• The default Beats input pipeline, /etc/logstash/conf.d/01-beats-input.conf
• The cloudimg first-boot service elk-firstboot.service (per-VM passwords and heap sizing)
• Ubuntu 24.04 LTS with unattended security upgrades turned on
• Azure Linux Agent (waagent) and cloud-init for Azure provisioning
• 24/7 cloudimg support

Connecting to your VM

Prerequisites

1. An Azure subscription and an SSH public key
2. For the command line route, the Azure CLI on your workstation, signed in with az login
3. A network security group that allows TCP 22 from your own address only

Choose a virtual machine size that suits your log volume and retention. Elasticsearch, Logstash and Kibana all run on the one VM, and the first-boot service sizes the Java heaps from the VM's memory (Step 12).

Step 1: Deploy from the Azure portal

On the listing, choose Get it now. The Azure portal then walks you through the basics: your subscription, resource group, region and size, your SSH key and inbound ports, then Review and create. For inbound ports, allow SSH (22) only.

Step 2: Deploy from the Azure CLI

The image carries a Marketplace purchase plan, so accept its terms once per subscription, then create the virtual machine. Replace <your-vm-size> with the size you chose:

az vm image terms accept --urn cloudimg1647283583153:elk-9-ubuntu-24-04:default:latest

az vm create \
  --resource-group myResourceGroup \
  --name my-elk-vm \
  --image cloudimg1647283583153:elk-9-ubuntu-24-04:default:latest \
  --size <your-vm-size> \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --nsg my-ssh-only-nsg \
  --public-ip-sku Standard

Step 3: Connect via SSH

Run these on your own workstation. Find the public IP, then connect:

az vm show --resource-group myResourceGroup --name my-elk-vm --show-details --query publicIps -o tsv
ssh azureuser@<vm-ip>

First boot takes a few minutes after the VM is running: the first-boot service sizes the heaps, restarts Elasticsearch, sets the per-VM passwords, then restarts Kibana and Logstash. If you connect straight away, wait for it:

sudo cloud-init status --wait

It prints status: done once first boot has finished.

Step 4: Check the services

systemctl is-active elasticsearch.service kibana.service logstash.service elk-firstboot.service
sudo ss -Htln '( sport = :9200 or sport = :5601 or sport = :5044 )' | awk '{print $4}'

All four report active. Elasticsearch listens on 127.0.0.1:9200 only (Java prints it as [::ffff:127.0.0.1]:9200), Kibana on 0.0.0.0:5601 and the Logstash Beats input on *:5044. Your network security group decides who can reach 5601 and 5044: with the SSH-only group from the prerequisites, nobody outside the VM can.

Step 5: Read the per-VM credentials

The elastic password is unique to this VM and is written at first boot to a root-only file (mode 0600):

sudo stat -c '%a %U:%G %n' /stage/scripts/elk-credentials.log
sudo grep -E '^(ELASTIC_USER|ELASTICSEARCH_URL|LOGSTASH_BEATS_PORT)=' /stage/scripts/elk-credentials.log

It prints 600 root:root /stage/scripts/elk-credentials.log, then ELASTIC_USER=elastic, ELASTICSEARCH_URL=http://127.0.0.1:9200 and LOGSTASH_BEATS_PORT=5044. The file's KIBANA_URL line holds the VM's private address; from your workstation, use the SSH tunnel in Step 9 instead.

When you need the password, read it into a variable rather than onto the screen. The commands in this guide use $ES_PW, so set it in each new shell:

export ES_PW="$(sudo sed -n 's/^ELASTIC_PASSWORD=//p' /stage/scripts/elk-credentials.log)"

Keep it safe: elastic is the superuser.

Step 6: The authentication gate

Elasticsearch refuses a request with no login and one with a wrong login, and accepts the VM's own:

curl -s -o /dev/null -w 'no login    -> HTTP %{http_code}\n' http://127.0.0.1:9200/
curl -s -o /dev/null -w 'wrong login -> HTTP %{http_code}\n' -u 'elastic:not-the-real-one' http://127.0.0.1:9200/
curl -s -o /dev/null -w 'VM login    -> HTTP %{http_code}\n' -u "elastic:$ES_PW" http://127.0.0.1:9200/

The first two return HTTP 401 and the last HTTP 200. Kibana does the same for its pages: curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' http://127.0.0.1:5601/ prints 302 http://127.0.0.1:5601/login?next=%2F.

Step 7: Cluster health and version

curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/_cluster/health?filter_path=cluster_name,status,number_of_nodes&pretty'
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/?filter_path=name,cluster_name,version.number&pretty'

The cluster cloudimg-elk reports "status" : "green" with one node, elk-node-1, and the version (9.3.4 on image version 1.0.20260502).

Step 8: Index and search a document

A single-node cluster cannot place replica shards, so create indices with no replicas and they stay green:

curl -s -u "elastic:$ES_PW" -X PUT http://127.0.0.1:9200/app-logs \
  -H 'Content-Type: application/json' -d '{"settings":{"number_of_replicas":0}}'
curl -s -u "elastic:$ES_PW" -X POST 'http://127.0.0.1:9200/app-logs/_doc?refresh=true' \
  -H 'Content-Type: application/json' \
  -d '{"@timestamp":"2026-11-02T09:00:00Z","service":"checkout","level":"error","message":"payment gateway timeout after 30s"}'
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/app-logs/_search?q=message:timeout&filter_path=hits.total.value,hits.hits._source.message&pretty'
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/_cat/indices/app-logs?v&h=health,index,docs.count'

The search finds the document ("value" : 1, payment gateway timeout after 30s) and the index is green with one document. An index created without that setting, such as the daily filebeat-* indices Logstash creates, shows yellow on one node: its data is complete, and only the replica has nowhere to go.

Step 9: Open Kibana through an SSH tunnel

Keep port 5601 closed in the network security group and carry Kibana over SSH. On your workstation:

ssh -N -L 5601:127.0.0.1:5601 azureuser@<vm-ip>

Leave it running and open http://localhost:5601 in your browser. The traffic travels inside the encrypted SSH connection. Sign in with the user elastic and the password from Step 5. Kibana opens, with Discover, Dashboards and its other applications in the main menu.

Step 10: Ship logs with Filebeat through Logstash

Logstash's default pipeline takes events from Beats agents on port 5044 and writes them to Elasticsearch as <beat>-<date> indices. Here is the whole path on the VM itself, with Filebeat reading one log file. Elastic's 9.x repository is already configured, so install Filebeat and give it a minimal configuration:

sudo apt-get -q update
sudo apt-get install -y filebeat
sudo tee /etc/filebeat/filebeat.yml >/dev/null <<'EOF'
filebeat.inputs:
  - type: filestream
    id: demo-app
    paths:
      - /var/log/demo-app.log
output.logstash:
  hosts: ["127.0.0.1:5044"]
EOF
sudo filebeat test config -c /etc/filebeat/filebeat.yml
sudo filebeat test output -c /etc/filebeat/filebeat.yml
sudo systemctl enable --now filebeat

filebeat test config prints Config OK, and filebeat test output ends with talk to server... OK. Write a few lines to the log and count what arrives:

for i in 1 2 3; do echo "$(date -Is) checkout ERROR payment gateway timeout order=$((1040+i))" | sudo tee -a /var/log/demo-app.log >/dev/null; done
sleep 15
curl -s -u "elastic:$ES_PW" 'http://127.0.0.1:9200/_cat/indices/filebeat-*?v&h=health,index,docs.count'

The three lines arrive in today's filebeat-* index (for example filebeat-2026.11.02 with docs.count 3).

To ship logs from your other servers, install Filebeat on each one from the same Elastic 9.x repository, set output.logstash.hosts to this VM's private address and port 5044, and add a network security group rule that allows TCP 5044 from those servers' private addresses only. The Beats input on this image accepts plain TCP with no client authentication, so never open 5044 to the internet (see Security notes).

Step 11: Explore the logs in Kibana

Create a Kibana data view for the Filebeat indices, either in Kibana (Stack Management, Data views) or with Kibana's API:

curl -s -u "elastic:$ES_PW" -X POST http://127.0.0.1:5601/api/data_views/data_view \
  -H 'kbn-xsrf: true' -H 'Content-Type: application/json' \
  -d '{"data_view":{"title":"filebeat-*","name":"Filebeat logs","timeFieldName":"@timestamp"}}'

Then open Discover in Kibana, choose the Filebeat logs data view, and search with KQL, for example message : timeout.

Step 12: Logstash pipelines and memory

The default pipeline is /etc/logstash/conf.d/01-beats-input.conf. Logstash loads every .conf file in /etc/logstash/conf.d/, so add your own filters and outputs there, and restart it after a change:

sudo systemctl restart logstash.service

Logstash takes a minute or two after a restart before it listens on 5044 again; sudo journalctl -u logstash -f shows Pipelines running when it is ready. Its Elasticsearch output signs in as elastic, with the password taken from the root-only environment file /etc/default/logstash (mode 0600), so a pipeline file never holds the password itself.

The first-boot service sets the Java heap sizes from the VM's memory and records what it chose:

sudo grep 'Adjusting heaps' /var/log/elk-firstboot.log

Elasticsearch takes about 40 per cent of memory, and Logstash about 15 per cent, up to 1 GB. To change them later, edit /etc/elasticsearch/jvm.options.d/cloudimg-heap.options (keep -Xms and -Xmx equal) or the -Xms/-Xmx lines in /etc/logstash/jvm.options, then restart that service. If you resize the VM, adjust them yourself: first boot runs once.

Step 13: Keep it updated

The image was patched when it was built. Ubuntu security updates install automatically (unattended-upgrades is active), and other updates do not, so update a new VM before you put it into service, and regularly afterwards:

sudo apt-get -q update
sudo apt-get -y upgrade

Elasticsearch, Logstash and Kibana are held, so this leaves them alone (apt-mark showhold lists all three). Elastic's 9.x repository carries newer 9.x releases than the image. To upgrade the stack within 9.x, release the hold, upgrade all three together while keeping cloudimg's configuration files, hold them again and restart:

sudo apt-mark unhold elasticsearch logstash kibana
sudo apt-get install -y -o Dpkg::Options::=--force-confold elasticsearch logstash kibana
sudo apt-mark hold elasticsearch logstash kibana
sudo systemctl daemon-reload
sudo systemctl restart elasticsearch.service kibana.service logstash.service

We ran exactly this from 9.3.4 to 9.5.5: the elastic password, the indices and the Beats pipeline carried over, and events flowed again once Logstash had restarted. --force-confold keeps the existing kibana.yml and logstash.yml (the package's new defaults are saved beside them as .dpkg-dist files). Read Elastic's release notes before you upgrade, take a snapshot of the OS disk first, and reboot if a new kernel was installed.

Security notes

• Network security group. SSH (22) is the only port you need to open. Reach Kibana through the SSH tunnel (Step 9), and allow 5044 only from your log shippers' private addresses.
• No TLS on this image. Elasticsearch and Kibana speak plain HTTP, which is safe while Elasticsearch stays on localhost and Kibana is reached through the SSH tunnel. Before you expose Kibana or Elasticsearch to a network, or accept Beats traffic from outside a private network, configure TLS by following Elastic's guide to securing the Elastic Stack.
• Users. Create users with only the roles they need instead of sharing elastic (Kibana, Stack Management, Users and Roles). If you change the elastic password, also update ELASTIC_PASSWORD in /etc/default/logstash and restart Logstash, because its output signs in as elastic. The credentials file is not rewritten, so update or remove it yourself.
• Credentials file. Once you have stored the password somewhere safe, such as Azure Key Vault, you can remove the file: sudo shred -u /stage/scripts/elk-credentials.log.

When you have finished with the examples, remove them:

curl -s -u "elastic:$ES_PW" -X DELETE http://127.0.0.1:9200/app-logs
sudo systemctl disable --now filebeat

Server components

Support and licensing

The Elasticsearch, Logstash and Kibana packages on this image are Elastic's default distributions, which Elastic licenses under the Elastic License 2.0. This image uses their free features only, and no Elastic subscription is included. cloudimg provides round-the-clock technical support for this image by email, with a guaranteed 24-hour response time: support@cloudimg.co.uk.

Elasticsearch, Logstash and Kibana are trademarks of Elasticsearch B.V. Ubuntu is a registered trademark of Canonical Ltd. cloudimg is not affiliated with or endorsed by either of them.