ELK Stack on AWS User Guide

Overview

This image runs the full ELK Stack as a single node deployment. Elasticsearch is the distributed search and analytics engine, Logstash is the ingestion pipeline, and Kibana is the visual analytics interface. All three are installed from the official Elastic 8.x APT repository and run on the same instance with conservative JVM heaps so the stack fits in a four gigabyte instance out of the box.

Elasticsearch listens on the loopback interface on port 9200 with HTTP authentication enabled, so the cluster is only reachable from processes running on the instance. Logstash runs an ingestion pipeline with a Beats input on port 5044 ready for Filebeat and Metricbeat agents, plus a syslog input on UDP 5140 on the loopback. Kibana serves the web user interface on port 5601 on the loopback only, and an nginx reverse proxy on port 80 publishes Kibana behind HTTP basic authentication so the web interface is never exposed without a password.

The Elasticsearch elastic superuser password and the Kibana web basic-auth password are generated on the first boot of every deployed instance. Two instances launched from the same Amazon Machine Image never share passwords. Both values are written to /root/elk-credentials.txt with mode 0600 so that only the root user can read them.

Five systemd units make up the stack: elasticsearch.service, logstash.service, kibana.service, nginx.service, and elk-firstboot.service. The firstboot unit is a oneshot that rotates the passwords and writes the credentials file, then disables itself; the other four run continuously and start at boot.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network, inbound port 80 from the network you will reach Kibana from, and inbound port 5044 from the network where your Beats agents run
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for ELK Stack. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger. ELK runs three JVM workloads plus an nginx proxy on the same host, so it benefits from at least two vCPUs and 8 GiB of memory. For production workloads with sustained ingest, m5.xlarge or m5.2xlarge gives the JVMs room to raise their heaps. Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that allows inbound port 22 from your management network, inbound port 80 from the network you will reach Kibana from, and inbound port 5044 from the network where your Beats agents run. Leave the root volume at the default size or larger.

Select Launch instance. First boot initialisation takes approximately two to three minutes after the instance state becomes Running and the status checks pass, because the firstboot service waits for Elasticsearch to be ready before rotating the credentials.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg ELK Stack Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22, 80 and 5044 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":30,"VolumeType":"gp3"}}]' \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=elk-stack-01}]'

The command prints a JSON document on success. Note the instance ID, then retrieve its public address once it is running with aws ec2 describe-instances --instance-ids <instance-id> --query "Reservations[].Instances[].PublicIpAddress" --output text.

Step 3: Connect and Retrieve Initial Credentials

Connect over SSH with the key pair you selected and the public IP address from step 2. The SSH login user depends on the operating system of the AMI variant you launched:

AMI variant	SSH login user
ELK Stack 8 on Ubuntu 24.04	ubuntu

The first boot service runs before Kibana is ready, so the credentials file is always in place by the time the web interface accepts a login.

sudo cat /root/elk-credentials.txt

You will see a plain text file containing the Kibana URL, the Kibana web user (cloudimg), the Kibana web password, the Elasticsearch URL, the Elasticsearch superuser (elastic), the Elasticsearch password, and the Logstash Beats endpoint. Copy these values somewhere secure (a password manager or encrypted vault). Do not commit them to source control.

From the same SSH session you can confirm Elasticsearch is healthy. The cluster health endpoint needs the elastic credentials:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" http://127.0.0.1:9200/_cluster/health | jq

A JSON document with "status": "green" or "status": "yellow" confirms the cluster is up. A single node Elasticsearch cluster reports yellow when an index has unallocated replica shards, which is normal — the replicas are unallocated because there is no second node for them to live on.

Step 4: First Login to Kibana

Open a web browser and navigate to http://<public-ip>/. nginx challenges you for HTTP basic authentication.

Enter the basic-auth user cloudimg and the password from the ELK_WEB_PASSWORD line of /root/elk-credentials.txt. Your browser then loads Kibana, which presents its own sign-in form. Sign into Kibana as the elastic user with the password from the ELASTIC_PASSWORD line of the same file.

On the first successful sign in Kibana shows a welcome screen and the navigation hamburger in the top left of every page.

Step 5: Explore Data with Discover

The Discover application is the entry point for searching and exploring documents in Elasticsearch. Open the navigation hamburger and select Analytics then Discover.

On a freshly launched instance there are no indices yet, so Discover invites you to create a data view. After data starts flowing through Logstash and into Elasticsearch (see step 6) you will see your indices listed here, with a time histogram at the top, a row of fields down the left, and your documents in the centre. Discover supports the Kibana Query Language (KQL) and the older Lucene syntax for filtering documents.

Step 6: Send Logs with Logstash and Beats

The image runs Logstash with a pipeline pre-wired to accept Beats traffic on TCP 5044 and forward it to the local Elasticsearch. The simplest way to start sending data is to install Filebeat on a separate host and point it at the Logstash endpoint on your instance.

On a host where you want to collect logs from, install Filebeat and point its Logstash output at your instance:

sudo apt-get update && sudo apt-get install -y filebeat
sudo tee /etc/filebeat/filebeat.yml >/dev/null <<'YAML'
filebeat.inputs:
- type: filestream
  id: syslog
  paths:
    - /var/log/syslog
    - /var/log/auth.log
output.logstash:
  hosts: ["<your-public-ip>:5044"]
YAML
sudo systemctl enable --now filebeat

Replace <your-public-ip> with your ELK Stack instance's public address. Filebeat reads the log files, ships events to Logstash, Logstash forwards them to Elasticsearch as cloudimg-YYYY.MM.dd indices, and they appear in Kibana Discover within seconds.

You can also send simple test events directly from the ELK instance itself with curl:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" \
  -H 'Content-Type: application/json' \
  -X POST 'http://127.0.0.1:9200/cloudimg-test/_doc?refresh=true' \
  -d '{"message":"hello from cloudimg","ts":"2026-05-23T12:00:00Z"}'

This writes a single document to the cloudimg-test index, which then appears in Discover.

Step 7: Index Lifecycle Management

ELK indices accumulate over time, so a production deployment defines an index lifecycle management (ILM) policy to age and retire indices automatically. Open the navigation hamburger and select Management then Index Lifecycle Policies.

Elasticsearch ships several built-in ILM policies (logs, metrics, synthetics) which most Beats data streams use by default. Select Create policy to define your own. A typical policy keeps an index in the hot phase for 7 days, rolls over to warm storage for 30 days, then deletes it. Apply the policy to a data stream via its index template; the Elastic documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html covers the full workflow.

Step 8: Monitor the Cluster with Stack Monitoring

To see the health of Elasticsearch, Logstash and Kibana on one screen, open the navigation hamburger and select Management then Stack Monitoring. If monitoring is not yet enabled Kibana will offer to turn it on by enabling xpack.monitoring.collection.enabled on the cluster. Once enabled, the Cluster overview card shows the health of each component.

The cluster overview shows Elasticsearch (number of nodes, indices, documents, JVM heap and disk usage), Kibana (request rate, response time and memory usage), and Logstash once metrics are flowing. Click any card to drill into a per-component view; the Logstash detail page shows throughput in events per second, latency per filter stage, and the input and output plugins in the running configuration. The shipped configuration exposes one pipeline named main with a Beats input on port 5044, a UDP syslog input on 5140, a grok filter for the syslog branch, and an Elasticsearch output.

You can also query the Logstash node API directly from the instance:

curl -fsS http://127.0.0.1:9600/_node/stats | jq .pipelines.main.events

The response shows the number of events the pipeline has received, filtered and emitted since it started.

Step 9: The systemd Service Layout

Each component is a dedicated systemd unit. Use systemctl to inspect and control them:

sudo systemctl status elasticsearch.service
sudo systemctl status logstash.service
sudo systemctl status kibana.service
sudo systemctl status nginx.service

Logs land in journalctl:

sudo journalctl -u elasticsearch.service -n 100
sudo journalctl -u logstash.service -n 100
sudo journalctl -u kibana.service -n 100

Configuration files live under their package's directory:

• Elasticsearch: /etc/elasticsearch/elasticsearch.yml and /etc/elasticsearch/jvm.options.d/heap.options
• Logstash: /etc/logstash/conf.d/cloudimg.conf and /etc/logstash/jvm.options
• Kibana: /etc/kibana/kibana.yml
• nginx: /etc/nginx/sites-available/kibana and /etc/nginx/kibana.htpasswd

To raise the Elasticsearch heap, edit /etc/elasticsearch/jvm.options.d/heap.options and replace the -Xms and -Xmx lines, then restart elasticsearch.service. As a rule of thumb, set the heap to half the instance memory but never above 30 GiB. The same applies to Logstash via /etc/logstash/jvm.options.

Step 10: Add a Kibana User

Kibana defaults to signing you in as elastic, which is the cluster superuser. For day-to-day use create a less privileged user. From a shell on the instance:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" \
  -H 'Content-Type: application/json' \
  -X POST 'http://127.0.0.1:9200/_security/user/analyst' \
  -d '{"password":"<choose-a-strong-password>","roles":["kibana_admin","viewer"],"full_name":"Analyst"}'

You can then sign in to Kibana as analyst. To rotate the elastic superuser password later, run:

sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password --user elastic --interactive

and update any tools that authenticate as elastic.

Step 11: Enable HTTPS with a Reverse Proxy

nginx already serves Kibana on port 80 with HTTP basic authentication. For any deployment beyond a trusted management network, terminate TLS at nginx so the basic-auth credential and Kibana session cookies cannot be intercepted. Install certbot and request a Let's Encrypt certificate for your hostname:

sudo apt-get update && sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d kibana.your-domain.example \
  --non-interactive --agree-tos -m you@your-domain.example --redirect

certbot edits the nginx server block to listen on 443 with the certificate and to redirect plain HTTP to HTTPS. Reload nginx with sudo systemctl reload nginx. Then restrict the instance security group so port 80 is reachable only from the proxy network.

Step 12: Backups and Maintenance

Elasticsearch data lives on its own EBS volume at /var/lib/elasticsearch, so the simplest backup strategy is an EBS snapshot of that volume. Stop the elasticsearch service first to flush in-memory state, or use the official snapshot and restore API for live snapshots:

ELASTIC_PASSWORD=$(sudo grep '^ELASTIC_PASSWORD=' /root/elk-credentials.txt | cut -d= -f2-)
curl -fsS -u "elastic:${ELASTIC_PASSWORD}" \
  -H 'Content-Type: application/json' \
  -X PUT 'http://127.0.0.1:9200/_snapshot/cloudimg-s3' \
  -d '{"type":"s3","settings":{"bucket":"<your-snapshot-bucket>","region":"<your-region>"}}'

The cluster needs the repository-s3 plugin and an IAM role with s3:PutObject on the bucket. The Elastic documentation at https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html covers the full workflow.

For kernel and package updates, Ubuntu's unattended-upgrades is enabled by default, so security patches apply automatically. To update ELK itself, install the new version with apt:

sudo apt-get update
sudo apt-get install -y elasticsearch logstash kibana
sudo systemctl restart elasticsearch.service logstash.service kibana.service

The Elastic APT repository pins to the 8.x major; major version upgrades require switching the repository source and following the Elastic upgrade documentation.

Step 13: Scaling Beyond a Single Instance

For larger deployments decouple ELK from the single instance pattern:

• Move Elasticsearch to a multi-node cluster by launching additional EC2 instances, switching discovery.type from single-node to the seed-hosts model, and configuring node.roles so the cluster has dedicated master, data and ingest nodes
• Or use Amazon OpenSearch Service for a fully managed cluster behind the same Kibana
• Run Logstash on its own fleet of instances with an Application Load Balancer in front, so Beats agents send to a load-balanced endpoint and Logstash can scale horizontally
• Put Kibana behind an Application Load Balancer with sticky sessions for high availability of the web interface
• Use Amazon S3 as a snapshot repository for point-in-time recovery of the Elasticsearch indices

Each of these is documented in the official Elastic documentation at https://www.elastic.co/guide/en/elastic-stack/current/index.html.

Support

cloudimg provides 24/7/365 expert technical support for this image. Guaranteed response within 24 hours, one hour average for critical issues. Contact support@cloudimg.co.uk.

For general ELK Stack questions consult the community resources at https://discuss.elastic.co/ and the documentation at https://www.elastic.co/guide/index.html.