Apache Druid Real-Time Analytics Database on AWS User Guide

Overview

This image runs Apache Druid, a real-time analytics database designed for fast aggregations and interactive, low-latency queries over large event and time-series datasets. It is deployed as a single-server bundle: the coordinator, overlord, broker, historical, middleManager and router processes run together, alongside a bundled Apache ZooKeeper and an embedded metadata store, and are launched by a systemd service that starts them on boot and restarts on failure.

Druid is installed under /opt/druid and runs as a dedicated unprivileged druid system account. Its entire persistent working tree - the deep-storage segments, the metadata store, the ZooKeeper data and the indexing task logs - lives on a dedicated, independently resizable EBS data volume mounted at /var/lib/druid, so your ingested data survives instance replacement and is captured into the image layout.

The Druid unified web console is served on port 80 by an nginx reverse proxy. Every Druid process binds only to loopback, so the single public entry point is nginx, which enforces HTTP basic authentication in front of the console. The console password is generated on the first boot of every deployed instance, so two instances launched from the same AMI never share a password. It is written to /root/druid-credentials.txt with mode 0600 so that only the root user can read it.

JVM memory is tuned to the instance size on every boot: a boot-time wrapper reads the instance's total memory, reserves headroom for the operating system, the bundled ZooKeeper and the page cache, and hands the remainder to Druid's automatic memory sizer, so the same image fits the recommended m5.large and scales up on larger instances without reconfiguration.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and port 80 for the web console
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

Sign in to the AWS Management Console, open the EC2 service, and select Launch instance. Under Application and OS Images choose AWS Marketplace AMIs and search for Apache Druid. Select the cloudimg listing and choose Select, then Continue on the subscription summary.

Pick an instance type of m5.large or larger (Druid is a JVM analytics workload; size it for your data volume and query concurrency). Choose your EC2 key pair under Key pair (login). Under Network settings select your VPC and subnet, and either create or select a security group that opens port 22 from your management network and port 80 for the web console. Leave the root volume at the default size or larger, and keep the attached data volume that holds Druid's segments and metadata.

Select Launch instance. First-boot initialisation runs immediately, and Druid then takes one to three minutes to bring up every process and become healthy after the instance state becomes Running and the status checks pass.

Step 2: Launch the Instance from the AWS CLI

The following block launches an instance from the cloudimg Apache Druid Marketplace AMI into an existing subnet and security group. Replace <ami-id> with the AMI ID shown on the Marketplace listing, <key-name> with your EC2 key pair name, <subnet-id> with your subnet ID, and <security-group-id> with a security group that opens ports 22 and 80 as described above.

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <key-name> \
  --subnet-id <subnet-id> \
  --security-group-ids <security-group-id> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-druid}]'

When the instance reaches the Running state and its status checks pass, note its public IP address or DNS name from the EC2 console or with aws ec2 describe-instances.

Step 3: Connect to Your Instance

Connect over SSH using your key pair and the login user for your operating system variant.

ssh -i <key-name>.pem ubuntu@<public-ip>

Step 4: Retrieve the Console Password

The Druid console password is unique to your instance and was generated on first boot. Read it as root:

sudo cat /root/druid-credentials.txt

The file lists the console URL, the administrator user (admin) and the generated password. It is mode 0600 and owned by root, so only the root user can read it. Every command below that needs the password reads it straight from this file into a shell variable, so you never have to copy or paste it.

Step 5: Confirm the Services Are Running

Druid and nginx run as systemd services. Confirm both are active:

systemctl is-active druid.service nginx.service

nginx serves an unauthenticated health endpoint on port 80 that is ideal for load-balancer health checks. Confirm it answers:

curl -fsS http://127.0.0.1/health

Step 6: Verify the Authenticated Console

The console itself requires the per-instance password. First confirm that an unauthenticated request is rejected:

curl -s -o /dev/null -w 'unauthenticated console: HTTP %{http_code}\n' http://127.0.0.1/

You should see HTTP 401. Now read the password from the credentials file into a shell variable (it is never printed) and confirm the authenticated console returns HTTP 200:

PW=$(sudo awk -F= '/^DRUID_ADMIN_PASSWORD=/{print $2}' /root/druid-credentials.txt)
curl -sL -o /dev/null -w 'authenticated console: HTTP %{http_code}\n' -u "admin:$PW" http://127.0.0.1/

Step 7: Open the Web Console

Browse to http://<public-ip>/ and sign in as user admin with the password from Step 4. The Druid unified console opens on the Datasources view, which lists every datasource with its availability, segment count, total rows and size.

The Query tab gives you an interactive SQL editor. You can run an aggregation over your data and get ranked results in well under a second.

The Tasks view shows your ingestion history and the status of every batch and streaming task.

The Services view lists every Druid process - coordinator, overlord, broker, historical, middleManager and router - so you can confirm the whole single-server bundle is healthy.

Step 8: Run a SQL Query from the Command Line

Druid exposes a SQL-over-HTTP endpoint. Run a simple connectivity query (it needs no data), reading the password from the credentials file so it is never printed:

PW=$(sudo awk -F= '/^DRUID_ADMIN_PASSWORD=/{print $2}' /root/druid-credentials.txt)
curl -s -u "admin:$PW" -H 'Content-Type: application/json' -XPOST http://127.0.0.1/druid/v2/sql -d '{"query":"SELECT 1 + 1 AS two"}'

The response is [{"two":2}], confirming the broker and the SQL engine are answering through the authenticated proxy.

Step 9: Load the Wikipedia Sample Dataset

Druid ships a sample Wikipedia edit-stream dataset and a batch ingestion spec. Submitting the spec to the overlord creates a wikipedia datasource you can query. Run the following on the instance (replace ADMIN_PASSWORD with the password from Step 4):

D=/opt/druid/current
sed "s#\"baseDir\" : \"quickstart/tutorial/\"#\"baseDir\" : \"$D/quickstart/tutorial/\"#" \
  "$D/quickstart/tutorial/wikipedia-index.json" > /tmp/wikipedia-index.json
curl -s -u "admin:ADMIN_PASSWORD" -H 'Content-Type: application/json' \
  -XPOST http://<public-ip>/druid/indexer/v1/task -d @/tmp/wikipedia-index.json

Watch the task reach SUCCESS in the console Tasks view. Once its segment is loaded (a few seconds later) the wikipedia datasource appears on the Datasources view and you can query it, for example:

SELECT page, COUNT(*) AS edits, SUM(added) AS chars_added
FROM wikipedia GROUP BY page ORDER BY edits DESC LIMIT 12

Alternatively, use the console's Load data wizard to ingest your own batch or streaming data through a guided flow.

Data Persistence and the Dedicated Data Volume

Druid's entire persistent working tree lives on a dedicated EBS data volume mounted at /var/lib/druid, separate from the operating-system disk. Confirm the mount and its free space:

findmnt /var/lib/druid
df -h /var/lib/druid

Because the segments, the metadata store, the ZooKeeper data and the task logs all live on this volume, you can resize it independently for capacity and IOPS, and your ingested data is decoupled from the instance's root disk. Snapshot this volume with Amazon EBS snapshots for backups.

Security Model

• Per-instance credentials. The console password is generated on the first boot of every instance and stored in /root/druid-credentials.txt (mode 0600). No shared or default password ships in the image.
• Loopback isolation. Every Druid process and the bundled ZooKeeper bind only to 127.0.0.1. Nothing Druid exposes is reachable off the instance except through nginx.
• Authenticated front door. nginx on port 80 is the single public entry point; it enforces HTTP basic authentication in front of the console. Only the unauthenticated /health endpoint is open, for load-balancer checks.
• Network hardening. Restrict port 80 to trusted sources in your security group, and deploy in a private VPC subnet where possible. The dedicated data volume can use AWS EBS encryption at rest.

Maintenance

• Restart Druid: sudo systemctl restart druid.service (it can take one to three minutes to become healthy again).
• JVM memory: memory is recomputed from the instance's total RAM on every boot, so resizing the instance automatically re-tunes Druid on the next restart - no manual JVM configuration is required.
• Upgrades: keep the operating system current with sudo apt-get update && sudo apt-get upgrade.

Support

This image is maintained by cloudimg with 24/7 technical support via email and live chat. Contact support@cloudimg.co.uk for deployment and instance-sizing guidance, ingestion and data-source configuration, query tuning, JVM administration, and any issues including requesting refunds.