Apache Hadoop on Ubuntu 24.04 on Azure User Guide

Overview

Apache Hadoop is the open-source framework for distributed storage (HDFS) and distributed processing (YARN + MapReduce) of large data sets. This cloudimg image runs a ready-to-use single-node, pseudo-distributed Hadoop 3.4.1 cluster: the HDFS NameNode and DataNode plus the YARN ResourceManager and NodeManager, each as a hardened systemd service. All HDFS data lives on a dedicated Azure data disk, and the cluster comes up automatically on first boot. Backed by 24/7 cloudimg support.

What is included:

• Apache Hadoop 3.4.1 (HDFS + YARN + MapReduce) on OpenJDK 11, at /opt/hadoop
• Four systemd services: hadoop-namenode, hadoop-datanode, hadoop-resourcemanager, hadoop-nodemanager, running as a dedicated hadoop user
• The NameNode web UI on :9870 and the ResourceManager web UI on :8088
• A dedicated Azure data disk at /var/lib/hadoop holding the HDFS NameNode/DataNode directories — separate from the OS disk and re-provisioned with every VM
• A per-VM /root/hadoop-info.txt written at first boot with the UI URLs
• 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet + subnet in the target region. Standard_B2ms (2 vCPU / 8 GiB RAM) is a sensible minimum for Hadoop and YARN; scale up for larger workloads. NSG inbound: allow 22/tcp from your management network, and 9870/tcp + 8088/tcp from the networks that need the web dashboards.

Step 1 — Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Apache Hadoop by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size (Standard_B2ms or larger); under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22). Review the dedicated data disk on the Disks tab, then Review + create → Create. Add rules for 9870 and 8088 afterwards (Step 2 shows the CLI).

Step 2 — Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name hadoop \
  --image <marketplace-image-urn> \
  --size Standard_B2ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name hadoop --port 9870 --priority 1010
az vm open-port --resource-group <your-rg> --name hadoop --port 8088 --priority 1020

Step 3 — Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 — Confirm the cluster is running

systemctl is-active hadoop-namenode hadoop-datanode hadoop-resourcemanager hadoop-nodemanager
curl -s -o /dev/null -w 'NameNode UI: %{http_code}\n' http://127.0.0.1:9870/
curl -s -o /dev/null -w 'ResourceManager UI: %{http_code}\n' http://127.0.0.1:8088/

All four services report active, and both web UIs return 200.

Step 5 — Retrieve your cluster details

The first boot writes the UI URLs for this VM:

sudo cat /root/hadoop-info.txt

Open the NameNode UI at http://<vm-public-ip>:9870/ and the ResourceManager UI at http://<vm-public-ip>:8088/ once the NSG ports are open.

Step 6 — Use HDFS

Run HDFS commands as the hadoop user. Create a directory, store a file and read it back:

sudo -u hadoop /opt/hadoop/bin/hdfs dfs -mkdir -p /demo
echo 'hello hadoop' | sudo -u hadoop /opt/hadoop/bin/hdfs dfs -put -f - /demo/hello.txt
sudo -u hadoop /opt/hadoop/bin/hdfs dfs -ls /demo
sudo -u hadoop /opt/hadoop/bin/hdfs dfs -cat /demo/hello.txt

Step 7 — Run a MapReduce job

Submit the bundled example MapReduce job (an estimate of Pi) to YARN to confirm the processing layer works:

sudo -u hadoop /opt/hadoop/bin/hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar pi 2 5

The job is accepted by the ResourceManager (visible at :8088) and prints an estimated value of Pi on completion.

Step 8 — Confirm the HDFS report

sudo -u hadoop /opt/hadoop/bin/hdfs dfsadmin -report | head -20

This shows the live DataNode, configured capacity and usage.

Backup and maintenance

All HDFS data lives on the dedicated data disk at /var/lib/hadoop. Snapshot that disk in Azure to back up your filesystem. Keep the OS patched with sudo apt update && sudo apt upgrade. The daemons restart cleanly with sudo systemctl restart hadoop-namenode hadoop-datanode hadoop-resourcemanager hadoop-nodemanager. Do not reformat the NameNode unless you intend to erase HDFS.

Support

This image is backed by 24/7 cloudimg support. Contact us by email and chat for help with HDFS, YARN, MapReduce, scaling out to multiple nodes, and tuning.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.