Apache Hop on Ubuntu 24.04 on Azure User Guide

Overview

Apache Hop (the Hop Orchestration Platform) is a visual data orchestration and integration platform, and the modern successor to Pentaho Data Integration / Kettle. You design pipelines (data transformations) and workflows (orchestration) in a browser, then run them locally or on a dedicated execution server. The cloudimg image installs Apache Hop 2.18.1 with two runtimes ready to use: Hop Web, the browser-based visual editor, served by the official Apache Hop Web container; and Hop Server, the headless pipeline and workflow execution server. Hop Web is published behind an nginx reverse proxy on port 80 with HTTP Basic authentication, and Hop Server listens on port 8081 with its own Basic authentication. A unique password is generated on the first boot of every VM. All projects, pipelines, workflows and Hop configuration live on a dedicated Azure data disk mounted at /var/lib/hop, and the image ships the official Apache Hop samples project so the editor has real content to explore. Backed by 24/7 cloudimg support.

What is included:

• Apache Hop 2.18.1 (Hop Orchestration Platform) on OpenJDK 21
• Hop Web, the browser-based visual pipeline and workflow editor, run as the official apache/hop-web container on 127.0.0.1:8080
• Hop Server, the headless execution server, on 127.0.0.1:8081 with Basic authentication (user cluster)
• nginx on :80 as a reverse proxy to Hop Web with HTTP Basic authentication (user admin)
• A per-VM password generated on first boot and recorded in a root-only file
• A dedicated Azure data disk at /var/lib/hop for projects, pipelines, workflows and Hop configuration
• The official Apache Hop samples project, preloaded so the editor opens with example pipelines and workflows
• docker.service, hop-web.service, hop-server.service and nginx.service as systemd units, enabled and active
• An unauthenticated /health endpoint for Azure Load Balancer health probes
• 24/7 cloudimg support

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region. Standard_B2ms (2 vCPU / 8 GiB RAM) is a reasonable starting point; size up for heavier pipelines. NSG inbound: allow 22/tcp from your management network and 80/tcp for Hop Web. Hop Web serves plain HTTP on port 80; for production, terminate TLS in front of it with your own domain.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Apache Hop by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Review the dedicated data disk on the Disks tab, then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name apache-hop \
  --image <marketplace-image-urn> \
  --size Standard_B2ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name apache-hop --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active docker.service hop-web.service hop-server.service nginx.service

All four report active. Hop Web runs as the official apache/hop-web container on the loopback address 127.0.0.1:8080; nginx fronts it on port 80. Hop Server runs as the dedicated hop user on 127.0.0.1:8081. All projects, pipelines, workflows and Hop configuration live on the dedicated Azure data disk mounted at /var/lib/hop.

Step 5 - Retrieve your password

A single password is generated uniquely on the first boot of your VM and used for both Hop Web (admin) and Hop Server (cluster). It is written to a root-only file:

sudo cat /root/hop-credentials.txt

This file contains HOP_ADMIN_USER (admin), HOP_CLUSTER_USER (cluster) and HOP_ADMIN_PASSWORD, along with the Hop Web and Hop Server URLs. Store the password somewhere safe.

Step 6 - Confirm the health endpoint

nginx serves an unauthenticated health endpoint for load balancers and probes:

curl -s http://localhost/health

It returns ok. This endpoint never requires authentication, so it is safe for an Azure Load Balancer health probe.

Step 7 - Confirm the Hop Web container

Hop Web runs as a container managed by hop-web.service. Confirm it is up:

sudo docker ps --filter name=hop-web --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

You will see the hop-web container Up and publishing 127.0.0.1:8080->8080/tcp.

Step 8 - Sign in to Hop Web

Browse to http://<vm-public-ip>/ and sign in when your browser prompts for HTTP Basic credentials: username admin and the password from Step 5. Hop Web opens on the preloaded samples project.

The file explorer on the left lists the project's pipelines, workflows, transforms and datasets. Double-click any pipeline (.hpl) to open it on the canvas.

Use the project and file explorer to browse the full set of example pipelines and workflows shipped with the image.

Step 9 - Confirm authentication from the command line

nginx protects Hop Web with HTTP Basic authentication: a wrong password returns 401, and the per-VM password returns 200. Because the command embeds your unique password, run it interactively rather than from a script - substitute the value of HOP_ADMIN_PASSWORD from Step 5 for <HOP_ADMIN_PASSWORD>:

curl -s -o /dev/null -w '%{http_code}\n' -u admin:<HOP_ADMIN_PASSWORD> http://127.0.0.1/

The response is 200. Repeating the command with a wrong password returns 401.

Step 10 - Use the Hop Server execution engine

Hop Server runs your pipelines and workflows headlessly and exposes a status page on port 8081, protected by Basic authentication for the cluster user. To view the status page from the command line, substitute your password for <HOP_ADMIN_PASSWORD>:

curl -s -u cluster:<HOP_ADMIN_PASSWORD> http://127.0.0.1:8081/hop/status/

The status page lists running pipelines and workflows and the server configuration. Point the Hop Server metadata element in Hop Web at 127.0.0.1:8081 with the cluster credentials to run your pipelines and workflows on the server.

Step 11 - Confirm project data lives on the dedicated disk

Your Hop projects, pipelines, workflows and Hop configuration are stored on the dedicated Azure data disk so they survive OS changes and can be resized independently:

findmnt /var/lib/hop

The mount is backed by a separate Azure data disk captured into the image and re-provisioned on every VM.

Maintenance

• Projects and pipelines: create and edit projects, pipelines and workflows in Hop Web; everything is stored under /var/lib/hop/project on the data disk.
• Hop Server: the execution server configuration (including the cluster credential) is at /var/lib/hop/server/hop-server-config.xml; restart with sudo systemctl restart hop-server after edits.
• Hop Web: the editor runs as the hop-web container; restart with sudo systemctl restart hop-web.
• Storage: projects, pipelines, workflows and configuration live under /var/lib/hop on the data disk; back up that volume to protect your work.
• TLS: Hop Web serves plain HTTP on port 80; front it with TLS (e.g. certbot) and your own domain before production use.
• Security patches: unattended-upgrades remains enabled so the OS continues to receive security updates automatically.

Support

cloudimg provides 24/7 expert support for this image. Contact support@cloudimg.co.uk.