Apache Hop on AWS User Guide

Overview

This image runs Apache Hop, the Hop Orchestration Platform, an open source data engineering and data orchestration platform for visually designing and running data pipelines and workflows. Hop is the successor to Pentaho Data Integration (Kettle) and provides a browser based, drag and drop editor for building, running and monitoring data integration, transformation and orchestration logic.

The image ships two Hop runtimes, both supervised by systemd:

• Hop Web is the browser based visual pipeline and workflow editor. It runs as the official apache/hop-web container on the loopback interface on port 8080. An nginx reverse proxy on port 80 fronts it with HTTP Basic authentication, so the editor is only reachable after signing in.
• Hop Server is the headless pipeline and workflow execution server. It runs as a dedicated unprivileged hop service user on the loopback interface on port 8081 and has its own HTTP Basic authentication. It is used to run pipelines and workflows on a schedule or remotely.

Apache Hop 2.18.1 runs on OpenJDK 21, installed from the Ubuntu 24.04 archive. The Hop client distribution (which provides the execution server and the hop-run command line runner) is installed under /opt/hop. The Hop projects, pipelines, workflows and the execution server configuration live on a dedicated EBS data volume mounted at /var/lib/hop, separate from the operating system disk, so the data tier can be resized independently.

Both Hop runtimes bind the loopback interface only. Nothing is exposed to the network except SSH on port 22 and the authenticated Hop Web editor on port 80. Because the Hop Server execution server can run pipelines remotely, replacing its default credentials on every instance is essential; this image does that automatically on first boot.

Per instance credentials

The password is generated fresh on the first boot of every deployed instance and applied to both the Hop Web editor (user admin) and the Hop Server execution server (user cluster). Two instances launched from the same Amazon Machine Image never share a password, and the well known Hop stock defaults (cluster / cluster) are never present on a deployed instance. The generated password is written to /root/hop-credentials.txt with mode 0600 so that only the root user can read it.

Prerequisites

Before you deploy this image you need:

• An Amazon Web Services account where you can launch EC2 instances
• IAM permissions to launch instances, create security groups, and subscribe to AWS Marketplace products
• An EC2 key pair in the target Region for SSH access to the instance
• A VPC and subnet in the target Region, with a security group allowing inbound port 22 from your management network and inbound port 80 from the trusted networks that will reach the Hop Web editor
• The AWS CLI (version 2) installed locally if you plan to deploy from the command line

Step 1: Launch the Instance from the AWS Marketplace

1. Open the product page in the AWS Marketplace and choose Continue to Subscribe, then Continue to Configuration.
2. Select the region and the delivery option (the Ubuntu 24.04 AMI), then Continue to Launch.
3. Choose an instance type. The recommended type is m5.large; Hop runs a JVM and a container, so give it at least 8 GiB of memory for production workloads.
4. Select your key pair, VPC, subnet, and a security group that allows inbound port 22 (SSH) and port 80 (the Hop Web editor) from your trusted networks, then launch.

Step 2: Launch the Instance from the AWS CLI

Alternatively, launch from the command line. Replace the AMI id, key pair name, security group and subnet with your own values:

aws ec2 run-instances \
  --image-id <ami-id> \
  --instance-type m5.large \
  --key-name <your-key-pair> \
  --security-group-ids <your-security-group> \
  --subnet-id <your-subnet> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-hop}]'

Step 3: Connect to Your Instance

SSH to the instance as the default login user for your operating system variant. This listing currently ships one variant:

ssh <login-user>@<public-ip>

Step 4: Retrieve Your Password

The per instance password is written to a root only file. Read it with sudo:

sudo cat /root/hop-credentials.txt

The file records the Hop Web URL, the admin user for the editor, the generated password, and the Hop Server URL with the cluster user. The same password is used for both the editor and the execution server.

Step 5: Verify the Server is Healthy

Confirm all four product services are active:

sudo systemctl is-active docker hop-web hop-server nginx

Confirm the unauthenticated health endpoint returns ok:

curl -fsS http://127.0.0.1/health

Confirm Hop Web (8080) and Hop Server (8081) are listening on the loopback interface:

LISTEN="$(sudo ss -tln)"
printf '%s\n' "$LISTEN" | grep -E '127.0.0.1:8080|127.0.0.1:8081'

Step 6: Open the Hop Web Editor

Browse to the instance over HTTP on port 80:

http://<public-ip>/

When prompted, sign in as the user admin with the password from /root/hop-credentials.txt. The Hop Web visual editor opens with the bundled samples project, which contains ready to run example pipelines and workflows, plus the cloudimg-hello pipeline this image ships. Drag transforms onto the canvas, connect them, and build your first pipeline.

You can confirm the reverse proxy enforces authentication directly on the instance. An unauthenticated request is refused with 401, and a request with the correct credentials returns 200:

curl -s -o /dev/null -w 'unauthenticated: HTTP %{http_code}\n' http://127.0.0.1/
curl -s -o /dev/null -w 'authenticated:   HTTP %{http_code}\n' -u "admin:<HOP_ADMIN_PASSWORD>" http://127.0.0.1/

Step 7: Run a Pipeline

The image ships a small deterministic pipeline, cloudimg-hello, that generates exactly ten rows and passes them to an output transform. You can run it with the hop-run command line runner to confirm the execution engine works end to end. The output transform reads and writes exactly ten rows:

OUT="$(sudo -u hop env JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 HOP_CONFIG_FOLDER=/var/lib/hop/server HOME=/var/lib/hop/server /bin/bash /opt/hop/hop-run.sh --project=samples --file='${PROJECT_HOME}/cloudimg-hello.hpl' --runconfig=local 2>&1)"
printf '%s\n' "$OUT" | grep -E 'generate 10 rows\.0 - Finished processing|output\.0 - Finished processing'
printf '%s\n' "$OUT" | grep -q 'output.0 - Finished processing (I=0, O=0, R=10, W=10, U=0, E=0)' \
  && echo 'PIPELINE ROUND-TRIP OK: output transform read and wrote exactly 10 rows' \
  || { echo 'PIPELINE ROUND-TRIP FAILED'; exit 1; }

You can also run the same pipeline from the Hop Web editor: open cloudimg-hello.hpl from the file browser, click the run button, choose the local run configuration, and launch. The execution log panel shows each transform finishing with ten rows.

Step 8: Use the Hop Server Execution Server

The Hop Server execution server runs on the loopback interface on port 8081 with HTTP Basic authentication as the cluster user (the same per instance password). It is not exposed to the network. To reach its status page from your workstation, forward the port over SSH:

ssh -L 8081:127.0.0.1:8081 <login-user>@<public-ip>

Then browse to http://127.0.0.1:8081/hop/status/ and sign in as cluster. From here you can submit pipelines and workflows for remote execution and monitor their status. You can confirm the server enforces authentication directly on the instance:

curl -s -o /dev/null -w 'unauthenticated: HTTP %{http_code}\n' http://127.0.0.1:8081/hop/status/
curl -s -o /dev/null -w 'authenticated:   HTTP %{http_code}\n' -u "cluster:<HOP_ADMIN_PASSWORD>" http://127.0.0.1:8081/hop/status/

Step 9: Filesystem and Service Layout

The /var/lib/hop tree is a dedicated EBS data volume, mounted by filesystem UUID, so it is independently resizable and separate from the operating system disk.

Product services:

Step 10: Start, Stop, and Check Status

sudo systemctl status hop-web hop-server nginx

To restart the editor or the execution server:

sudo systemctl restart hop-web hop-server

Step 11: Rotate the Password

To set a new password after first boot, update the nginx Basic auth file and the Hop Server configuration, then reload. Replace the value with your own strong secret:

sudo htpasswd -b /etc/nginx/.hop_htpasswd admin '<new-password>'
sudo systemctl reload nginx

Update the <password> element in /var/lib/hop/server/hop-server-config.xml to the same value and restart the execution server with sudo systemctl restart hop-server so the cluster credential matches.

Step 12: Security Recommendations

• The Hop Web editor is served over plain HTTP with Basic authentication. Restrict inbound port 80 in your security group to trusted networks, and for production place your own TLS termination (for example an Application Load Balancer or an nginx TLS server block with a CA signed certificate) in front of the editor so credentials are not sent in clear text.
• The Hop Server execution server can run pipelines remotely, which is powerful. It binds the loopback interface only and is never exposed to the network by this image; keep it that way and reach it over an SSH tunnel, or restrict it tightly if you choose to expose it.
• Keep the per instance password secret and rotate it as described above. Never restore the Hop stock cluster / cluster default.
• Apply operating system security updates regularly.

Step 13: Backups and Maintenance

Your pipelines, workflows and project metadata live under /var/lib/hop/project. Back up that directory (or snapshot the EBS data volume) to preserve your work. Because the data tier is a separate volume, you can snapshot it independently of the operating system disk.

Screenshots

The Hop Web visual editor, showing the welcome screen and the bundled samples project:

The cloudimg-hello pipeline open on the visual canvas, with the generate rows transform feeding an output transform:

The run options dialog, where you choose the run configuration and launch a pipeline:

A successful pipeline run: both transforms complete (green ticks) and the execution log confirms the rows processed:

Support

cloudimg provides 24/7 technical support for this product by email at support@cloudimg.co.uk and via live chat. Our engineers assist with Hop deployment, upgrades, pipeline and workflow design, execution server configuration and performance tuning. Critical issues receive a one hour average response time.

Apache Hop, Hop and Apache are trademarks of the Apache Software Foundation. All other product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.