Apache Hop 2.20.0 on AWS User Guide

Overview

This guide covers launching and using the Apache Hop 2.20.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg big data and data engineering listing on AWS Marketplace; you pick it as the software version when you launch.

Apache Hop (Hop Orchestration Platform) is an open source data orchestration tool. You design pipelines (row by row data transformations: read, join, clean, write) and workflows (the steps around them: run a pipeline, check a file, call a script, send a notification) in a visual editor, and run them locally, on a Hop Server, or on Apache Spark, Flink or Beam. Hop Web is that visual editor in a browser.

Upstream Hop Web ships as a container with sign in turned off by default, a demo user switch, plain HTTP and no limits on where a pipeline may connect. The cloudimg image makes it a server you can put in a VPC:

• Sign in, TLS and a password from the first boot. nginx serves Hop Web and the Hop Server API on port 443 (https://<host>/ui) with a certificate generated on your instance's first boot for its DNS name and IP addresses. First boot also generates the admin password, stored only as a PBKDF2 hash in Hop's user store, and an AES encryption key for the passwords you save in database connections. Nothing secret is baked into the AMI, and Hop Web refuses to start if its login has been turned off.
• An egress firewall for Hop. A pipeline can read from any URL, database or REST endpoint it is given, so an unprotected server can be pointed at addresses inside your network. On this image every connection Hop opens to link local addresses (including the instance metadata service), to 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 and their IPv6 equivalents is refused, by an iptables rule on the hop user and again by systemd inside the Hop service. Public endpoints work. You open specific private networks (your RDS subnet, say) with one command.
• Locked down. Hop Web runs on Apache Tomcat bound to loopback as an unprivileged hop user in a sandboxed service, with Hop, Tomcat and the web application read only. Cross site requests are refused, WebSocket upgrades are not forwarded, a request for a host name the instance does not serve is dropped, and the AMI requires IMDSv2.
• Ready to use. Hop's configuration, the default and samples projects and the audit log live on a separate 20 GiB gp3 data volume mounted at /var/lib/hop, which you can resize and snapshot on its own. sudo hop-admin runs Hop's command line tools (hop-run, hop-conf, hop-encrypt and others) against the same configuration.
• Built from the upstream release (pinned by SHA-256), with the bundled Guava and Proton-J libraries raised to versions that fix their published advisories and the Property Input transform removed (its ini4j library has an advisory with no fix), on a fully patched Ubuntu 24.04 LTS.

What is on the image: Apache Hop 2.20.0 (/opt/hop), Hop Web 2.20.0 on Apache Tomcat 10.1.61 (/opt/hop-web, /opt/tomcat), OpenJDK 21 from Ubuntu, nginx, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent. Licences and build provenance are in /usr/share/doc/apache-hop.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet.
• A security group that allows TCP 22 (SSH) from your own networks only, and TCP 443 from the people who use Hop Web and the systems that call the Hop Server API. Nothing else needs to be open.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Apache Hop 2.20.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (Hop Web keeps each signed in user's designer in server memory: use a larger type for several designers or large pipelines), your key pair, and a security group as above. The AMI attaches the 20 GiB data volume for /var/lib/hop automatically; increase its size in the launch wizard's storage section if your projects will hold large files.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=apache-hop}]'

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot generates the certificate, the admin password and the encryption key, and creates Hop's configuration and projects; the egress firewall, Hop Web and nginx then start. Hop Web takes about 40 seconds to start:

test -f /var/lib/cloudimg/hop-firstboot.done && echo "first boot complete"
systemctl is-active cloudimg-hop-egress hop-web nginx

first boot complete
active
active
active

Step 4: Get your URL and password

The credentials file is readable by root only. This prints it with the password hidden; sudo cat /root/hop-credentials.txt shows it:

sudo sed -E 's/^(HOP_[A-Z_]*PASSWORD=).*/\1<hidden>/' /root/hop-credentials.txt

# Apache Hop 2.20.0 (Hop Web) - generated on first boot by hop-firstboot.service
# Every value below is unique to this instance; none was baked into the image. Keep this file secret.

HOP_URL=https://ec2-54-84-58-220.compute-1.amazonaws.com/ui
HOP_HOST=ec2-54-84-58-220.compute-1.amazonaws.com
HOP_ADMIN_USER=admin
HOP_ADMIN_PASSWORD=<hidden>
HOP_CA_CERT=/etc/hop/tls/hop.crt
HOP_CERT_SHA256=7157bc4250dfef8b08a3c9f61febf8a41d6129aeedd282c55f7badb9e31093d5
#
# Hop Web: https://<HOP_HOST>/ui, sign in as HOP_ADMIN_USER / HOP_ADMIN_PASSWORD (self-signed certificate; see the guide).
# New admin password / locked out:  sudo hop-reset-admin
# Run a pipeline from the shell:    sudo hop-admin hop-run --project samples --file '<pipeline.hpl>'
# Encrypt a connection password:    sudo hop-admin hop-encrypt -hop '<password>'
# Change the name it serves as:     sudo hop-set-hostname <your-dns-name>
# Allow a private database host:    sudo hop-egress-allow add <cidr>
# User guide: https://www.cloudimg.co.uk/guides/apache-hop-2-20-0-on-ubuntu-24-04-aws/

Your instance shows its own DNS name and certificate fingerprint. HOP_CERT_SHA256 is the SHA-256 fingerprint of the certificate: compare it with the one your browser shows before you accept the certificate the first time.

Step 5: Open Hop Web

Browse to the HOP_URL from Step 4. The certificate is self signed, so the browser warns once; check that the fingerprint matches HOP_CERT_SHA256, then continue. Use the instance's DNS name as shown: a request for any other host name (such as the bare IP address) is dropped. To serve Hop under your own name, see Step 10.

Sign in as admin with HOP_ADMIN_PASSWORD:

Hop Web opens with its welcome dialog over the file explorer of the default project. The project selector in the toolbar switches between default and samples; the samples project holds Apache's example pipelines and workflows, a good place to see how transforms are put together. Give your own projects a home folder under /var/lib/hop (for example /var/lib/hop/projects/sales), so that they are on the data volume; Step 6 shows how to create one from the command line.

Step 6: Run a pipeline from the command line

sudo hop-admin runs Hop's command line tools as the hop service account, against the same configuration and projects as Hop Web and inside the same egress firewall. Run the generate-rows example from the samples project:

sudo hop-admin hop-run --project samples --file '${PROJECT_HOME}/transforms/generate-rows.hpl' --runconfig local --level Basic 2>&1 | grep -E 'Finished processing|Pipeline duration|Execution finished' | sed -E 's/^[0-9/: .]+- //'

generate 10 rows.0 - Finished processing (I=0, O=0, R=0, W=10, U=0, E=0)
generate-rows - Pipeline duration : 0.126 seconds [  0.126" ]
generate-rows - Execution finished on a local pipeline engine with run configuration 'local'

${PROJECT_HOME} is in single quotes so that Hop, not the shell, expands it.

Create a project of your own on the data volume (the folder must exist and belong to the hop user); it then appears in Hop Web's project selector. For a project called sales:

sudo install -d -o hop -g hop -m 0750 /var/lib/hop/projects /var/lib/hop/projects/sales
sudo hop-admin hop-conf --project-create --project=sales --project-home=/var/lib/hop/projects/sales

Schedule a pipeline or workflow with cron or a systemd timer that calls hop-admin hop-run (for workflows, --file points at a .hwf file). The other tools are hop-conf (projects and environments), hop-encrypt, hop-search and hop-import; sudo hop-admin hop-run --help lists the options.

Hop Web includes the Hop Server API under /hop/ (pipeline and workflow status, remote execution), behind the same sign in. The status page as XML, with the password passed to curl on its standard input rather than on the command line:

H=$(sudo sed -n 's/^HOP_HOST=//p' /root/hop-credentials.txt)
sudo sh -c 'printf "user = \"admin:%s\"\n" "$(sed -n "s/^HOP_ADMIN_PASSWORD=//p" /root/hop-credentials.txt)"' \
  | sudo curl -s --cacert /etc/hop/tls/hop.crt -K - "https://$H/hop/status/?xml=Y" | grep -oE '<(statusdesc|cpu_cores|os_name)>[^<]*'

<statusdesc>Online
<cpu_cores>2
<os_name>Linux

Step 7: Connect to databases

Hop ships JDBC drivers for PostgreSQL, Microsoft SQL Server, Amazon Redshift, IBM Db2 for i (jt400), MonetDB, CrateDB, H2, HSQLDB, Apache Derby, SQLite and Microsoft Access in /opt/hop/lib/jdbc. To add another driver (MySQL or Oracle, for example, which Hop does not bundle for licence reasons), copy its jar there as root (sudo install -m 0644 -o root -g root <driver>.jar /opt/hop/lib/jdbc/) and restart Hop Web with sudo systemctl restart hop-web.

Passwords in connections are encrypted. This image sets Hop's AES2 password encoder with a key generated on your instance's first boot (/etc/hop/encoder.key, readable by root and the hop service only). Passwords you type into a connection in Hop Web are stored encrypted with it. To put a password into a metadata file or an environment variable yourself, encrypt it first:

sudo hop-admin hop-encrypt -hop '<database-password>'

The output is an AES2 ... value to paste in place of the password. Keep a copy of /etc/hop/encoder.key with your backups: projects copied to another server can only decrypt their passwords with the same key.

Databases in your VPC are blocked until you allow them. The egress firewall refuses connections from Hop to private addresses, so a connection to an RDS instance or a database on 10.x, 172.16-31.x or 192.168.x fails with "connection refused" until you allow its network. Allow the database's address as a /32 (or its whole subnet, such as 10.0.5.0/24):

sudo hop-egress-allow add <private-ip>/32

This opens the range in both firewall layers, keeps it across reboots and restarts Hop Web. sudo hop-egress-allow list shows what is open and sudo hop-egress-allow remove <cidr> closes it again. Only private ranges (inside 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10 or fc00::/7) are accepted; the instance metadata service and link local addresses can never be allowed. Public endpoints (SaaS APIs, public S3 URLs, Maven Central) need no change.

Instance role credentials are not available to Hop. Because Hop cannot reach the instance metadata service, it cannot pick up the EC2 instance role. For Amazon S3, give Hop's S3 configuration an access key of an IAM user (or role) limited to the buckets it needs.

Step 8: Check the egress firewall

Run as the hop user, the metadata service and a private address are refused, and a public endpoint works:

for u in http://169.254.169.254/latest/meta-data/ http://10.0.0.1/ https://repo1.maven.org/maven2/; do
  sudo setpriv --reuid=hop --regid=hop --clear-groups -- curl -s -m 20 -o /dev/null -w "%{http_code}  $u\n" "$u" || true
done

000  http://169.254.169.254/latest/meta-data/
000  http://10.0.0.1/
200  https://repo1.maven.org/maven2/

000 means the connection was refused before any HTTP was exchanged. The rules themselves:

sudo iptables -w -S CLOUDIMG-HOP-EGRESS | grep -E -- '-d (169.254|10\.|172\.16|192\.168)' | grep tcp

-A CLOUDIMG-HOP-EGRESS -d 169.254.0.0/16 -p tcp -j REJECT --reject-with tcp-reset
-A CLOUDIMG-HOP-EGRESS -d 10.0.0.0/8 -p tcp -j REJECT --reject-with tcp-reset
-A CLOUDIMG-HOP-EGRESS -d 172.16.0.0/12 -p tcp -j REJECT --reject-with tcp-reset
-A CLOUDIMG-HOP-EGRESS -d 192.168.0.0/16 -p tcp -j REJECT --reject-with tcp-reset

Step 9: Check the security posture

Everything but the health check needs a password, a cross site request is refused, a request for a host name the instance does not serve is dropped without an answer, and nothing answers on port 80:

H=$(sudo sed -n 's/^HOP_HOST=//p' /root/hop-credentials.txt)
C=/etc/hop/tls/hop.crt
curl -s --cacert $C -o /dev/null -w 'Hop Web, no password        -> HTTP %{http_code}\n' -H 'Accept: application/json' https://$H/ui
curl -s --cacert $C -o /dev/null -w 'Hop Server API, no password -> HTTP %{http_code}\n' -H 'Accept: application/json' https://$H/hop/status/
printf 'user = "admin:not-the-password"\n' | curl -s --cacert $C -K - -o /dev/null -w 'wrong password              -> HTTP %{http_code}\n' -H 'Accept: application/json' https://$H/hop/status/
curl -s --cacert $C -o /dev/null -w 'cross-site request          -> HTTP %{http_code}\n' -H 'Origin: https://other.example' https://$H/ui
curl -s --cacert $C -o /dev/null -w 'health check                -> HTTP %{http_code}\n' https://$H/healthz
curl -sk -H 'Host: other.example' -o /dev/null -w 'unknown host                -> %{http_code}\n' https://127.0.0.1/healthz || true
curl -s -o /dev/null -w 'plain http :80              -> %{http_code}\n' --connect-timeout 3 http://127.0.0.1/ || true

Hop Web, no password        -> HTTP 401
Hop Server API, no password -> HTTP 401
wrong password              -> HTTP 401
cross-site request          -> HTTP 403
health check                -> HTTP 200
unknown host                -> 000
plain http :80              -> 000

A browser without a session is sent to the sign in page instead of the 401. The health check (/healthz) returns only ok and is meant for load balancers.

Hop Web listens on loopback only; nginx is the only service on 443:

sudo ss -Htlnp | awk '{print $4, $6}' | grep -E ':(443|8080) ' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g' | sort -u

0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
[::]:443 users:(("nginx"),("nginx"),("nginx"))
[::ffff:127.0.0.1]:8080 users:(("java"))

Step 10: Rotate the password, recover access, use your own DNS name

sudo hop-rotate-password gives admin a new random password, writes it to /root/hop-credentials.txt and restarts Hop Web; the old password stops working at once and open sessions end.

sudo hop-rotate-password

If you lose the password, or Hop's user store (/var/lib/hop/config/security/users.json) no longer has a working admin (disabled, deleted or without the admin role), sudo hop-reset-admin restores admin with the admin role and a new password, and forces Hop Web's sign in back on. Other users are kept as they are. Neither command prints the password; read it from the credentials file.

sudo hop-reset-admin

Point a DNS record at the instance, then set the name it is served as. This mints a new self signed certificate for the name (and the instance's IP addresses), updates nginx and the URL in the credentials file, and keeps the name across reboots. The fingerprint changes: check the new HOP_CERT_SHA256 in your browser.

sudo hop-set-hostname <your-domain>

To use a certificate from your own CA instead, replace /etc/hop/tls/hop.crt and /etc/hop/tls/hop.key (mode 0600, owner root) with yours for the same name and run sudo systemctl reload nginx. Or put an Application Load Balancer with an ACM certificate in front of port 443.

Step 11: Server components

sudo hop-admin hop-run --version 2>/dev/null | tail -1
java -cp /opt/tomcat/lib/catalina.jar org.apache.catalina.util.ServerInfo | head -1
grep -E '^(hop|tomcat|removed):' /usr/share/doc/apache-hop/BUILD-PROVENANCE.txt | cut -c1-100

2.20.0
Server version: Apache Tomcat/10.1.61
hop:      https://downloads.apache.org/hop/2.20.0/apache-hop-client-2.20.0.zip
removed:  plugins/transforms/propertyinput (Property Input transform; its ini4j 0.5.4 has CVE-2022-4
tomcat:   https://downloads.apache.org/tomcat/tomcat-10/v10.1.61/bin/apache-tomcat-10.1.61.tar.gz

Not included: the Property Input transform. This image does not ship Hop's optional Property Input transform (plugins/transforms/propertyinput), which reads Java .properties and INI files into a pipeline. It is the only part of Hop that uses the ini4j 0.5.4 library, and ini4j has a published vulnerability, CVE-2022-41404, for which no fixed release exists upstream, so the transform and ini4j were removed rather than shipped with a known unfixed advisory. Every other transform is present and Hop starts without it. Pipelines that use Property Input will report the transform as missing; read such files with the Text File Input transform instead, or add the plugin yourself from the upstream Hop 2.20.0 release if you accept that risk.

Logs: sudo journalctl -u hop-web for Hop Web and Tomcat, /var/log/nginx/access.log for requests (with the real client address).

Step 12: Security recommendations

• Allow port 22 in the security group from your own networks only, and port 443 only from the networks your designers and API clients use.
• Treat HOP_ADMIN_PASSWORD like a root password: anyone who signs in to Hop Web can run pipelines and workflows, including shell actions, as the hop user.
• Open private networks with hop-egress-allow one subnet at a time, only for the databases and services your pipelines need.
• Keep /etc/hop/encoder.key and EBS snapshots of the data volume with your backups: the snapshots hold your projects, the key decrypts the passwords in them.
• Use your own DNS name (Step 10), and rotate the password when someone leaves or it may have been exposed.
• Keep the operating system patched; unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.

Support and licensing

Apache Hop and Apache Tomcat are open source under the Apache License 2.0; there is no per user or per CPU licence fee. The licences are in /usr/share/doc/apache-hop/. Apache Hop, Hop and the Apache feather are trademarks of the Apache Software Foundation; cloudimg is not affiliated with or endorsed by the Apache Software Foundation. The AMI price covers cloudimg's hardening, first boot setup and support.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk
• Support: 24/7, with a 24 hour response SLA