Apache ManifoldCF 2.30 on AWS User Guide

Overview

This guide covers launching and using Apache ManifoldCF 2.30 on Ubuntu 24.04 from cloudimg on Amazon EC2. It is one version of the cloudimg developer and API tools listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Apache ManifoldCF is an open source framework for moving content from the systems where it lives (file shares, web sites, SharePoint, CMIS repositories, databases, RSS feeds, email stores and more) into a search engine, on a schedule, incrementally, and with each document's access rights carried along so that search results can be filtered by who is allowed to see them. You define repository connections (where documents come from), transformation connections (for example text extraction with Apache Tika), output connections (where documents go) and jobs that bind them together, either in the browser based crawler UI or through a JSON API.

This image runs ManifoldCF 2.30 as a single node with PostgreSQL 16 as its database and a local Apache Solr 9.11.0 search core named cloudimg as a ready made output target, plus a small sample corpus so you can prove a crawl end to end in one command.

The cloudimg image adds:

• HTTPS from the first boot. nginx serves the crawler UI and the API on port 443 with a certificate generated on your instance's first boot. Port 80 redirects to HTTPS, and requests for a host name the instance does not serve are dropped.
• Passwords unique to your instance, no admin/admin. First boot generates the crawler UI password for admin (upstream ships admin/admin), a separate HTTP Basic credential for the JSON API, the PostgreSQL password and a Solr administrator password, and writes them to /root/manifoldcf-credentials.txt. No password is baked into the image; the database itself is created on your instance.
• ManifoldCF's web applications are not on the network. ManifoldCF's built in Jetty listens on a private Unix socket that only nginx can open, not on a TCP port. Upstream's API and authority service have no authentication of their own, so this keeps them away from other local users and from the crawler's own outbound requests. The authority service is not published at all.
• Solr locked down. Solr listens on 127.0.0.1 only, its administration APIs (core admin, configuration, schema, security, metrics) require the Solr administrator password, remote streaming is off and its stop port is disabled.
• Crawler safety. Every service, the crawler included, is blocked from the instance metadata service, so a crawl job pointed at 169.254.169.254 gets nothing. The XML external entity flaw in the bundled Apache Tika 1.28.1 (CVE-2025-54988 / CVE-2025-66516, triggered by a crafted PDF) is closed in configuration and re-tested on every build.
• Hardened defaults. ManifoldCF runs as its own unprivileged user in a systemd sandbox, upstream's example configurations (with their default passwords) are removed, the browser session cookie is Secure and HttpOnly, and cross site form posts and WebSocket upgrades are refused. The AMI requires IMDSv2.
• Fully patched Ubuntu 24.04 LTS at build time, with unattended security upgrades switched on.

What is on the image: Apache ManifoldCF 2.30 (official binary distribution, checksum verified), Apache Solr 9.11.0 (checksum verified), PostgreSQL 16, OpenJDK 21 and nginx from Ubuntu, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet (or a private route you use for management).
• A security group that allows TCP 22 (SSH), TCP 443 (crawler UI and API) and TCP 80 (redirects to HTTPS) from your own networks only.
• Outbound network access from the instance to the repositories you plan to crawl.

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Apache ManifoldCF 2.30 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large (8 GiB; ManifoldCF and Solr size their memory from the instance at first boot, so a larger type is used automatically), your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=manifoldcf}]'

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check the services

First boot creates the database and the passwords (about a minute), then PostgreSQL, Solr, ManifoldCF and nginx are active:

systemctl is-active manifoldcf-firstboot postgresql@16-main solr manifoldcf nginx

active
active
active
active
active

Only SSH and nginx listen on the network. PostgreSQL and Solr (java) listen on loopback, and ManifoldCF itself has no TCP port at all: nginx reaches it through a Unix socket that only the manifoldcf user and nginx's group can open.

sudo ss -tlnp | awk 'NR==1 || /java|nginx|sshd|postgres/' | awk '{print $1, $4, $6}' | sed -E 's/,pid=[0-9]+,fd=[0-9]+//g'
sudo ls -l /run/manifoldcf/jetty.sock | awk '{print $1, $3, $4, $NF}'

State Local Peer
LISTEN 0.0.0.0:22 users:(("sshd"),("systemd"))
LISTEN 0.0.0.0:80 users:(("nginx"),("nginx"),("nginx"))
LISTEN 0.0.0.0:443 users:(("nginx"),("nginx"),("nginx"))
LISTEN 127.0.0.1:5432 users:(("postgres"))
LISTEN [::]:22 users:(("sshd"),("systemd"))
LISTEN [::]:80 users:(("nginx"),("nginx"),("nginx"))
LISTEN [::ffff:127.0.0.1]:8983 users:(("java"))
LISTEN [::]:443 users:(("nginx"),("nginx"),("nginx"))
srwxrwx--- manifoldcf www-data /run/manifoldcf/jetty.sock

The versions on the image:

sudo grep -o 'versionstring" value="[^"]*"' /var/lib/manifoldcf/properties.xml
readlink -f /opt/solr
java -version 2>&1 | head -1
psql --version

versionstring" value="2.30"
/opt/solr-9.11.0
openjdk version "21.0.12.1" 2026-08-18
psql (PostgreSQL) 16.15 (Ubuntu 16.15-0ubuntu0.24.04.1)

Step 4: Get your URLs and passwords

sudo cat /root/manifoldcf-credentials.txt

The file holds the address ManifoldCF is served as (your instance's public DNS name by default) and every credential generated on this instance:

MCF_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/mcf-crawler-ui/
MCF_ADMIN_USER=admin
MCF_ADMIN_PASSWORD=<24 letters and digits>
MCF_API_URL=https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/mcf-api-service/json
MCF_API_USER=admin
MCF_API_PASSWORD=<24 letters and digits>
# Internal (loopback only, not published):
MCF_DB_NAME=manifoldcf
MCF_DB_USER=manifoldcf
MCF_DB_PASSWORD=<32 letters and digits>
SOLR_URL=http://127.0.0.1:8983/solr/cloudimg
SOLR_ADMIN_USER=solradmin
SOLR_ADMIN_PASSWORD=<24 letters and digits>

The crawler UI password and the API password are different on purpose: the API is protected by nginx with its own credential, because ManifoldCF 2.30's API service has no working login of its own.

Step 5: Sign in to the crawler UI

Open MCF_URL in your browser. The certificate is self signed (generated on your instance), so accept the browser warning once, or install your own certificate (Step 10). Sign in with admin and MCF_ADMIN_PASSWORD.

The left menu groups everything ManifoldCF manages: Outputs (transformation and output connections), Authorities (document security), Repositories, Jobs, and the status and history reports.

Step 6: Run the sample crawl

The image ships a sample corpus of six short documents in /var/lib/manifoldcf/sample-corpus, each carrying a unique marker phrase. One command proves the whole pipeline through the public HTTPS edge: it checks that the API refuses a missing or wrong credential and accepts yours, defines a file system repository connection, a Tika extraction stage and the Solr output connection, runs a job, and then checks the documents in Solr by count and by content (a crawl that ingests nothing still reports "done", so the job status alone proves nothing):

sudo manifoldcf-crawl-roundtrip.sh

[roundtrip] API: no credential 401, wrong credential 401, per-instance credential 200
[roundtrip] Solr output, Tika extraction and file-system repository connections report working
[roundtrip] started job 1792486675643
[roundtrip] job reached 'done' (informational; the assertions are on Solr)
ROUNDTRIP_OK: 6 documents crawled from /var/lib/manifoldcf/sample-corpus into Solr with content verified

The job now appears in the crawler UI under Jobs > List all Jobs:

View shows the job's pipeline: the file system repository, the Tika transformation and the Solr output:

Jobs > Status and Job Management shows each run, with Start to run it again:

(The 7 documents are the six files plus the folder that holds them.)

Step 7: Search what you crawled

The documents are in the local Solr core. Searching is open on loopback, so this works from the instance as any user:

curl -s --get --data-urlencode 'q="document level security"' --data-urlencode 'fl=id' --data-urlencode 'wt=json' http://127.0.0.1:8983/solr/cloudimg/select | python3 -c 'import json,sys; r=json.load(sys.stdin)["response"]; print("matches:", r["numFound"]); [print(d["id"]) for d in r["docs"]]'

matches: 1
file:/var/lib/manifoldcf/sample-corpus/05-document-security.txt

The Solr connection the crawl uses is listed under Outputs > List Output Connections:

To send documents to a different search engine (for example OpenSearch, Elasticsearch or a remote Solr), add an output connection of that type and point a job at it.

Step 8: Use the JSON API

The API answers at MCF_API_URL with HTTP Basic authentication (MCF_API_USER / MCF_API_PASSWORD). Pass the credential to curl from a file, so it never appears on a command line. For example, list the output connector types this installation knows:

H=$(sudo sed -n 's#^MCF_URL=https://\([^/]*\)/.*#\1#p' /root/manifoldcf-credentials.txt)
CFG=$(mktemp); chmod 600 "$CFG"
sudo sed -n 's/^MCF_API_PASSWORD=\(.*\)/user = "admin:\1"/p' /root/manifoldcf-credentials.txt > "$CFG"
curl -sk -K "$CFG" --resolve "$H:443:127.0.0.1" "https://$H/mcf-api-service/json/outputconnectors" | python3 -c 'import json,sys; [print(c["_children_"][0]["_value_"]) for c in json.load(sys.stdin)["outputconnector"]]' | head -5
rm -f "$CFG"

AmazonCloudSearch
CMIS
ElasticSearch
File system
HDFS

The ManifoldCF documentation describes every API resource (connections, jobs, start/<job id>, jobstatuses, history reports). When you script against the API from another machine, keep its address restricted in the security group and send the credential the same way.

Step 9: Security checks

Port 80 sends browsers to HTTPS, the crawler UI answers, the API refuses a request without its credential, the unauthenticated authority service is not published, and nginx closes the connection for a host name the instance does not serve (000 means no answer):

H=$(sudo sed -n 's#^MCF_URL=https://\([^/]*\)/.*#\1#p' /root/manifoldcf-credentials.txt)
curl -s -o /dev/null -w 'port 80: %{http_code} -> %{redirect_url}\n' --resolve "$H:80:127.0.0.1" "http://$H/"
curl -sk -o /dev/null -w 'crawler UI sign-in page: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/mcf-crawler-ui/login.jsp"
curl -sk -o /dev/null -w 'API without a credential: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/mcf-api-service/json/jobs"
curl -sk -o /dev/null -w 'authority service: %{http_code}\n' --resolve "$H:443:127.0.0.1" "https://$H/mcf-authority-service/UserACLs?username=test"
curl -s -o /dev/null -w 'unknown host name: %{http_code}\n' -H 'Host: example.com' http://127.0.0.1/ || true

port 80: 301 -> https://ec2-<public-ip-dashed>.<region>.compute.amazonaws.com/
crawler UI sign-in page: 200
API without a credential: 401
authority service: 403
unknown host name: 000

Solr's administration APIs need the Solr administrator password:

curl -s -o /dev/null -w 'Solr core admin without a credential: %{http_code}\n' 'http://127.0.0.1:8983/solr/admin/cores?action=STATUS'
CFG=$(mktemp); chmod 600 "$CFG"
sudo sed -n 's/^SOLR_ADMIN_PASSWORD=\(.*\)/user = "solradmin:\1"/p' /root/manifoldcf-credentials.txt > "$CFG"
curl -s -o /dev/null -w 'Solr core admin as solradmin: %{http_code}\n' -K "$CFG" 'http://127.0.0.1:8983/solr/admin/cores?action=STATUS'
rm -f "$CFG"

Solr core admin without a credential: 401
Solr core admin as solradmin: 200

ManifoldCF runs unprivileged in a sandbox, and no service (the crawler included) can reach the instance metadata service:

systemctl show manifoldcf -p User -p ProtectSystem -p NoNewPrivileges -p IPAddressDeny
for u in solr nginx postgresql@16-main; do echo "$u: $(systemctl show $u -p IPAddressDeny --value)"; done

IPAddressDeny=fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32
User=manifoldcf
ProtectSystem=strict
NoNewPrivileges=yes
solr: fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32
nginx: fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32
postgresql@16-main: fd00:ec2::254/128 169.254.169.254/32 168.63.129.16/32

The certificate was generated on this instance and names its own addresses, and nginx passes ManifoldCF the configured host name and the real client address from the connection, never values a client supplied:

H=$(sudo sed -n 's#^MCF_URL=https://\([^/]*\)/.*#\1#p' /root/manifoldcf-credentials.txt)
echo | openssl s_client -connect 127.0.0.1:443 -servername "$H" 2>/dev/null | openssl x509 -noout -subject -ext subjectAltName
grep -E 'proxy_set_header (Host|X-Forwarded-For|X-Real-IP)|proxy_pass' /etc/nginx/manifoldcf-proxy.conf

subject=CN = ec2-<public-ip-dashed>.<region>.compute.amazonaws.com, O = cloudimg
X509v3 Subject Alternative Name:
    DNS:ec2-<public-ip-dashed>.<region>.compute.amazonaws.com, IP Address:<public-ip>, IP Address:<private-ip>, IP Address:127.0.0.1, DNS:localhost
proxy_pass http://unix:/run/manifoldcf/jetty.sock:;
proxy_set_header Host $mcf_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;

Step 10: Your own DNS name and certificate

Point a DNS name at the instance (an Elastic IP keeps the address stable), then tell ManifoldCF to serve under that name. This generates a new self signed certificate for it, updates nginx and the credentials file, and is remembered across reboots:

sudo manifoldcf-set-hostname manifoldcf.example.com

To use a certificate your browsers trust, replace /etc/nginx/ssl/manifoldcf.crt and /etc/nginx/ssl/manifoldcf.key (mode 0600) with your certificate chain and key, then run sudo systemctl reload nginx. To use Let's Encrypt, open port 80 to the validation servers and use certbot's webroot or DNS method with the same two paths.

Step 11: Change passwords

Each helper generates a new random password (or reads one of 16 to 64 letters and digits from standard input with --stdin), applies it, and records it in /root/manifoldcf-credentials.txt. The password is never placed on a command line:

sudo manifoldcf-set-password ui      # crawler UI login for admin (restarts ManifoldCF)
sudo manifoldcf-set-password api     # HTTP Basic credential for the JSON API (bcrypt, no restart)
sudo manifoldcf-set-password solr    # Solr administrator solradmin (restarts Solr)

The PostgreSQL password is used only by ManifoldCF on this instance. To rotate it, set the new value with ALTER ROLE manifoldcf PASSWORD in sudo -u postgres psql (type the statement at the prompt rather than on the command line), update org.apache.manifoldcf.database.password and org.apache.manifoldcf.dbsuperuserpassword in /var/lib/manifoldcf/properties.xml, update MCF_DB_PASSWORD in the credentials file and restart ManifoldCF with sudo systemctl restart manifoldcf.

Step 12: Crawling your own content

• Repository connections for most sources are built in: file systems (mount shares under /mnt or /srv; the crawler runs as the manifoldcf user and can read only what that user can read), web sites, RSS, CMIS, SharePoint, JDBC databases, email and others. Connectors that need a vendor's own client library (for example some document management systems) expect you to place that library in /opt/manifoldcf/connector-lib-proprietary under its licence.
• Web crawls: set each job's Inclusions and Exclusions so the crawler stays on the sites you mean to index. The crawler cannot reach the instance metadata service or ManifoldCF's own API, but any other address the instance can reach (including other hosts in your VPC) is reachable if a crawled page links to it.
• Text extraction uses ManifoldCF's Tika transformation (as in the sample job). On this image Tika does not extract PDF form (AcroForm/XFA) fields, the setting that closes the Tika XML external entity flaw; ordinary PDF page text, Office documents, HTML and plain text are extracted as usual. RAR archives are not parsed.
• Security trimming: add an authority connection for your source (for example Active Directory or SharePoint) and the matching authority group to the job, so each document is indexed with its access tokens.
• Memory: first boot gives ManifoldCF about 20% and Solr about 13% of the instance's memory. Larger crawls want a larger instance type; adjust MCF_HEAP in /etc/default/manifoldcf and SOLR_HEAP in /etc/default/solr.in.sh if you need to.

Step 13: Server components

For more than one node (separate agents and web processes, or ZooKeeper based synchronisation), follow the ManifoldCF documentation's multi process deployment; this image's single process layout is the upstream ManifoldCFJettyRunner.

Step 14: Security recommendations

• Allow ports 22, 80 and 443 in the security group from your own networks only.
• Keep the crawler UI, API and Solr passwords secret, and rotate them if they may have been shared (Step 11).
• Use your own DNS name and a certificate your browsers trust (Step 10).
• Restrict every web crawl job with inclusion and exclusion rules, and give repository connections read only service accounts.
• Keep the operating system patched; unattended security upgrades are on, and sudo apt-get update && sudo apt-get upgrade applies everything else.
• Back up the PostgreSQL database (sudo -u postgres pg_dump manifoldcf) and /var/lib/manifoldcf/properties.xml; the Solr index can be rebuilt by re-running your jobs.

Support and licensing

Apache ManifoldCF and Apache Solr are open source under the Apache License 2.0 (licences and notices at /usr/share/doc/manifoldcf/ and /usr/share/doc/solr/); PostgreSQL, OpenJDK and nginx are distributed by Ubuntu under their own open source licences. There is no per user or per CPU licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk
• Support: 24/7, with a 24 hour response SLA