Apache ManifoldCF on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and operation of Apache ManifoldCF on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. ManifoldCF is an open source content ingestion framework. Repository connectors pull documents out of the systems they live in — filesystems, web sites, SharePoint, RSS feeds, relational databases, email stores and document management systems — and output connectors push them into a search engine. Between the two sits a crawl scheduler that records per document state, so a repeat run reprocesses only what actually changed instead of re indexing everything from scratch.

What distinguishes ManifoldCF from a plain crawler is its security model. Its authority services translate a user identity into the access tokens that user holds in the source repository, and those tokens are indexed alongside each document. A search front end can then return only the documents a given person is entitled to open, which matters in front of SharePoint or filesystem content carrying real access control lists — showing someone a document title they are not cleared for is itself a disclosure.

The cloudimg image ships the free and open source, Apache-2.0 licensed ManifoldCF 2.30, and does the part that is normally a day of work: a complete, wired up pipeline. Apache Solr 9.10.1 is installed locally as a working output target, PostgreSQL 16 backs the crawl database in place of the evaluation grade embedded store upstream's examples use, and a small self authored sample corpus plus a one command round trip script let you prove the whole path — crawl, extract, index, retrieve — before you point it at anything of your own. Backed by 24/7 cloudimg support.

Apache, Apache ManifoldCF, Apache Solr and their logos are trademarks of The Apache Software Foundation. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by the ASF. It ships the free and open source software, unmodified.

What is included:

• Apache ManifoldCF 2.30 (Apache-2.0), installed from the official Apache CDN and SHA-512 verified at build time
• Apache Solr 9.10.1 (Apache-2.0) as a local, ready to use output target, with a cloudimg core already created
• PostgreSQL 16 as the crawl database, with the schema initialised and every connector registered
• OpenJDK 21 — required, not merely supported, because ManifoldCF 2.30's classes are Java 21 bytecode
• nginx as the only public surface, publishing the crawler console and the JSON API and nothing else
• The Tika transformation stage wired into the crawl pipeline so document text is extracted and indexed, not just filenames
• A six document sample corpus and manifoldcf-crawl-roundtrip.sh, a one command end to end proof
• Four secrets generated per VM at first boot — console login, API credential, database password and Jetty shutdown token
• Ubuntu 24.04 LTS base, fully patched at build time, with unattended security upgrades enabled

How the pieces fit together

It is worth understanding the topology before you deploy, because it explains several of the choices below.

Component	Port	Bound to	Reachable from outside?
nginx	80	all interfaces	Yes — the only public surface
ManifoldCF (Jetty)	8345	127.0.0.1	No
Apache Solr	8983	127.0.0.1	No
PostgreSQL	5432	127.0.0.1	No

ManifoldCF runs as a single JVM hosting the agents daemon in process alongside the three standard web contexts (/mcf-crawler-ui, /mcf-api-service, /mcf-authority-service) at their normal paths, so upstream documentation and tutorials apply to this image unchanged. All three contexts share one Jetty connector, and the authority service has no credential mechanism upstream — so publishing Jetty directly would expose an unauthenticated service. That is why Jetty is held on loopback and nginx republishes only the two contexts that can be protected, returning 403 for the authority service explicitly.

If you outgrow a single node, ManifoldCF supports multiprocess and ZooKeeper topologies; that is a migration away from this image's layout rather than a setting, and upstream documents it.

Prerequisites

• An Azure subscription with permission to create VMs and edit network security groups
• The Azure CLI installed and signed in, if you are deploying from the command line
• An SSH key pair for administrative access to the VM

Sizing

The recommended size for this image is Standard_B2s (2 vCPU, 4 GiB). The image is tuned to fit that honestly: the ManifoldCF JVM is capped at a 768 MB heap and Solr at 512 MB, leaving room for PostgreSQL and the OS. First boot recomputes both heaps from the memory actually present, so a larger VM benefits automatically without you editing anything.

Choose a larger size when your crawl is larger. The two things that push you up are corpus size (Solr's index and its caches) and crawl concurrency. The image ships crawler.threads=10 and a database handle pool of 20 against a PostgreSQL max_connections of 50 — a deliberate reduction from upstream's defaults of 50 threads and 100 handles, which would exhaust PostgreSQL's connection limit on any machine this size. If you raise the thread count, raise the pool and max_connections with it, and keep upstream's documented constraint: workers + deletes + expirations + cleanup + 10 must stay below the maximum handle count.

Step 1: Deploy from the Azure Portal

1. In the Azure Portal, search the Marketplace for Apache ManifoldCF on Ubuntu 24.04 LTS by cloudimg and select Create.
2. Choose your subscription, resource group and region.
3. Set the VM size to Standard_B2s or larger.
4. Provide an administrator username and SSH public key.
5. On the Networking tab, allow inbound HTTP (80) and SSH (22). Restrict the source to your own address range rather than leaving either open to the internet.
6. Review and create.

Step 2: Deploy from the Azure CLI

Run these on your own machine. Replace the resource group, name and region to suit.

az group create --name manifoldcf-rg --location eastus

az vm create \
  --resource-group manifoldcf-rg \
  --name manifoldcf-vm \
  --image cloudimg:manifoldcf-ubuntu-24-04:default:latest \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

az vm open-port \
  --resource-group manifoldcf-rg \
  --name manifoldcf-vm \
  --port 80 --priority 1010

The az vm open-port call publishes the console port. Restrict its source to your own network afterwards, in the network security group, rather than leaving port 80 open to the internet.

Step 3: Connect to your VM

ssh azureuser@<vm-ip>

First boot generates this VM's secrets, initialises the database schema and starts the crawler. It takes a minute or two on a Standard_B2s. The steps below all confirm it finished.

Step 4: Confirm the services are running

Four services make up the appliance. All four should report active, and all four are enabled at boot:

systemctl is-active postgresql solr manifoldcf nginx

active
active
active
active

systemctl is-enabled postgresql solr manifoldcf nginx

enabled
enabled
enabled
enabled

Confirm the JVM is OpenJDK 21. This is not cosmetic — ManifoldCF 2.30's class files are Java 21 bytecode and an older JVM physically cannot load them:

java -version 2>&1 | head -1

openjdk version "21.0.11" 2026-04-21

Step 5: Read the per instance credentials

This image ships no default password. Four secrets are generated on the first boot of every VM and written to a file only root can read.

sudo ls -l /root/manifoldcf-credentials.txt

-rw------- 1 root root 659 Jul 20 05:39 /root/manifoldcf-credentials.txt

sudo cat /root/manifoldcf-credentials.txt

The file records the console URL, the API URL and the following secrets:

Key	Protects
MCF_ADMIN_USER / MCF_ADMIN_PASSWORD	The crawler console login
MCF_API_USER / MCF_API_PASSWORD	HTTP Basic on the JSON API, enforced by nginx
MCF_DB_USER / MCF_DB_PASSWORD	The PostgreSQL role backing the crawl database

A fourth secret, the Jetty shutdown token, is rotated at the same time and is not something you need to use. Upstream ships that token as the literal string secret_token in every options file it publishes, which is exactly the kind of value an image must not inherit.

The URLs in the file are written against the VM's own address. To reach the console from your workstation, use the VM's public IP.

Step 6: Sign in to the crawler console

Open http://<vm-ip>/mcf-crawler-ui/ in a browser. The root path redirects there for you.

Sign in with the MCF_ADMIN_USER and MCF_ADMIN_PASSWORD values from Step 5. You land on the console home with the main navigation down the left: Outputs, Authorities, Repositories, Jobs, Status Reports and History Reports.

Step 7: Verify the security model

Four checks, each of which you can run yourself. They matter because a crawler holds credentials for every repository it reads.

The console is published and answers on port 80 — a 302 because an anonymous request is redirected to the login form rather than being shown any content:

curl -s -o /dev/null -w 'crawler console: %{http_code}\n' http://127.0.0.1/mcf-crawler-ui/

crawler console: 302

The authority service is explicitly refused rather than merely unpublished, so the intent is legible in the configuration:

curl -s -o /dev/null -w 'authority service: %{http_code}\n' http://127.0.0.1/mcf-authority-service/

authority service: 403

The JSON API refuses an unauthenticated caller:

curl -s -o /dev/null -w 'API without credentials: %{http_code}\n' http://127.0.0.1/mcf-api-service/json/outputconnections

API without credentials: 401

And the three internal components are bound to loopback only, so nothing but nginx is exposed:

ss -ltn '( sport = :8345 or sport = :8983 or sport = :5432 )' | awk 'NR>1{print $4}'

127.0.0.1:5432
[::ffff:127.0.0.1]:8983
[::ffff:127.0.0.1]:8345

A note on how the API is protected. ManifoldCF's own API servlet authenticates with a hardcoded empty credential internally, so the upstream apilogin.* properties cannot be used to secure it — setting them does not add a password, it makes every request fail permanently with no way to supply one. This image therefore leaves those properties at their upstream empty default and enforces authentication where it genuinely can be enforced: nginx HTTP Basic on /mcf-api-service, with Jetty unreachable except through nginx. Do not "harden" the image by populating apilogin.name and apilogin.password; it will lock you out of your own API.

Step 8: Run the built in crawl round trip

The image ships a six document sample corpus, self authored by cloudimg so no third party content with unclear licensing is included:

ls -1 /var/lib/manifoldcf/sample-corpus

01-what-is-manifoldcf.txt
02-repository-connectors.txt
03-output-connectors.txt
04-jobs-and-scheduling.txt
05-document-security.txt
06-operating-the-appliance.txt

Each document carries a distinct planted sentinel phrase, which is what makes the round trip a real proof rather than a smoke test. Run it:

sudo /usr/local/sbin/manifoldcf-crawl-roundtrip.sh

It ends with:

ROUNDTRIP_OK: 6 documents crawled from /var/lib/manifoldcf/sample-corpus into Solr with content verified

The script defines the Solr output connection, the Tika text extraction stage and a filesystem repository connection over the corpus, creates a job, runs it to completion, and then asserts against Solr — document count and document content — rather than against the job's status. That distinction is the point: a ManifoldCF job that ingests zero documents still finishes and still reports a terminal status, so asserting on job status would pass while the product was entirely broken.

Step 9: Retrieve the indexed content

Confirm the documents are in the index:

curl -s 'http://127.0.0.1:8983/solr/cloudimg/select?q=*:*&rows=0' | grep numFound

    "numFound":6,

Then retrieve one by its planted content, not by its filename. Note the quoted phrase — Solr's default analyzer splits CLOUDIMG-SENTINEL-ALPHA on the hyphens and matches any of the parts, so an unquoted query would match every document in the corpus and prove nothing:

curl -s --get --data-urlencode 'q="CLOUDIMG-SENTINEL-ALPHA"' 'http://127.0.0.1:8983/solr/cloudimg/select?fl=id&rows=3'

Exactly one document comes back, and it is the right one:

    "numFound":1,
    "docs":[{
        "id":"file:/var/lib/manifoldcf/sample-corpus/01-what-is-manifoldcf.txt"
    }]

A control query for a sentinel that was never planted returns nothing, which confirms the query is discriminating rather than matching everything:

curl -s --get --data-urlencode 'q="CLOUDIMG-SENTINEL-NEVERPLANTED"' 'http://127.0.0.1:8983/solr/cloudimg/select?rows=0' | grep numFound

    "numFound":0,

Step 10: Inspect the crawl in the console

Everything the round trip built is visible in the console, and this is where you will work when you build your own crawls.

Under Jobs → Status and Job Management you can see the sample job, its terminal status and — the number that actually matters — how many documents it processed:

Under Repositories → List Repository Connections you can see the filesystem connection the job crawls, along with its connection type and authority group:

Step 11: Build your own crawl

The workflow is always the same three objects, in this order:

1. An output connection — where documents go. cloudimg-solr already exists and points at the local Solr core, so you can reuse it. Add your own under Outputs → List Output Connections if you are indexing elsewhere.
2. A repository connection — where documents come from. Under Repositories → List Repository Connections, choose a connection type (File system, Web, RSS, JDBC, and so on), give it the credentials for that system, and use the View action to confirm it reports working before going further.
3. A job — what to crawl and when. Under Jobs → List all Jobs, create a job that binds a repository connection to an output connection, set its document specification (paths, include and exclude rules), and optionally give it a schedule.

Two things that are easy to miss:

• Add the Tika extraction stage to the job's pipeline if you want document text indexed rather than just metadata. This image's sample job does exactly that, so use it as a reference. Text extraction is performed in the crawler tier, which is both the Solr project's own recommendation and what makes extraction work reliably here.
• Test the connection before you build the job. A repository connection that does not report working will produce a job that runs, completes and ingests nothing.

Step 12: Using the JSON API

The API is at /mcf-api-service/json behind HTTP Basic. Anything the console can do, the API can do — it is the right way to script crawls.

sudo bash -c 'source /root/manifoldcf-credentials.txt; curl -s -u "$MCF_API_USER:$MCF_API_PASSWORD" http://127.0.0.1/mcf-api-service/json/outputconnections' | head -c 200

Job status is available the same way, and is how you would poll a scheduled crawl:

sudo bash -c 'source /root/manifoldcf-credentials.txt; curl -s -u "$MCF_API_USER:$MCF_API_PASSWORD" http://127.0.0.1/mcf-api-service/json/jobstatuses' | head -c 400

One upstream quirk worth knowing before you script against it. ManifoldCF's JSON serialiser emits attribute keys as _attribute_name_ (with a trailing underscore) while its parser expects _attribute_name (without one). Reading an object with GET and feeding that response straight back into a PUT therefore stores a subtly corrupted configuration, and the connector will report a missing parameter for a value you can plainly see in the JSON. Build request bodies to the parser's spelling rather than by round tripping a response.

Step 13: Managing the services

systemctl status manifoldcf --no-pager --lines=0

To restart the crawler after a configuration change:

sudo systemctl restart manifoldcf.service

Logs go to the journal:

sudo journalctl -u manifoldcf -n 100 --no-pager

Solr and PostgreSQL are ordinary systemd services too (solr.service, postgresql.service) and are started before ManifoldCF at boot.

Step 14: Changing the credentials

The console login lives in ManifoldCF's properties file, which is 0640 root:manifoldcf:

sudo sed -i 's|^\(.*org.apache.manifoldcf.login.password.*value="\)[^"]*|\1<new-password>|' /opt/manifoldcf/properties.xml
sudo systemctl restart manifoldcf.service

The API credential is an nginx htpasswd entry and does not require restarting ManifoldCF:

sudo htpasswd -B /etc/nginx/manifoldcf-api.htpasswd <api-user>
sudo systemctl reload nginx

Keep /root/manifoldcf-credentials.txt in step with any change you make, or treat the live configuration as authoritative from then on.

On password storage. ManifoldCF writes passwords into properties.xml in plaintext, and this image does not pretend otherwise. ManifoldCF does offer an "obfuscated" property variant, but its obfuscation uses a passcode and salt compiled into the shipped jar and identical in every installation worldwide — anyone holding the jar can reverse it. Using it would be security theatre, so protection here is filesystem permissions: the file is readable only by root and the service account.

Step 15: Use your own domain and HTTPS (production)

For anything beyond a trial, put a real hostname and a certificate in front of nginx. Install a certificate with Certbot against the nginx site this image already ships, then reload nginx. Everything reaches the appliance through nginx already, so no other component needs changing — and because the console login and the API credential both travel over that connection, HTTPS is the difference between authenticating privately and authenticating in clear text over the network.

Step 16: Security recommendations

• Restrict the source addresses on the network security group rule for port 80 to your own network. The console can configure crawls against your internal systems.
• Terminate TLS before using this in production, per Step 15.
• Treat the repository credentials as the sensitive part. A crawler necessarily holds working credentials for every system it reads; the console is as privileged as the sum of those systems.
• Do not populate apilogin.* — see the note in Step 7. It does not harden the API, it disables it.
• Use the authority services if your source content carries access control lists and you are building a user facing search experience. Indexing permissions alongside documents is what stops search results leaking titles to people who cannot open them.
• Keep the OS patched. Unattended security upgrades are enabled on this image by default.

Step 17: Support and Licensing

Apache ManifoldCF and Apache Solr are free and open source software under the Apache-2.0 licence. The ManifoldCF binary distribution bundles third party components under BSD (the PostgreSQL JDBC driver, HSQLDB), CDDL 1.0 (several Sun/GlassFish components), EPL 1.0 (the Eclipse compiler) and MIT (json-simple). No GPL-only, SSPL or AGPL code ships in this image — the ASF excludes connectors requiring such jars from its release, and this image does not add them.

Apache Solr was chosen as the bundled search target deliberately: it is ManifoldCF's best supported output connector, and it is Apache-2.0. Elasticsearch was not viable for an image whose listing states Apache-2.0, because Elasticsearch is distributed under the SSPL. ManifoldCF's Elasticsearch and OpenSearch output connectors remain available to you if you are pointing at an existing cluster of your own.

This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by The Apache Software Foundation. The Apache, ManifoldCF and Solr names are used only to identify the software included.

For help with the image, contact cloudimg support, which is available 24/7. For questions about ManifoldCF itself, see the upstream project at manifoldcf.apache.org.