Apache Nutch 1.22 on Ubuntu 24.04 on Azure User Guide

Overview

Apache Nutch is a mature, extensible open-source web crawler built on Apache Lucene. This cloudimg appliance ships Nutch 1.22 running in local mode (Hadoop libraries bundled, but no Hadoop cluster) wired to a bundled Apache Solr 9.10 search index, so you can crawl web content and full-text search it from a single node. Nutch fetches, parses, de-duplicates and scores pages as batch jobs and hands the results to Solr over authenticated HTTP; Solr serves faceted, full-text and vector search over everything crawled. Both components are Apache-2.0 — there is no Elasticsearch or any SSPL-licensed datastore in the stack.

What is included:

• Apache Nutch 1.22 (web crawler) at /opt/nutch, running in local mode on OpenJDK 11
• Apache Solr 9.10.1 (Lucene 9 search index) at /opt/solr, running on OpenJDK 17, listening on 8983/tcp
• A pre-created Solr nutch core (schemaless _default configset) that Nutch indexes into
• solr.service running as the unprivileged solr user; Nutch is a batch CLI (no daemon)
• BasicAuthPlugin + RuleBasedAuthorizationPlugin protecting every Solr endpoint
• A per-VM cloudimg admin password generated at first boot (never shipped), injected into both Solr and Nutch's Solr-indexer config
• /usr/local/bin/nutch-demo-crawl — a bundled end-to-end demo that crawls a local page, indexes it into Solr and queries it back
• 24/7 cloudimg support

Prerequisites

Active Azure subscription, SSH key, VNet + subnet. Standard_B2s (4 GB RAM) runs a crawl-and-index cycle out of the box (Solr heap 512m, Nutch batch JVM up to 1024m). For large-scale crawling and bigger indexes raise to D4s/D8s and increase SOLR_HEAP and NUTCH_HEAPSIZE. NSG inbound: allow 22/tcp from your management CIDR and 8983/tcp from any client CIDR that needs the Solr Admin UI or REST API.

Step 1-3: Deploy + SSH (standard pattern)

ssh azureuser@<vm-ip>

Step 4: Service Status + Versions

sudo systemctl is-active solr.service
/opt/solr/bin/solr version
NUTCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 /opt/nutch/bin/nutch version | grep -i nutch
ss -tln | grep 8983

solr.service is active and listening on 8983; the bundled Solr reports 9.10.1 and Nutch reports 1.22. Nutch has no long-running daemon — it is invoked as a batch job when you crawl.

Step 5: Auth Wall + System Info

curl -s -o /dev/null -w 'no-auth: HTTP %{http_code}\n' http://127.0.0.1:8983/solr/admin/info/system
PASS=$(sudo grep '^SOLR_ADMIN_PASSWORD=' /stage/scripts/nutch-credentials.log | cut -d= -f2-)
curl -sf -u "cloudimg:${PASS}" 'http://127.0.0.1:8983/solr/admin/info/system?wt=json' | head -c 300

The first call returns 401 — the BasicAuthPlugin auth wall — so no unauthenticated Solr admin is reachable off-host. The second call authenticates as cloudimg and returns the system info JSON including solr-spec-version.

Step 6: Read Per-VM Credentials

sudo cat /stage/scripts/nutch-credentials.log

Pick up SOLR_ADMIN_URL, SOLR_ADMIN_USER (cloudimg), SOLR_ADMIN_PASSWORD, and NUTCH_CORE_URL. The password is a fresh 32-character hex string generated on this VM's first boot — no two VMs share credentials, and the same secret is already injected into Nutch's Solr-indexer config so the crawler authenticates automatically.

Step 7: End-to-End Crawl → Index → Search Demo

The image bundles a self-contained demo that proves the whole pipeline: it serves a throwaway HTML page on 127.0.0.1:8899, runs one Nutch crawl round that indexes into the Solr nutch core, then queries the document back out with the per-VM credentials.

sudo nutch-demo-crawl
PASS=$(sudo grep '^SOLR_ADMIN_PASSWORD=' /stage/scripts/nutch-credentials.log | cut -d= -f2-)
curl -sf -u "cloudimg:${PASS}" 'http://127.0.0.1:8983/solr/nutch/select?q=title:cloudimg&fl=id,title,url&wt=json' | head -c 400

nutch-demo-crawl prints NUTCH_E2E_OK once the crawled document is searchable, and the follow-up query returns it under response.docs with numFound: 1.

Step 8: Solr Admin UI — Dashboard

Browse to http://<vm-ip>:8983/solr/ and sign in as cloudimg with the per-VM password. The Dashboard shows the Solr instance, JVM memory (-Xmx512m on Standard_B2s), versions, and the active BasicAuthPlugin / RuleBasedAuthorizationPlugin.

Step 9: The nutch Core

Pick nutch in the Core Selector. The core overview lists the schema, document statistics and the query interface. This is the core Nutch indexes every crawled page into.

Step 10: Search Crawled Content in the Query UI

Open the Query tab on the nutch core, set q to title:cloudimg and click Execute Query. The crawled demo document is returned — id is the page URL, with title, url, content and host fields populated by Nutch's indexing filters.

Step 11: Crawl Your Own Website

To crawl a real site, drop its seed URL(s) into a seeds directory and run one or more crawl rounds. bin/crawl -i indexes each round straight into the Solr nutch core using the per-VM credentials already wired into conf/index-writers.xml:

sudo mkdir -p /var/lib/nutch/urls
echo 'https://<your-domain>/' | sudo tee /var/lib/nutch/urls/seed.txt
export NUTCH_JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 NUTCH_HEAPSIZE=1024
cd /opt/nutch
sudo -E bin/crawl -i -s /var/lib/nutch/urls /var/lib/nutch/crawl 2

Set http.agent.name (and ideally http.agent.email) in /opt/nutch/conf/nutch-site.xml to identify your crawler to the sites you fetch, and edit /opt/nutch/conf/regex-urlfilter.txt to restrict the crawl scope. Increase the round count (the trailing 2) to crawl deeper.

Step 12: How Nutch Indexes into Solr

Nutch's Solr indexer target lives in /opt/nutch/conf/index-writers.xml — it points at the local nutch core with BasicAuth enabled:

sudo grep -E 'name="url"|name="auth"|name="username"' /opt/nutch/conf/index-writers.xml

The url is http://localhost:8983/solr/nutch, auth is true, and username is cloudimg. The password is injected at first boot (never baked into the image). The enabled plugins — protocol-http, parse-html/parse-tika, index-basic/index-anchor, indexer-solr — are set in /opt/nutch/conf/nutch-site.xml.

Step 13: Tune Heap and Scaling

Solr heap is set in /etc/default/solr.in.sh (SOLR_HEAP="512m" on Standard_B2s); Nutch's batch JVM is capped by NUTCH_HEAPSIZE (MB). Both fit a crawl-and-index cycle inside 4 GB:

grep -E '^SOLR_HEAP|^SOLR_JAVA_HOME' /etc/default/solr.in.sh

For production crawling raise SOLR_HEAP to 4096m+ and NUTCH_HEAPSIZE to 4096+ on D4s/D8s instances, and restart Solr with sudo systemctl restart solr.service. For very large crawls, migrate Nutch to a Hadoop cluster and Solr to SolrCloud (add ZooKeeper) — the crawl commands are unchanged.

Step 14: Logs and Troubleshooting

sudo journalctl -u solr.service --no-pager -n 40
sudo journalctl -u nutch-firstboot.service --no-pager -n 20
sudo tail -40 /var/solr/logs/solr.log 2>/dev/null

Solr writes its main log to /var/solr/logs/solr.log. Each Nutch crawl round prints its Hadoop-local job output to the terminal; redirect it to a file for long crawls. The systemd journal captures the first-boot credential-rotation script.

Security

• BasicAuthPlugin enforces authentication on every endpoint under /solr/ — there is no open, unauthenticated Solr admin reachable off-host
• /var/solr/data/security.json stores double-SHA-256 + salt password digests; the plaintext per-VM copy in /stage/scripts/nutch-credentials.log is 0600 root:root
• Per-VM credentials are generated at first boot — no shared secret between deployments, and the upstream Solr default (no security) is never shipped
• Nutch authenticates to Solr with the same per-VM credential (injected into index-writers.xml at first boot, restored to a placeholder in the shipped image)
• Restrict NSG inbound on :8983 to the IP ranges that need the Solr Admin UI or REST API
• Terminate TLS at an upstream Azure Application Gateway / Front Door, or enable Solr's own SSL (jetty-ssl.xml + keystore) for production HTTPS

Support

cloudimg provides 24/7/365 expert technical support. Guaranteed response within 24 hours, one hour average for critical issues. Contact support@cloudimg.co.uk.