SpiderFoot on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and operation of SpiderFoot on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. SpiderFoot is an open source platform for OSINT automation and attack surface reconnaissance. You give it a target — a domain name, an IP address or range, a hostname, an email address, a username or a person's name — and it runs a library of over two hundred modules that gather what public sources already know about it, then correlates the findings into a picture of the target's exposed surface.

What it collects depends on the modules you enable, and typically includes DNS and WHOIS records, subdomains and netblocks, certificate transparency entries, open ports and service banners, appearances in breach and paste data, mentions in social media and code repositories, and hits in malicious activity feeds. Every finding is stored as a typed event linked back to the source that produced it, so you can trace any conclusion to where it came from, browse and filter the results, view them as a graph, and export them as CSV, JSON or GEXF.

Everything runs on your own VM. There is no cloud service in the middle, and no third party is told what you are investigating unless you deliberately enable a module that queries one.

You are responsible for what you scan

Only scan systems you own, or systems you have explicit, documented authorisation to test.

Scanning third party systems without authorisation may be unlawful in your jurisdiction and in the target's, and may breach your agreement with Microsoft Azure. Some SpiderFoot modules connect directly to the target; others ask third party services about it, which discloses your interest in that target to those services.

This image ships with no target configured, no scan scheduled and no scheduler enabled. It does nothing until you tell it to. The same statement ships inside the image at /usr/share/spiderfoot/AUTHORISED-USE.

The cloudimg image ships the free and open source, MIT licensed SpiderFoot from a reviewed upstream commit, running as a plain Python application under a dedicated account with no login. No password and no API key is baked into the image. Two independent authentication layers are generated uniquely on each VM at first boot, and the public listener will not start until first boot has proved that an unauthenticated request is answered with 401. Backed by 24/7 cloudimg support.

SpiderFoot is developed by Steve Micallef and is licensed under the MIT licence. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by the SpiderFoot project. It ships the free and open source software unmodified. The name is used here solely to identify the software this image contains.

What is included:

• SpiderFoot 4.0.0 from upstream commit 0f815a20, pinned by commit id, never by a moving tag
• The full module library, over two hundred modules, loaded and working with zero API keys configured
• A dedicated spiderfoot service account with no login shell, and a private 0700 data directory
• An nginx access gate on port 80, installed as a host package so unattended security upgrades keep the internet facing component patched
• SpiderFoot's own authentication left enabled as a genuine second layer, with a per VM password
• Embedded SQLite storage — no database server of any kind on the instance, and no container runtime
• A shipped dependency licence inventory at /usr/share/spiderfoot/dependency-licences.txt covering all 62 Python distributions, plus third party notices and a recorded source manifest

Prerequisites

• An Azure subscription with permission to create VMs
• A target you own or are explicitly authorised to test. Nothing is preconfigured, and you should not launch this image until you know what you are entitled to scan
• A Standard_B2ms VM (2 vCPU, 8 GiB) or larger. See Step 9: sizing and the honest scan envelope before you commit to a size — a broad scan is genuinely compute hungry
• Inbound TCP 80 open only to the networks your administrators connect from. Treat access to this appliance the way you would treat access to your own reconnaissance results
• An SSH key pair for administrative access

Step 1: Deploy from the Azure Portal

1. In the Azure Portal, search the Marketplace for SpiderFoot | Support by cloudimg and select Create.
2. Choose your subscription, resource group and region.
3. Set the VM size to Standard_B2ms or larger.
4. Under Administrator account, select SSH public key and supply your own key.
5. Under Inbound port rules, allow HTTP (80) and SSH (22), then tighten both to your own address ranges on the networking blade before the VM is created.
6. Leave the OS disk at its default. Review and create.

Step 2: Deploy from the Azure CLI

# Create the VM from the cloudimg image.
az vm create \
  --resource-group my-resource-group \
  --name my-spiderfoot \
  --image cloudimg:spiderfoot-ubuntu-24-04:spiderfoot-ubuntu-24-04:latest \
  --size Standard_B2ms \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --public-ip-sku Standard

# Then open the two ports you need, scoped to YOUR OWN network rather than the
# internet. These commands run on your workstation, not on the appliance.
MY_CIDR="203.0.113.0/24"   # replace with your own range

az network nsg rule create \
  --resource-group my-resource-group \
  --nsg-name my-spiderfootNSG \
  --name allow-ssh-from-office --priority 1000 \
  --source-address-prefixes "${MY_CIDR}" \
  --destination-port-ranges 22 --protocol Tcp --access Allow

az network nsg rule create \
  --resource-group my-resource-group \
  --nsg-name my-spiderfootNSG \
  --name allow-http-from-office --priority 1010 \
  --source-address-prefixes "${MY_CIDR}" \
  --destination-port-ranges 80 --protocol Tcp --access Allow

Step 3: Connect to your VM

ssh azureuser@<vm-ip>

Step 4: Confirm the services are running

First boot generates this VM's two credentials and proves the access gate before anything is allowed onto port 80, so give it a minute or so after the VM first comes up.

# All three must report active. spiderfoot-firstboot is a oneshot that stays
# "active (exited)" once it has completed successfully.
systemctl is-active spiderfoot.service spiderfoot-firstboot.service nginx.service

# The gate's own health literal. This is the ONLY unauthenticated route on the
# appliance, and it never reaches SpiderFoot, so it discloses nothing.
curl -s http://127.0.0.1/nginx-health

# SpiderFoot must be on loopback only, and nginx must be the only listener on 80.
sudo ss -lntp | grep -E ':(80|5001)\b'

You should see SpiderFoot on 127.0.0.1:5001 and nginx on 0.0.0.0:80. SpiderFoot is never published directly — there is no route to it that skips the gate.

Step 5: Read this VM's credentials

Both passwords were generated on this machine, at this boot, and exist nowhere else. Nothing in this file was baked into the image.

sudo cat /root/spiderfoot-credentials.txt

SPIDERFOOT_URL=http://<vm-ip>
SPIDERFOOT_UNLOCK_URL=http://<vm-ip>/cloudimg-unlock

# 1. The cloudimg gate (the browser's FIRST prompt)
GATE_USERNAME=cloudimg
GATE_PASSWORD=<GATE_PASSWORD>

# 2. SpiderFoot's own sign-in (the SECOND prompt)
SPIDERFOOT_USERNAME=sfadmin
SPIDERFOOT_PASSWORD=<SPIDERFOOT_PASSWORD>

Step 6: Open the interface

Open http://<vm-ip>/cloudimg-unlock first, not http://<vm-ip>/. There is a reason, explained in Step 7, and following this order makes the sign in flow work first time in every browser.

1. Your browser prompts for the cloudimg gate credentials. Enter GATE_USERNAME and GATE_PASSWORD. You will see a short confirmation page.
2. Follow the Continue to SpiderFoot link. Your browser now prompts a second time, and this prompt will show the realm as 127.0.0.1 — that is SpiderFoot identifying itself by the loopback address it is bound to, and it is correct. Enter SPIDERFOOT_USERNAME and SPIDERFOOT_PASSWORD.
3. The SpiderFoot interface loads, with no scans, no targets and no API keys.

Step 7: Understand the access model

This appliance has two independent authentication layers, and it is worth understanding why, because it explains the two prompts and the unlock URL.

Why SpiderFoot must never run unauthenticated. SpiderFoot only enables its own authentication if a password file exists. Without one it does not merely leave the UI open — measured on the stock upstream project, an anonymous POST /startscan returns 303 and a scan actually starts. Anyone who could reach the port could make your VM scan anything, from your IP address, and the requests that appear to be rejected are in fact missing argument errors rather than authentication denials. This appliance therefore never runs SpiderFoot without its own password file, and the service is held on a systemd condition that refuses to start it if that file is missing or malformed.

Layer 1 — the cloudimg gate. nginx on port 80 is the sole ingress and denies by default at server scope, so every route inherits it: the interface, all forty one of SpiderFoot's exposed endpoints, and its static assets. Exactly one route is unauthenticated, /nginx-health, and it is a static literal that never reaches SpiderFoot.

Layer 2 — SpiderFoot's own sign in. Left enabled, with a per VM password. Passing the gate alone gets you SpiderFoot's own 401, and no further.

Why there is an unlock URL. SpiderFoot's own authentication is HTTP Digest, and the gate is HTTP Basic. Both schemes travel in the same Authorization request header and a browser may only send one at a time, so a gate that demanded Basic on every single request would deadlock: the browser sends Basic, SpiderFoot demands Digest; the browser sends Digest and drops the Basic header, the gate demands Basic; and so on forever. The gate solves this by making its Basic requirement conditional on a per VM cookie issued when you authenticate. Visiting /cloudimg-unlock obtains that cookie in one clean step, after which the single Authorization header is free to carry SpiderFoot's Digest credentials and the interface works normally.

None of this weakens the gate. You can prove that from the appliance itself:

# Every route is closed to an anonymous caller with exactly 401 — including the
# endpoint that starts scans, which stock upstream answers with 303 and a
# running scan.
for p in / /modules /scanlist /startscan /optsraw /static/css/spiderfoot.css; do
  printf '%-32s %s\n' "$p" \
    "$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1$p")"
done

# The gate credential ALONE reaches SpiderFoot's own challenge and no further.
# Expect 401, with a Digest challenge — that is layer 2 doing its job.
GU=$(sudo grep -m1 '^GATE_USERNAME=' /root/spiderfoot-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^GATE_PASSWORD=' /root/spiderfoot-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -D - -u "$GU:$GP" http://127.0.0.1/ | head -1
curl -s -D - -o /dev/null -u "$GU:$GP" http://127.0.0.1/ | grep -i '^www-authenticate'

# And a VALID SpiderFoot credential with no gate cookie is still refused, so the
# gate cannot be bypassed just by switching authentication scheme. Expect 401.
SU=$(sudo grep -m1 '^SPIDERFOOT_USERNAME=' /root/spiderfoot-credentials.txt | cut -d= -f2-)
SP=$(sudo grep -m1 '^SPIDERFOOT_PASSWORD=' /root/spiderfoot-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code}\n' --digest -u "$SU:$SP" http://127.0.0.1/

Step 8: Run your first scan

Re-read the authorisation notice at the top of this guide before you start. A good first target is a domain you own.

1. Select New Scan.
2. Give the scan a name, and enter your target in Scan Target. SpiderFoot detects the target type automatically.
3. Choose how to scan. By Use Case is the simplest starting point:
4. Passive — never contacts the target directly. The right choice when you want to stay entirely off the target's logs, and the right choice for a first run.
5. Footprint — maps the target's externally visible network and web presence.
6. Investigate — Footprint plus reputation and malicious activity checks.
7. All — every module that applies. Thorough, and by far the heaviest; see Step 9.
8. Select Run Scan Now. The scan page updates as results arrive.

Results are grouped by type. Select any type to see the individual findings and, for each one, the module and source that produced it. Graph draws the relationships between findings, and the export buttons produce CSV, JSON or GEXF for tools such as Gephi.

A completed scan looks like this — findings grouped by type, with the number of unique results and the module that produced each one:

Start with Passive on a target you own. It gives you a feel for the output and the runtime without touching the target at all.

Step 9: Sizing, and the honest scan envelope

Standard_B2ms (2 vCPU, 8 GiB) is the recommended size, and there is a concrete measurement behind that recommendation rather than a guess.

A single All use case scan on the build machine saturated both vCPUs hard enough to make other work on the VM sluggish. SpiderFoot runs many modules concurrently and most of them are network bound, but parsing, correlating and writing results is real CPU work, and a broad scan does a great deal of it.

So the honest envelope for Standard_B2ms is:

• one scan at a time, with a moderate module selection (Passive, Footprint, or a hand picked module list)
• an All scan will complete, but expect the appliance to be busy for the duration and do not plan to run anything else alongside it
• do not run several scans concurrently on this size

If you need concurrent scans, or routinely run All against large targets, move to Standard_B4ms (4 vCPU, 16 GiB) or larger. Scan storage is modest — results are text in a SQLite database — so CPU, not disk, is what you are buying.

# Watch what a scan is actually costing you while it runs.
top -b -n 1 | head -15

# The database grows with results; check it between scans.
sudo du -sh /var/lib/spiderfoot/

Step 10: Adding your own API keys

A number of modules need an API key for a third party service, and every key field in this image is empty. That is deliberate: cloudimg never bakes a credential into an image, and a shared key would be both a leaked secret and a rate limit you would be sharing with strangers.

SpiderFoot works well with no keys at all — the module library loads in full, and the passive and footprinting use cases are productive without any. To add your own:

1. Open Settings.
2. Find the module in the list on the left and enter your key in its API Key field.
3. Select Save Changes.

Keys are stored in this VM's own SQLite database at /var/lib/spiderfoot/, which is 0700 and owned by the spiderfoot account, so they are not readable by other local users.

Be aware of what a keyed module discloses. Querying a third party service about a target tells that service you are interested in that target. Choose your modules accordingly.

Step 11: What is in the image, and under what licence

Everything below ships in the image so you can audit it without contacting us.

# The exact upstream commit, tree, and a content manifest of the shipped source.
cat /usr/share/spiderfoot/source-provenance.txt

# Third party notices, including the two copyleft dependencies and the bundled
# wordlist's advertising acknowledgment obligation.
less /usr/share/spiderfoot/THIRD-PARTY-NOTICES

# Every Python distribution in the image and the licence signal it published.
column -t -s $'\t' /usr/share/spiderfoot/dependency-licences.txt | less

# The exact resolved version of every dependency.
cat /usr/share/spiderfoot/python-requirements.lock

# Re-run the licence gate yourself. It exits non-zero on any AGPL, SSPL, BUSL,
# Elastic v2, Commons Clause, non-commercial, PolyForm, Prosperity or RSAL
# licence, and fails closed on anything it cannot resolve.
sudo /opt/spiderfoot/venv/bin/python \
  /usr/share/spiderfoot/spiderfoot-licence-gate.py

The application is MIT licensed, Copyright 2022 Steve Micallef. MIT permits use, modification, distribution and sale, and the full text ships at /opt/spiderfoot/app/LICENSE.

The bundled ispell wordlist at /opt/spiderfoot/app/spiderfoot/dicts/ispell/ is four clause BSD, Copyright 1993 Geoff Kuenning. Redistribution is permitted, but clause 4 requires an advertising acknowledgment, which is discharged in the shipped notices and reproduced here:

This product includes software developed by Geoff Kuenning and other unpaid contributors.

The Python dependencies were examined by an executable licence gate at build time: 62 distributions, zero restrictive licences, zero unresolved. Two carry file level copyleft — certifi and publicsuffixlist, both MPL-2.0, both unmodified. There is no AGPL code, and in particular no PyMuPDF: the PDF text extraction path in this image is PyPDF2, which is BSD-3-Clause.

One dependency was deliberately removed. Upstream's requirements.txt lists pygexf, which is GPL-3.0. Nothing in SpiderFoot imports it — the GEXF export you use is networkx's BSD-3-Clause writer — and pygexf is not even importable on Python 3. Leaving it out removes a copyleft obligation at no functional cost, and the build proves that by generating a real GEXF graph without it. Your Graph view and GEXF export are unaffected.

Day to day operation

# Service control.
sudo systemctl restart spiderfoot.service
sudo systemctl status  spiderfoot.service
sudo journalctl -u spiderfoot.service -n 100 --no-pager

# SpiderFoot's own logs.
sudo tail -n 50 /var/log/spiderfoot/spiderfoot.error.log

# The gate's logs — useful for spotting unauthenticated probing of port 80.
sudo tail -n 50 /var/log/nginx/spiderfoot-access.log

# Back up the results database. Stop the service first so SQLite is quiescent.
sudo systemctl stop spiderfoot.service
sudo cp /var/lib/spiderfoot/spiderfoot.db /root/spiderfoot-backup-$(date +%F).db
sudo systemctl start spiderfoot.service

Rotating either password. Both layers are independent, so rotate them independently.

# Layer 1, the cloudimg gate. Dry-run the command shape against a scratch file
# first (note -c, which CREATES the file — without it htpasswd refuses), so a
# typo cannot lock you out of your own appliance.
sudo htpasswd -cbB /tmp/scratch.htpasswd cloudimg '<new-password>'
sudo htpasswd -bB /etc/nginx/spiderfoot.htpasswd cloudimg '<new-password>'
sudo rm -f /tmp/scratch.htpasswd
sudo systemctl reload nginx

# Layer 2, SpiderFoot's own sign in. Its password file is username:password, and
# the password is necessarily stored in clear because HTTP Digest requires the
# server to know it in order to compute HA1. Keep the file 0600 and owned by the
# service account, and restart so SpiderFoot re-reads it.
printf 'sfadmin:%s\n' '<new-password>' | \
  sudo tee /var/lib/spiderfoot/passwd >/dev/null
sudo chown spiderfoot:spiderfoot /var/lib/spiderfoot/passwd
sudo chmod 0600 /var/lib/spiderfoot/passwd
sudo systemctl restart spiderfoot.service

Both rotations are independent of /root/spiderfoot-credentials.txt, which records what first boot generated and is not rewritten by a later rotation. Update your own password manager, not that file.

Security hardening

• Keep port 80 restricted to the networks your administrators connect from. This appliance is a reconnaissance tool and its results are sensitive.
• Put TLS in front of it if it will be reached across an untrusted network. Both authentication layers send credentials that plain HTTP does not protect: Basic sends them base64 encoded, and Digest, while it does not send the password itself, still leaves the session in the clear. Terminate TLS on an Azure Application Gateway, or install a certificate on the appliance's own nginx.
• Do not remove SpiderFoot's password file. Without it, SpiderFoot disables authentication entirely. The service will refuse to start rather than come up unauthenticated, which is the intended behaviour — restore the file rather than working around the refusal.
• Unattended security upgrades are enabled and nginx is a host package, so the internet facing component is patched by the OS.
• Review what your enabled modules disclose before scanning anything sensitive.

Troubleshooting

The browser asks for a password over and over. You went to / before obtaining the gate cookie, and the browser is caught between the two authentication schemes. Close the tab, open http://<vm-ip>/cloudimg-unlock, and sign in there first. Step 7 explains why.

The interface returns 502. SpiderFoot is not answering yet. It loads its whole module library at start up, which takes a few seconds:

systemctl is-active spiderfoot.service
sudo journalctl -u spiderfoot.service -n 50 --no-pager

spiderfoot.service will not start. Its start conditions and its preflight guard both refuse to run it in a state where it would be unauthenticated. Ask the guard directly — it prints the specific reason:

systemctl status spiderfoot.service --no-pager | head -20
sudo /usr/local/sbin/spiderfoot-preflight.sh --app; echo "guard exit: $?"

nginx will not start. The same pattern applies to the gate.

sudo /usr/local/sbin/spiderfoot-preflight.sh --nginx; echo "guard exit: $?"
sudo nginx -t

A scan produces very little. Check the use case you chose — Passive is deliberately narrow. Then check the scan's own error log in the interface, and consider whether the modules you want need an API key you have not supplied.

The appliance feels slow during a scan. That is expected on 2 vCPU with a broad module selection. See Step 9.

Support

cloudimg provides 24/7 support for this image. Raise a request at cloudimg.co.uk and include the output of systemctl status spiderfoot.service nginx.service and the relevant journal lines. Please do not include your scan results or your API keys in a support request.