Developer Tools Azure

Scrapyd on Ubuntu 24.04 on Azure User Guide

Last updated: 2026-07-26 | Product: Scrapyd on Ubuntu 24.04 LTS on Azure

Overview

Scrapyd is an open source service for deploying and running Scrapy spiders. You package a Scrapy project as a Python egg and upload it, then start, queue, monitor and cancel crawls through a JSON HTTP API, so scheduling a spider becomes a single call instead of a session on a machine. It holds several versions of a project side by side, runs a bounded number of spiders at once, and writes a separate log and a separate scraped items file for every job, all reachable from a small web console that lists what is pending, what is running and what has finished.

The cloudimg image installs Scrapyd 1.6.0 with Scrapy 2.17.0 into a dedicated Python virtualenv, runs it as a hardened non root scrapyd systemd service bound to loopback, and fronts it on port 80 with nginx. Two separate credentials are generated on the first boot of every VM. A small demo project and a sample catalogue served on loopback ship with the image, so a real crawl works the moment your VM boots, with nothing for you to supply and no traffic sent to anybody else's website. Backed by 24/7 cloudimg support.

What is included:

Scrapyd 1.6.0 and Scrapy 2.17.0 pinned in a dedicated Python 3.12 virtualenv at /opt/scrapyd/venv, run as the non root scrapyd systemd service bound to loopback 127.0.0.1:6800
An nginx reverse proxy on :80 applying a per VM HTTP Basic credential to the whole API and console
A second, different per VM credential that only nginx and Scrapyd share, so a project you deploy cannot read the credential you sign in with
scrapyd-client 2.0.3 for scrapyd-deploy, so you can deploy a project from this VM as well as from your workstation
Durable job history in SQLite, so finished crawls survive a restart or a reboot
Per job crawl logs at /logs/ and per job scraped items as JSON Lines at /items/
A bundled cloudimg_demo project with a catalogue spider, and a static sample catalogue served on 127.0.0.1:6810, so the first crawl works immediately
The demo project source at /usr/share/scrapyd/demo-project to copy as a starting point
scrapyd.service + nginx.service as systemd units, enabled and active, both of which refuse to start if their credential is missing
An unauthenticated /healthz endpoint for Azure Load Balancer health probes
24/7 cloudimg support

Scrapyd, Scrapy and scrapyd-client are licensed under the BSD 3 Clause License (Copyright (c) Scrapy developers), a permissive licence with no fee and no key to enter. SCRAPY is a registered trademark of Zyte Group Limited. cloudimg is not affiliated with, endorsed by, or sponsored by Zyte or the Scrapy project.

Security model, and why it matters here

A default Scrapyd install is not safe to put on a public address, and the reason is sharper than it first looks. Scrapyd's /addversion.json endpoint accepts an arbitrary Python egg, and its runner then executes code from that egg. Upstream ships with its username and password settings empty, and Scrapyd treats either one being empty as "authentication off" - it logs Basic authentication disabled and serves every route to anybody. So an unauthenticated Scrapyd reachable from a network is not an open job queue; it is remote code execution as the service account. Upstream's own mitigation is to bind to 127.0.0.1 by default, which a deployable appliance cannot keep and still be useful.

This image closes that with two independent credentials, both unique to your VM and both generated on its first boot:

Layer	What it is	Where its secret lives
1. The gate	nginx on `:80`, HTTP Basic, bcrypt. This is the credential you use.	`/var/lib/scrapyd/.htpasswd`, `0640 root:www-data`
2. Scrapyd itself	Scrapyd's own native HTTP Basic, a different password	`/etc/scrapyd/scrapyd.env`, `0600 root:root`

nginx does not pass your credential through to Scrapyd. It verifies it and then replaces the Authorization header with Scrapyd's own, so your credential is never accepted by Scrapyd and Scrapyd's credential never leaves the VM.

That separation is doing real work rather than looking tidy. Anything that can talk to Scrapyd can run code as the scrapyd account, so a project you deploy can read whatever that account can read - including Scrapyd's own credential, however it is supplied. What survives that is the gate credential, because its bcrypt file is not readable by scrapyd at all. A careless or hostile spider therefore cannot recover the password you use to reach the appliance. It only holds because the two are different, which is why the image generates two and refuses to start if they match.

Three further things follow from the same reasoning:

No credential placeholder exists anywhere in the image. For this product a placeholder would be one of two disasters: a non empty one is a known password on every VM, and an empty one silently switches authentication off. So /etc/scrapyd/scrapyd.conf ships with username and password genuinely empty, and the real values only ever exist in a file created on your first boot.
The services refuse to start without a credential rather than starting open. scrapyd.service is gated on both the credential file and a bootstrap marker, loads the file in a way that makes a missing file a startup failure, and runs a preflight guard that rejects a missing, empty, short or wrongly owned credential. nginx.service carries the same shape, and its configuration includes the credential file so nginx cannot even load without it.
Scrapyd and the sample catalogue are bound to loopback. Port 80 is the only ingress. Nothing on 6800 or 6810 is reachable from the network.

One limitation, stated plainly: HTTP Basic Auth sends your credential in every request, and this image serves plain HTTP on port 80. Do not expose port 80 to the internet without TLS in front of it. Maintenance below covers adding it.

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet plus subnet in the target region. Standard_B2s (2 vCPU / 4 GiB RAM) is a comfortable starting point and is what the concurrency defaults in this image are sized for: at most two spiders run at once. Crawling is more network and memory bound than CPU bound, so size up if you intend to run many spiders concurrently or crawl very large sites, and raise max_proc_per_cpu to match (see Maintenance). NSG inbound: allow 22/tcp from your management network, 80/tcp for the API and console, and 443/tcp if you add TLS.

Step 1 - Deploy from the Azure Marketplace

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Scrapyd by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size; under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Then Review + create -> Create.

Step 2 - Deploy from the Azure CLI

az vm create \
  --resource-group <your-rg> \
  --name scrapyd \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_ed25519.pub \
  --vnet-name <your-vnet> --subnet <your-subnet> \
  --public-ip-sku Standard

az vm open-port --resource-group <your-rg> --name scrapyd --port 80 --priority 1010

Step 3 - Connect to your VM

ssh azureuser@<vm-public-ip>

Step 4 - Confirm the services are running

systemctl is-active scrapyd.service nginx.service scrapyd-firstboot.service
/opt/scrapyd/venv/bin/python -c 'import scrapyd, scrapy; print("scrapyd", scrapyd.__version__, "| scrapy", scrapy.__version__)'
sudo ss -tlnp | grep -E ':(80|6800|6810) ' | sed 's/users:.*((/ /; s/))$//'

All three units report active, and the version line confirms the pinned releases. Scrapyd listens on the loopback address 127.0.0.1:6800 and the bundled sample catalogue on 127.0.0.1:6810; only nginx is bound to 0.0.0.0:80, so neither is reachable from the network.

scrapyd-firstboot.service is a one shot unit. It runs once on the first boot of your VM, generates both credentials, and then does nothing on subsequent boots because it is gated on a sentinel file it creates. active here means "it ran and completed", which is the expected end state.

The Scrapyd, nginx and first boot units all reporting active, the pinned Scrapyd 1.6.0 and Scrapy 2.17.0 versions, the loopback listeners on 127.0.0.1 port 6800 for Scrapyd and port 6810 for the bundled sample catalogue with nginx on port 80, and the authenticated daemon status call returning ok with its job counters

Step 5 - Retrieve your per VM credential

sudo cat /root/scrapyd-credentials.txt
sudo stat -c '%n  %a %U:%G' /root/scrapyd-credentials.txt /var/lib/scrapyd/.htpasswd /etc/scrapyd/scrapyd.env

SCRAPYD_USERNAME and SCRAPYD_PASSWORD are the credential for your instance. Store them in your password manager. The file is 0600 root:root, the gate's bcrypt file is 0640 root:www-data, and Scrapyd's own credential is 0600 root:root - which is what keeps a deployed project from reading either of them.

Nothing here is a cloudimg default. These values were generated by openssl on your VM's first boot and no other cloudimg instance has them.

The per VM credentials file showing the instance URL and gate username with the password redacted, the file permissions on the credentials file, the bcrypt gate file and Scrapyd's own credential file, the bcrypt only hash in the gate file, and the deployed project, version and spider list returned by the authenticated API

Step 6 - Verify the security posture yourself

Do not take the previous section on trust. Every claim in it is a single command away:

for R in / /jobs /listprojects.json; do
  printf 'anonymous GET %-22s -> HTTP ' "$R"
  curl -s -o /dev/null -w '%{http_code}\n' -m 15 "http://127.0.0.1$R"
done
for R in /schedule.json /addversion.json; do
  printf 'anonymous POST %-21s -> HTTP ' "$R"
  curl -s -o /dev/null -X POST -d 'project=cloudimg_demo&spider=catalogue' \
    -w '%{http_code}\n' -m 15 "http://127.0.0.1$R"
done
grep -E '^(bind_address|http_port|username|password|items_dir|max_proc_per_cpu) ' /etc/scrapyd/scrapyd.conf

Every route answers 401, including the two that would amount to remote code execution on an unauthenticated instance. The configuration dump shows username and password genuinely empty, which is the proof that no credential is baked into this image.

Now prove the two layers are separate. Read your gate credential from the file rather than typing it, so it never lands in your shell history:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-)
printf 'gate credential, port 80             -> HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 15 -u "$GU:$GP" http://127.0.0.1/
printf 'gate credential, Scrapyd port 6800   -> HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 15 -u "$GU:$GP" http://127.0.0.1:6800/daemonstatus.json
printf 'no credential,   Scrapyd port 6800   -> HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 15 http://127.0.0.1:6800/daemonstatus.json

The first is 200. The second and third are both 401: your credential opens the front door and is refused by Scrapyd itself, and Scrapyd refuses an anonymous caller on its own port too. That is the two layer design working, and it is why a spider you deploy cannot use the credential it might find to reach anything except what it could already reach.

The shipped Scrapyd configuration with a loopback bind address and genuinely empty username and password, three anonymous GET requests and two anonymous POST requests including addversion all returning HTTP 401, and the gate credential returning HTTP 200 on port 80 while being refused with HTTP 401 by Scrapyd itself on port 6800 alongside an anonymous request that is also refused

Step 7 - Open the web console

Browse to http://<vm-public-ip>/ and sign in with SCRAPYD_USERNAME and SCRAPYD_PASSWORD from Step 5. The console lists the deployed projects and links to the Jobs, Items and Logs views.

Scrapyd's console is deliberately read only - it says so on its own home page. Deploying projects and scheduling crawls is done through the JSON API, which is what the next steps use.

The Scrapyd web console home page behind the per VM credential, showing links to the Jobs, Items and Logs views alongside the bundled cloudimg_demo project in the Scrapy projects list, and upstream's own note that the console is for monitoring only

Step 8 - Schedule your first crawl

The image ships a cloudimg_demo project whose catalogue spider crawls a static sample catalogue that this VM serves to itself on 127.0.0.1:6810. It is entirely self contained: scheduling it sends no traffic to any third party and depends on no external website.

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-)
curl -s -u "$GU:$GP" http://127.0.0.1/listprojects.json; echo
curl -s -u "$GU:$GP" 'http://127.0.0.1/listspiders.json?project=cloudimg_demo'; echo
JOB=$(curl -s -u "$GU:$GP" http://127.0.0.1/schedule.json \
        -d project=cloudimg_demo -d spider=catalogue \
      | /opt/scrapyd/venv/bin/python -c 'import json,sys; print(json.load(sys.stdin)["jobid"])')
echo "scheduled job: $JOB"

schedule.json returns immediately with a job id; the crawl runs in the background. Poll listjobs.json until it moves out of pending and running:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-)
for i in $(seq 1 40); do
  BUSY=$(curl -s -m 20 -u "$GU:$GP" 'http://127.0.0.1/listjobs.json?project=cloudimg_demo' \
    | /opt/scrapyd/venv/bin/python -c 'import json,sys; d=json.load(sys.stdin); print(len(d["pending"])+len(d["running"]))')
  [ "$BUSY" = "0" ] && break
  sleep 3
done
curl -s -u "$GU:$GP" 'http://127.0.0.1/listjobs.json?project=cloudimg_demo' \
  | /opt/scrapyd/venv/bin/python -m json.tool | head -30

Step 9 - Read the crawl log and the scraped items

Every job writes two files: a Scrapy log and a JSON Lines feed of whatever the spider yielded. Both are named after the job id, and both are served over the same authenticated API.

LOG=$(ls -t /var/lib/scrapyd/logs/cloudimg_demo/catalogue/*.log | head -1)
IT=$(ls -t /var/lib/scrapyd/items/cloudimg_demo/catalogue/*.jl | head -1)
printf 'log:   %s\n' "$(basename "$LOG")"
grep -oE "'(item_scraped_count|response_received_count|finish_reason|robotstxt/request_count)': [^,}]+" "$LOG"
printf 'items: %s  (%s records)\n' "$(basename "$IT")" "$(grep -c . "$IT")"
head -2 "$IT" | cut -c1-150

item_scraped_count is 12 and finish_reason is finished, which together are the honest answer to "did the crawl actually work". robotstxt/request_count being 1 shows the spider fetched the target's robots.txt before crawling it, because the demo project sets ROBOTSTXT_OBEY = True.

Scheduling a crawl over the authenticated API returning a job id and status ok, the job list showing zero pending, zero running and two finished crawls with their start and finish times, the crawl log reporting seventeen responses, a finished reason and an item scraped count of twelve, and the items feed holding twelve records with the first two shown as real product JSON

The same information is in the console. The Jobs view lists the finished crawl with its runtime and links straight to its log and its items:

The Scrapyd Jobs view showing empty Pending and Running sections above a Finished section listing two completed cloudimg_demo catalogue crawls, each with its job id, start time, two second runtime, finish time and links to its crawl log and its scraped items

Following the Items link shows the records the spider extracted, one JSON object per line, ready to pipe into whatever consumes them:

The scraped items feed opened in a browser, listing twelve JSON records extracted by the crawl, each with a SKU, title, category, price, currency, availability and the source page URL, spanning the Hardware, Tools and Media categories of the sample catalogue

Following the Log link shows Scrapy's own account of the run. The statistics block at the end is the part worth reading: request and response counts, the item count, the feed export destination, and the finish reason.

The end of the crawl log in a browser, showing the last scraped item, the feed export line naming the JSON Lines file it wrote, and Scrapy's full statistics block including a request count of seventeen, an item scraped count of twelve, a robots.txt request and a finish reason of finished

Step 10 - Fetch the items with the API instead of the browser

Anything the console shows is available as a plain HTTP fetch, which is how you would collect results from a pipeline:

GU=$(sudo grep -m1 '^SCRAPYD_USERNAME=' /root/scrapyd-credentials.txt | cut -d= -f2-)
GP=$(sudo grep -m1 '^SCRAPYD_PASSWORD=' /root/scrapyd-credentials.txt | cut -d= -f2-)
IT=$(ls -t /var/lib/scrapyd/items/cloudimg_demo/catalogue/*.jl | head -1)
JOB=$(basename "$IT" .jl)
curl -s -u "$GU:$GP" "http://127.0.0.1/items/cloudimg_demo/catalogue/$JOB.jl" \
  | /opt/scrapyd/venv/bin/python -c 'import json,sys; [print(r["sku"], r["price"], r["title"]) for r in map(json.loads, sys.stdin)]'
printf 'without a credential -> HTTP '
curl -s -o /dev/null -w '%{http_code}\n' -m 15 "http://127.0.0.1/items/cloudimg_demo/catalogue/$JOB.jl"

Your scraped data is behind the same credential as everything else: an anonymous request for it returns 401.

Step 11 - Deploy your own Scrapy project

The demo project's source ships at /usr/share/scrapyd/demo-project, so the fastest way to start is to copy it and edit the spider:

cp -r /usr/share/scrapyd/demo-project ~/my-project
cd ~/my-project

scrapyd-client is already installed, so you can deploy from this VM. Add your credential to the [deploy] section of scrapy.cfg (scrapyd-deploy supports username and password there), then deploy:

[deploy]
url = http://127.0.0.1/
project = my_project
username = <SCRAPYD_USERNAME>
password = <SCRAPYD_PASSWORD>

/opt/scrapyd/venv/bin/scrapyd-deploy default -p my_project

To deploy from your workstation instead, install scrapyd-client there (pip install scrapyd-client), point url at http://<vm-public-ip>/, and use the same credential. Deploying from anywhere other than the VM itself means your credential and your project egg cross the network, so put TLS in front of nginx first - see Maintenance.

If you prefer not to use scrapyd-client, addversion.json takes the egg directly:

curl http://<vm-public-ip>/addversion.json -u <SCRAPYD_USERNAME>:<SCRAPYD_PASSWORD> \
  -F project=my_project -F version=1 -F egg=@dist/my_project-1.0-py3.12.egg

Each upload is a new version of the project. listversions.json shows them all, schedule.json accepts an optional _version to pin a crawl to one, and Scrapyd uses the highest version by default - so rolling back a bad spider is a matter of scheduling against the previous version rather than redeploying.

Step 12 - Cancel a crawl, and remove a project

These are the destructive endpoints. cancel.json stops a running or pending job; delversion.json removes one version of a project and delproject.json removes it entirely, including every version.

curl http://127.0.0.1/cancel.json -u "$GU:$GP" -d project=my_project -d job=<job-id>
curl http://127.0.0.1/delversion.json -u "$GU:$GP" -d project=my_project -d version=1
curl http://127.0.0.1/delproject.json -u "$GU:$GP" -d project=my_project

Deleting a project removes its eggs. It does not remove the logs and items of crawls that already ran, which stay under /var/lib/scrapyd/logs and /var/lib/scrapyd/items until you remove them or jobs_to_keep rotates them out.

Maintenance

Rotate the gate credential: run sudo htpasswd -B /var/lib/scrapyd/.htpasswd scrapyd, then update /root/scrapyd-credentials.txt for your records. No restart is needed - nginx re reads the file per request. Do not edit /var/lib/scrapyd/.htpasswd by hand into something that is not a bcrypt hash: the preflight guard will refuse to start nginx on its next restart, which is deliberate.
Rotate Scrapyd's own credential: the password lives in /etc/scrapyd/scrapyd.env and its base64 counterpart in /etc/nginx/cloudimg-scrapyd-upstream-auth.conf. Both must change together, so change them as a pair and restart both services. There is normally no reason to touch this: it never leaves the VM.
Raise the concurrency: max_proc_per_cpu = 1 in /etc/scrapyd/scrapyd.conf limits this image to one spider per CPU, which is two on the recommended size. On a larger VM raise it (upstream's own default is 4) and sudo systemctl restart scrapyd. Each spider is a separate Python process, so watch memory rather than CPU when you tune it.
Keep more or fewer job records: finished_to_keep (200 here) caps the job history the console shows, and jobs_to_keep (20) caps how many log and item files are retained per spider. Both are in /etc/scrapyd/scrapyd.conf.
Add TLS for production: HTTP Basic Auth sends your credential in every request, so anything beyond a private network needs TLS. Put your own domain in front of nginx and terminate TLS with a certificate from your CA or Let's Encrypt, then close port 80 at the NSG. Until you do, keep the NSG rule for port 80 scoped to your own address range rather than the internet.
Where the data lives: eggs in /var/lib/scrapyd/eggs, crawl logs in /var/lib/scrapyd/logs, scraped items in /var/lib/scrapyd/items, and the spider queues plus job history in /var/lib/scrapyd/dbs. The whole tree is owned by the scrapyd account and is not world readable, because a crawl log and an items feed are a record of what you crawled and what you took. Back up eggs and items; the rest is reproducible.
Crawl responsibly: this appliance runs crawlers you write, and nothing is crawled unless you schedule it. Only crawl sites you own or are permitted to crawl, respect robots.txt and rate limits, and identify your crawler with a USER_AGENT that gives a contact point. The full statement ships in the image at /usr/share/scrapyd/AUTHORISED-USE, and the demo project's settings are a deliberately polite starting point.
Updates: Scrapyd is pinned to 1.6.0 and Scrapy to 2.17.0 in this image for reproducibility. Note that setuptools is pinned below 81 on purpose: Scrapyd's egg loader imports pkg_resources, which setuptools 81 removed, so upgrading it inside the virtualenv would break every deployed project. cloudimg publishes refreshed images; for security patches apply OS updates with sudo apt update && sudo apt upgrade.

Support

This image is maintained by cloudimg with 24/7 support. For help deploying or operating Scrapyd on Azure, contact cloudimg support through the Azure Marketplace listing.