Scrapyd on AWS User Guide

Overview

This guide covers launching and using the Scrapyd 1.6.0 on Ubuntu 24.04 AMI from cloudimg on Amazon EC2. It is one version of the cloudimg Container Platforms and PaaS listing on AWS Marketplace; you pick it as the software version when you subscribe and launch.

Scrapyd is an open source, BSD 3 Clause licensed service for deploying and running Scrapy spiders. You package a Scrapy project as a Python egg and upload it, then start, queue, monitor and cancel crawls through a JSON HTTP API, so running a spider becomes one call instead of a session on a machine. It keeps several versions of a project side by side, runs a bounded number of spiders at once, and writes a separate log and a separate scraped items file for every job, all visible in a small read only web console.

A default Scrapyd is not safe on a network address. Its /addversion.json route accepts a Python egg and runs code from it, and upstream ships with authentication switched off, so an exposed Scrapyd is remote code execution. This image closes that before anything is reachable:

• Two credentials, both unique to your instance. nginx checks the credential you sign in with (bcrypt, in a file the Scrapyd account cannot read), then replaces it with Scrapyd's own, different credential. A project you deploy cannot recover the credential you use, and your credential is refused by Scrapyd itself.
• Closed until first boot. No password, placeholder or TLS key is baked into the image. Scrapyd's configuration ships with authentication fields empty and the services refuse to start without the credential files first boot creates, so Scrapyd never runs open, not even for a moment.
• HTTPS from the first boot. nginx serves the console and API on port 443 with a certificate generated for your instance. Port 80 redirects to HTTPS. Requests for a host name the instance does not serve are dropped.
• Loopback only, fenced by user. Scrapyd (6800) and the bundled demo catalogue (6810) listen on 127.0.0.1 only, and nftables lets only nginx and root open Scrapyd's port.
• An egress fence for your spiders. Spiders fetch whatever their code asks for, so the scrapyd account may reach public addresses only. The instance metadata service, link local addresses, loopback, private and reserved ranges, your VPC's own address blocks and IPv6 are refused, so a redirect or a host name that resolves to an internal address fails too. You can allow a private site in one file (Step 9).
• Request hygiene at nginx. Cross site requests are refused, so a browser holding your credential cannot be driven into deploying or scheduling by another site; forwarding headers a client sends are ignored; encoded or doubled slashes and dot segments are rejected; and requests are rate limited per IPv4 address and per IPv6 /64.
• Hash pinned, no compiler. Scrapyd 1.6.0, Scrapy 2.20.0 and scrapyd-client 2.0.3 are installed from 54 hash pinned wheels into a root owned virtualenv the service cannot modify. Ubuntu 24.04 LTS is fully patched at build time and unattended security upgrades are on.
• Works on arrival without crawling anybody else. A demo project crawls a small sample catalogue that the instance serves to itself on loopback, so the first crawl works the moment the instance boots and sends no traffic to any third party.

Scrapyd and Scrapy are independent open source projects by the Scrapy developers. SCRAPY is a registered trademark of Zyte Group Limited, used here only to identify the unmodified open source software. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by Zyte or the Scrapy project.

What is on the image: Scrapyd 1.6.0, Scrapy 2.20.0 and scrapyd-client 2.0.3 in /opt/scrapyd/venv (Python 3.12), the cloudimg_demo project (source in /usr/share/scrapyd/demo-project), nginx, nftables, the licences and build provenance in /usr/share/doc/scrapyd and /usr/share/scrapyd, Ubuntu 24.04 LTS, cloud-init and the AWS Systems Manager agent.

Default user: ubuntu · Architecture: x86_64 · Recommended instance type: m5.large

Prerequisites

• An AWS account subscribed to the listing, an EC2 key pair and a VPC subnet with a route to the internet.
• A security group that allows TCP 22 (SSH), TCP 443 (the console and API) and TCP 80 (redirects to HTTPS) from your own networks only.
• m5.large (2 vCPU, 8 GiB) is a comfortable start: this image runs one spider per vCPU at a time. Crawling is bound by network and memory more than CPU, so choose a larger type if you will run many spiders at once or crawl very large sites, and raise the concurrency to match (see Day to day operation).

Step 1: Launch the instance

In the AWS Marketplace listing choose Continue to Subscribe, then Continue to Configuration, pick Scrapyd 1.6.0 on Ubuntu 24.04 as the software version and your Region, then Launch through EC2. Choose m5.large, your key pair and a security group as above.

With the AWS CLI (replace the values in angle brackets):

aws ec2 run-instances --region <region> --image-id <ami-id-from-the-listing> \
  --instance-type m5.large --key-name <your-key-pair> \
  --security-group-ids <your-security-group> --subnet-id <your-subnet> \
  --metadata-options HttpTokens=required,HttpEndpoint=enabled \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=scrapyd}]'

First boot takes about a minute: it generates the instance's two credentials and its certificate, loads the egress fence, places the demo project and then starts Scrapyd and nginx.

Step 2: Connect to your instance

ssh -i <your-key.pem> ubuntu@<public-ip>

Step 3: Check that Scrapyd answers

Check the service itself, not just the unit state. The units, then the health endpoint through nginx on HTTPS, and an anonymous API call that must be refused:

systemctl is-active scrapyd-egress-guard scrapyd-firstboot scrapyd nginx
H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt)
sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" "$@"; }
C=$(sd -o /dev/null -w '%{http_code}' "https://$H/healthz")
A=$(sd -o /dev/null -w '%{http_code}' "https://$H/daemonstatus.json")
echo "GET /healthz -> HTTP $C"
echo "anonymous GET /daemonstatus.json -> HTTP $A"
[ "$C" = 200 ] && [ "$A" = 401 ] && echo SCRAPYD_OK

active
active
active
active
GET /healthz -> HTTP 200
anonymous GET /daemonstatus.json -> HTTP 401
SCRAPYD_OK

The certificate check passes because --cacert names your instance's own certificate: curl verifies it exactly as a browser would once you trust it.

Port 80 redirects to HTTPS. Scrapyd and the demo catalogue answer on loopback only, and only for the accounts that need them: any other user on the instance is refused, and nothing but 22, 80 and 443 listens on the network:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt)
R=$(curl -s -o /dev/null -w '%{http_code}' --resolve "$H:80:127.0.0.1" "http://$H/")
echo "http://$H/ -> HTTP $R (redirect to https)"
sudo ss -Htln | awk '{print $4}' | sort -u | tr '\n' ' '; echo
S=$(sudo -u nobody curl -s -o /dev/null -m 4 -w '%{http_code}' http://127.0.0.1:6800/daemonstatus.json || true)
D=$(sudo -u nobody curl -s -o /dev/null -m 4 -w '%{http_code}' http://127.0.0.1:6810/ || true)
echo "another user to Scrapyd :6800 -> HTTP $S, to the demo catalogue :6810 -> HTTP $D"
[ "$R" = 301 ] && [ "$S" = 000 ] && [ "$D" = 000 ] && echo FENCED_OK

http://ec2-<public-ip-dashed>.compute-1.amazonaws.com/ -> HTTP 301 (redirect to https)
0.0.0.0:22 0.0.0.0:443 0.0.0.0:80 127.0.0.1:6800 127.0.0.1:6810 127.0.0.53%lo:53 127.0.0.54:53 [::]:22 [::]:443 [::]:80 
another user to Scrapyd :6800 -> HTTP 000, to the demo catalogue :6810 -> HTTP 000
FENCED_OK

HTTP 000 means the connection was refused: nothing reaches Scrapyd except through nginx.

Step 4: Get your URL and credential

sudo cat /root/scrapyd-credentials.txt

The file is readable by root only. Your values differ:

# cloudimg Scrapyd - this instance's credentials (written at first boot; root 0600). Keep them private.
# Web console and JSON API (sign in with the user name and password below):
SCRAPYD_URL=https://ec2-<public-ip-dashed>.compute-1.amazonaws.com/
SCRAPYD_USERNAME=scrapyd
SCRAPYD_PASSWORD=<32 characters generated on your instance>
# The certificate is self-signed for this instance; compare this fingerprint with the one your browser shows:
SCRAPYD_TLS_SHA256_FINGERPRINT=<unique to your instance>
# Deploy a project from your workstation (pip install scrapyd-client):  see the user guide for scrapy.cfg.
# ONLY CRAWL SITES YOU OWN OR ARE PERMITTED TO CRAWL - /usr/share/scrapyd/AUTHORISED-USE

SCRAPYD_URL uses the instance's public DNS name from EC2 instance metadata, and the certificate also names the public and private IP addresses. To serve your own DNS name, see Step 10. Store the password in your password manager: it was generated on your instance and no other instance has it.

The commands in the rest of this guide read the password from that file into a curl configuration file only you can read, so it never appears on a command line or in your shell history:

( umask 077; sudo sed -n 's/^SCRAPYD_PASSWORD=\(.*\)$/user = "scrapyd:\1"/p' /root/scrapyd-credentials.txt > ~/.scrapyd-curlrc )
stat -c '%n %a' ~/.scrapyd-curlrc
sudo stat -c '%n  %a %U:%G' /root/scrapyd-credentials.txt /etc/nginx/scrapyd.htpasswd /etc/scrapyd/scrapyd.env
[ "$(stat -c %a ~/.scrapyd-curlrc)" = 600 ] && echo CURLRC_OK

/home/ubuntu/.scrapyd-curlrc 600
/root/scrapyd-credentials.txt  600 root:root
/etc/nginx/scrapyd.htpasswd  640 root:www-data
/etc/scrapyd/scrapyd.env  600 root:root
CURLRC_OK

Step 5: Open the web console

Open SCRAPYD_URL in your browser. The certificate is self signed for your instance, so the browser warns once; compare the fingerprint it shows with SCRAPYD_TLS_SHA256_FINGERPRINT before you accept it. Sign in with SCRAPYD_USERNAME and SCRAPYD_PASSWORD.

The console is deliberately read only, as its own home page says: it lists projects and jobs and links to every job's log and items. Deploying projects and scheduling crawls is done through the JSON API, which the next steps use.

Step 6: The access model

Every route except the health check is refused without the credential, the credential you use is refused by Scrapyd itself, and Scrapyd's own configuration carries no credential at all:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt)
sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" "$@"; }
OK=1
for P in / /jobs /listprojects.json /daemonstatus.json; do
  C=$(sd -o /dev/null -w '%{http_code}' "https://$H$P"); echo "anonymous GET  $P -> HTTP $C"; [ "$C" = 401 ] || OK=0
done
for P in /schedule.json /addversion.json; do
  C=$(sd -o /dev/null -w '%{http_code}' -d project=cloudimg_demo -d spider=catalogue "https://$H$P"); echo "anonymous POST $P -> HTTP $C"; [ "$C" = 401 ] || OK=0
done
G=$(sd -K ~/.scrapyd-curlrc -o /dev/null -w '%{http_code}' "https://$H/daemonstatus.json")
D=$(sudo curl -s -K ~/.scrapyd-curlrc -o /dev/null -w '%{http_code}' http://127.0.0.1:6800/daemonstatus.json)
N=$(sudo curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:6800/daemonstatus.json)
echo "your credential through nginx -> HTTP $G"
echo "your credential straight to Scrapyd :6800 -> HTTP $D"
echo "no credential straight to Scrapyd :6800 -> HTTP $N"
sudo grep -E '^(bind_address|http_port|username|password) ' /etc/scrapyd/scrapyd.conf
[ "$OK" = 1 ] && [ "$G" = 200 ] && [ "$D" = 401 ] && [ "$N" = 401 ] && echo ACCESS_MODEL_OK

anonymous GET  / -> HTTP 401
anonymous GET  /jobs -> HTTP 401
anonymous GET  /listprojects.json -> HTTP 401
anonymous GET  /daemonstatus.json -> HTTP 401
anonymous POST /schedule.json -> HTTP 401
anonymous POST /addversion.json -> HTTP 401
your credential through nginx -> HTTP 200
your credential straight to Scrapyd :6800 -> HTTP 401
no credential straight to Scrapyd :6800 -> HTTP 401
bind_address      = 127.0.0.1
http_port         = 6800
username          =
password          =
ACCESS_MODEL_OK

• Two layers. nginx accepts your credential and replaces it with Scrapyd's own, which lives in /etc/scrapyd/scrapyd.env (root only) and in nginx's /etc/nginx/scrapyd-upstream-auth.conf. Anything that can talk to Scrapyd can run code as the scrapyd account, so a deployed project can learn Scrapyd's credential; it still cannot read /etc/nginx/scrapyd.htpasswd, the TLS key or /root, so the credential you sign in with stays safe.
• No placeholder. username and password in /etc/scrapyd/scrapyd.conf are empty on purpose: upstream treats empty as "authentication off", so the real values exist only in the file first boot writes, and scrapyd.service refuses to start if that file is missing, empty or malformed. nginx refuses to start in the same way if its credential file is missing, unreadable, empty or malformed.

Step 7: Run your first crawl

The image ships a cloudimg_demo project whose catalogue spider crawls a static sample catalogue served on 127.0.0.1:6810 by this instance. Scheduling it sends no traffic to any third party:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt)
sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" -K ~/.scrapyd-curlrc "$@"; }
sd "https://$H/listprojects.json"; echo
sd "https://$H/listspiders.json?project=cloudimg_demo"; echo
JOB=$(sd "https://$H/schedule.json" -d project=cloudimg_demo -d spider=catalogue | python3 -c 'import json,sys; print(json.load(sys.stdin)["jobid"])')
echo "scheduled job: $JOB"
for i in $(seq 1 40); do
  ST=$(sd "https://$H/status.json?job=$JOB" | python3 -c 'import json,sys; print(json.load(sys.stdin)["currstate"])')
  [ "$ST" = finished ] && break
  sleep 3
done
echo "job state: $ST"
echo "$JOB" > ~/.scrapyd-last-job
[ "$ST" = finished ] && echo CRAWL_OK

{"projects": ["cloudimg_demo"], "status": "ok", "node_name": "ip-<private-ip-dashed>"}
{"spiders": ["catalogue"], "status": "ok", "node_name": "ip-<private-ip-dashed>"}
scheduled job: <job-id>
job state: finished
CRAWL_OK

schedule.json returns at once with a job id and the crawl runs in the background; status.json reports pending, running and then finished.

Step 8: Read the crawl log and the scraped items

Every job writes a Scrapy log and a JSON Lines file of whatever the spider yielded, both named after the job id and both served behind the same credential:

H=$(sudo sed -n 's#^SCRAPYD_URL=https://\([^/]*\)/.*#\1#p' /root/scrapyd-credentials.txt)
sd() { curl -s --cacert /etc/nginx/ssl/scrapyd.crt --resolve "$H:443:127.0.0.1" "$@"; }
JOB=$(cat ~/.scrapyd-last-job)
sd -K ~/.scrapyd-curlrc "https://$H/logs/cloudimg_demo/catalogue/$JOB.log" > /tmp/scrapyd-job.log
sd -K ~/.scrapyd-curlrc "https://$H/items/cloudimg_demo/catalogue/$JOB.jl" > /tmp/scrapyd-job.jl
grep -oE "'(item_scraped_count|response_received_count|finish_reason|robotstxt/request_count)': [^,}]+" /tmp/scrapyd-job.log
N=$(python3 -c 'import json,sys; print(sum(1 for l in open(sys.argv[1]) if json.loads(l).get("sku")))' /tmp/scrapyd-job.jl)
echo "items: $N records"
python3 -c 'import json,sys; [print(r["sku"], r["price"], r["currency"], r["title"]) for r in map(json.loads, list(open(sys.argv[1]))[:3])]' /tmp/scrapyd-job.jl
A=$(sd -o /dev/null -w '%{http_code}' "https://$H/items/cloudimg_demo/catalogue/$JOB.jl")
echo "the same items without a credential -> HTTP $A"
rm -f /tmp/scrapyd-job.log /tmp/scrapyd-job.jl
[ "$N" = 12 ] && [ "$A" = 401 ] && echo ITEMS_OK

'finish_reason': 'finished'
'item_scraped_count': 12
'response_received_count': 17
'robotstxt/request_count': 1
items: 12 records
CI-3004 84.00 GBP Windshield Kit
CI-3003 197.50 GBP Boom Pole Carbon
CI-3002 289.99 GBP Reference Monitor 5in
the same items without a credential -> HTTP 401
ITEMS_OK

item_scraped_count of 12 and finish_reason of finished are the honest answer to "did the crawl work". robotstxt/request_count of 1 shows the spider fetched robots.txt first, because the demo project sets ROBOTSTXT_OBEY = True. Logs and items are served as plain text that a browser will not run as a page.

Step 9: What your spiders can reach

Spiders run as the scrapyd account under an nftables egress fence. Public addresses are allowed; the instance metadata service, loopback, private and reserved ranges, your VPC and IPv6 are refused at connect, whatever URL or redirect the spider follows:

probe() { sudo -u scrapyd python3 -I -c 'import socket,sys
s = socket.socket(); s.settimeout(5)
try:
    s.connect((sys.argv[1], int(sys.argv[2]))); print("OPEN")
except OSError as e:
    print("REFUSED", type(e).__name__)' "$1" "$2"; }
M=$(probe 169.254.169.254 80); echo "spider -> instance metadata 169.254.169.254:80 : $M"
L=$(probe 127.0.0.1 6800);     echo "spider -> Scrapyd's own port 127.0.0.1:6800 : $L"
V=$(probe "$(hostname -I | awk '{print $1}')" 22); echo "spider -> this instance's private address :22 : $V"
P=$(probe 10.0.0.1 80);        echo "spider -> a private address 10.0.0.1:80 : $P"
W=$(probe example.com 443);    echo "spider -> a public site example.com:443 : $W"
case "$M$L$V$P" in *OPEN*) false ;; esac && [ "$W" = OPEN ] && echo EGRESS_FENCE_OK

spider -> instance metadata 169.254.169.254:80 : REFUSED ConnectionRefusedError
spider -> Scrapyd's own port 127.0.0.1:6800 : REFUSED ConnectionRefusedError
spider -> this instance's private address :22 : REFUSED ConnectionRefusedError
spider -> a private address 10.0.0.1:80 : REFUSED ConnectionRefusedError
spider -> a public site example.com:443 : OPEN
EGRESS_FENCE_OK

To let your spiders crawl a site inside your VPC or on your private network, list its address range, one IPv4 or IPv6 CIDR per line, in /etc/cloudimg/scrapyd-egress-allow and reload the fence. A range that touches the metadata service, link local or loopback addresses is rejected and the running fence is kept:

echo '10.0.5.25/32' | sudo tee -a /etc/cloudimg/scrapyd-egress-allow
sudo systemctl reload scrapyd-egress-guard

The instance's own addresses are reached over loopback, so services on the instance itself stay refused to spiders whatever the file lists.

Step 10: Deploy your own Scrapy project

The demo project's source is at /usr/share/scrapyd/demo-project; copy it as a starting point and edit the spider. Its settings.py is a deliberately polite template: it obeys robots.txt, identifies itself with a USER_AGENT, throttles itself and switches off Scrapy's telnet and remote control consoles. scrapyd-client is installed, so you can deploy from the instance. Add the URL and credential to the [deploy] section of scrapy.cfg:

[deploy]
url = https://<SCRAPYD_URL host name>/
project = my_project
username = scrapyd
password = <SCRAPYD_PASSWORD>

The certificate is your instance's own, so point Python at it when you deploy:

cp -r /usr/share/scrapyd/demo-project ~/my-project && cd ~/my-project
SSL_CERT_FILE=/etc/nginx/ssl/scrapyd.crt /opt/scrapyd/venv/bin/scrapyd-deploy default -p my_project

To deploy from your workstation, pip install scrapyd-client there, copy /etc/nginx/ssl/scrapyd.crt from the instance (it is public) and use the same SSL_CERT_FILE setting, or give the instance your own DNS name and certificate (below). Without scrapyd-client, addversion.json takes an egg directly:

curl --cacert scrapyd.crt -u scrapyd:<SCRAPYD_PASSWORD> https://<SCRAPYD_URL host name>/addversion.json \
  -F project=my_project -F version=1 -F egg=@dist/my_project-1.0-py3.12.egg

Each upload is a new version of the project. listversions.json lists them, schedule.json accepts _version to pin a crawl to one, and Scrapyd uses the newest by default, so rolling back a bad spider means scheduling the previous version. cancel.json stops a job, delversion.json removes one version and delproject.json removes a project; logs and items of crawls that already ran stay until jobs_to_keep rotates them out.

Your own DNS name and certificate. Point a DNS name at the instance, then let the image serve it; this mints a new certificate for the name, updates SCRAPYD_URL and reloads nginx:

sudo scrapyd-set-hostname <your-domain>

To use a certificate from your own CA, replace /etc/nginx/ssl/scrapyd.crt and /etc/nginx/ssl/scrapyd.key (keep the key 0600 root) and run sudo systemctl reload nginx.

Step 11: What is in the image, and under what licence

/opt/scrapyd/venv/bin/python -c 'import scrapyd, scrapy; print("scrapyd", scrapyd.__version__, "| scrapy", scrapy.__version__)'
/opt/scrapyd/venv/bin/pip list --format=freeze 2>/dev/null | grep -icE '==' | sed 's/^/python distributions: /'
ls /usr/share/doc/scrapyd/ | head -20
head -5 /usr/share/scrapyd/AUTHORISED-USE
/opt/scrapyd/venv/bin/python -c 'import scrapyd' && echo LICENCE_OK

scrapyd 1.6.0 | scrapy 2.20.0
python distributions: 54
LICENSE.scrapy
LICENSE.scrapyd
LICENSE.scrapyd_client
requirements.lock
wheels-sha256.txt
cloudimg Scrapyd image - authorised use
=======================================
This image runs web crawlers that YOU write and deploy. cloudimg supplies the runtime; the crawling is entirely under
your control and is your responsibility.
LICENCE_OK

There is no Java runtime, no compiler and no optional cloud SDK on the image. The only change to any installed package is that the published private keys Twisted ships as test fixtures for its own test suite are removed, so no private key material is baked into the image. /usr/share/scrapyd/THIRD-PARTY-NOTICES lists every bundled component, and /usr/share/doc/scrapyd/wheels-sha256.txt records the hash of every installed wheel.

Day to day operation

• Concurrency: max_proc_per_cpu = 1 in /etc/scrapyd/scrapyd.conf runs one spider per vCPU, two on m5.large. On a larger type raise it (upstream's default is 4) and run sudo systemctl restart scrapyd. Each spider is its own Python process, so watch memory when you tune it.
• Job history: finished_to_keep (200) caps the finished jobs the console lists and jobs_to_keep (20) caps the log and items files kept per spider.
• Where the data lives: eggs in /var/lib/scrapyd/eggs, logs in /var/lib/scrapyd/logs, items in /var/lib/scrapyd/items, queues and job history in /var/lib/scrapyd/dbs. The tree belongs to the scrapyd account and is not world readable. Back up eggs and items.
• Rotate your credential: sudo htpasswd -B /etc/nginx/scrapyd.htpasswd scrapyd, then update /root/scrapyd-credentials.txt for your records and recreate ~/.scrapyd-curlrc. nginx reads the file on every request; keep it a bcrypt entry, because nginx refuses to start on anything else.
• Updates: Scrapyd and Scrapy are pinned for reproducibility, and setuptools stays below 81 on purpose: Scrapyd's egg runner imports pkg_resources, which setuptools 81 removed. Ubuntu security updates install automatically; cloudimg publishes refreshed images for new releases.
• Crawl responsibly: nothing is crawled unless you schedule it. Crawl only sites you own or are permitted to crawl, respect robots.txt and rate limits, identify your crawler with a USER_AGENT that gives a contact point, and follow the AWS Acceptable Use Policy. Handling any personal data you collect lawfully is your responsibility. The full statement is in /usr/share/scrapyd/AUTHORISED-USE.

Troubleshooting

• The browser shows a certificate warning. Expected with the self signed certificate: compare the fingerprint with SCRAPYD_TLS_SHA256_FINGERPRINT, or use your own name and certificate (Step 10).
• The connection is closed with no page. You used a host name the instance does not serve. Use SCRAPYD_URL, or set your name with scrapyd-set-hostname.
• A deploy or schedule from your own web page is refused with 403. Cross site requests are blocked on purpose. Call the API from scripts or scrapyd-deploy, not from another site's page.
• A spider cannot reach an internal site. The egress fence refuses private ranges until you list them (Step 9). The job log shows Connection refused for the request.
• Your job log shows Remote control HTTP server failed to start. Scrapy 2.20 starts, by default, a small HTTP server inside each crawl that runs Python code sent to it. Spiders on this image have no writable profile directory, so it cannot start and the crawl carries on without it. Add REMOTE_CONTROL_ENABLED = False and TELNETCONSOLE_ENABLED = False to your project's settings.py, as the demo project does, to switch both consoles off cleanly.
• A job stays pending. All slots are busy: check max_proc_per_cpu and the running jobs on the console's Jobs page.
• After a stop and start the URL changed. The name is re-derived from EC2 at each boot unless you pin one with scrapyd-set-hostname; read the new URL from /root/scrapyd-credentials.txt.

Security recommendations

• Allow ports 22, 80 and 443 in the security group from your own networks only.
• Use your own DNS name and a certificate your clients trust (Step 10), so deploys from your workstation are verified.
• Treat anyone who can deploy as able to run code on the instance as the scrapyd account: share the credential only with people you trust with that.
• List only the private ranges your spiders need in the egress allow-list.
• Keep the operating system patched; unattended security upgrades are on.

Support and licensing

Scrapyd, Scrapy and scrapyd-client are open source under the BSD 3 Clause licence; there is no licence fee. The AMI price covers cloudimg's hardening, first boot setup and support.

• Email: support@cloudimg.co.uk
• Website: https://www.cloudimg.co.uk
• Support: 24/7, with a 24 hour response SLA