Base
Hardened build
minimal ports, security patches applied at build time
Scrapyd on Ubuntu 24.04
Azure
Developer Tools
Scrapyd, the service you deploy your Scrapy spiders to, so crawls are scheduled, queued and logged in one place instead of run by hand.
Overview
Scrapyd is an open source daemon for deploying and running Scrapy spiders. You package a Scrapy project as a Python egg and upload it, then start, queue, monitor and cancel crawls through a JSON HTTP API, so scheduling a spider becomes a single call rather than a session on a machine. It keeps several versions of a project side by side, runs a bounded number of spiders at once, and writes a separate log and a separate scraped items file for every job, all reachable from a small web console that lists what is pending, what is running and what has finished. Because the API is plain HTTP, crawls can be triggered from cron, a CI pipeline or your own application, and the same instance can hold many unrelated projects.
Why the cloudimg image
Upstream turns authentication off whenever its username or password is left unset, and the endpoint that accepts a project upload runs code from it, so an unauthenticated instance is far more than an open job queue. cloudimg never ships it that way. Each instance generates two separate credentials of its own on first boot: one for the front door that you use, and a different one that only the reverse proxy and the daemon share, so a project you deploy cannot read the credential you sign in with. Nothing usable is baked into the image, the daemon and its sample crawl target are bound to loopback with a single hardened front door in front, and the services refuse to start at all if a credential is missing rather than starting open. A bundled sample project and a catalogue served on loopback mean the first crawl works straight away and sends no traffic to anyone else's site. Every deployment ships fully patched, with a paired deploy guide and 24/7 cloudimg support.
Common uses
- Schedule and queue Scrapy crawls from cron, CI or your own application
- Keep several project versions deployed and roll a spider back if a run goes wrong
- Give a team one place to read crawl logs and download scraped items
See it running
Real screenshots taken while testing this image against its deployment guide.
More in Developer Tools
Accent - Translation and Localization Management
Developer Tools
Accent is an open source, developer-oriented translation and localization management platform.
Altair GraphQL Client - Self-Hosted GraphQL IDE
Developer Tools
Altair GraphQL Client is an open-source, self-hosted GraphQL IDE for building, sending and inspecting GraphQL requests.
Aa
Apache Archiva Repository Manager
Developer Tools
This image runs Apache Archiva, the open source build-artifact repository manager - a central, governed store for the bi...
Apache Camel Karavan Visual Integration Designer
Developer Tools
This image runs Apache Camel Karavan, an open source visual integration designer and toolkit for Apache Camel.
Apache JMeter Load Testing
Developer Tools
Apache JMeter is the de-facto open-source tool for load and performance testing of web applications, APIs, databases and...
Apache ManifoldCF Content Crawler
Developer Tools
Launch a content crawler and search index in minutes. ManifoldCF, PostgreSQL and Apache Solr come preinstalled and wired...