Firecrawl on Ubuntu 24.04 LTS

Overview

Firecrawl is an open source web scraping and crawling API. Point it at a URL and it fetches the page, renders it in a real headless browser so JavaScript heavy sites work, and returns clean markdown or structured data ready to feed to a language model or a data pipeline. It crawls whole sites, following links up to a depth and page limit you choose, and returns each page as markdown, so you can build search indexes, knowledge bases and retrieval augmented generation datasets from live web content. Scraping and crawling run entirely on the instance using the bundled browser service, with a Postgres backed job queue and Redis for rate limiting and caching, so large crawls run reliably in the background.

It suits developers and data teams who need web content as clean text without building and operating a browser farm: feeding retrieval augmented generation and AI agents with up to date pages, monitoring and extracting data from sites, and turning documentation or catalogues into structured datasets. The self hosted deployment does real scraping and crawling with no external cloud key, so your requests and the pages you fetch stay inside your own cloud account.

Common uses

• Turn any website into clean markdown to feed retrieval augmented generation, AI agents and language model pipelines
• Crawl entire sites on a schedule to build search indexes, knowledge bases and structured datasets from live web content
• Own your scraping infrastructure inside your own cloud account with a token gated API and per instance secrets