Firecrawl on Ubuntu 24.04 on Azure User Guide

Overview

This guide covers the deployment and configuration of Firecrawl on Ubuntu 24.04 on Azure using cloudimg Azure Marketplace images. Firecrawl is an open source web scraping and crawling API: point it at a URL and it fetches the page, renders it in a real headless browser so JavaScript heavy sites work, and returns clean markdown or structured data ready to feed to a language model or a data pipeline. It crawls whole sites, following links up to a depth and page limit you choose, so you can build search indexes, knowledge bases and retrieval augmented generation datasets from live web content.

The cloudimg image ships the free and open source, AGPL-3.0 licensed Firecrawl, run the officially supported self hosted way as the upstream containers pinned by image digest: the Node/TypeScript API and worker harness, a bundled headless Chromium renderer, a Postgres job queue, a RabbitMQ broker and a Redis cache. All images are pinned by digest and captured into the VM, so your instance starts in seconds and scrapes entirely on the box with no external cloud key. Because self hosted Firecrawl has no built in authentication and is open by default, the image is secure by default: a unique API bearer token, and unique database, cache, broker and admin secrets, are generated for each VM on first boot, before the port is reachable, and the token is enforced at the nginx front door so every request except the health check must carry it. Backed by 24/7 cloudimg support.

Firecrawl is a trademark of its respective owner. This image is produced by cloudimg and is not affiliated with, endorsed by, or sponsored by Firecrawl. It ships the free and open source AGPL-3.0 licensed self hosted software, unmodified.

What is included:

• Firecrawl v2.11.118 (the AGPL-3.0 licensed self hosted API and worker), pinned by image digest
• A bundled headless Chromium renderer (playwright service), pinned by digest, that renders JavaScript pages on the instance so scraping needs no external cloud key
• A bundled Postgres job queue, RabbitMQ broker and Redis cache, pinned by digest, reachable only inside a private Docker network (never published to a host port)
• Docker Engine with the Firecrawl API published to the loopback interface only, fronted by nginx on port 80
• firecrawl.service, firecrawl-firstboot.service, firecrawl-postboot.service and nginx.service as systemd units, enabled and active on boot
• A unique API bearer token, and unique Postgres, Redis, RabbitMQ and admin secrets, generated per VM on first boot, never baked into the image, with the token enforced at the nginx front door
• A clean, empty queue database on first boot: no default token, no shipped secret, no prior data
• Ubuntu 24.04 LTS base with latest security patches applied at build time
• Azure Linux Agent for seamless cloud integration and SSH key injection
• 24/7 cloudimg support with guaranteed 24 hour response SLA

Prerequisites

• Active Azure subscription, SSH public key, VNet + subnet in target region
• Subscription to the Firecrawl listing on Azure Marketplace

Recommended virtual machine size: the stack runs a Node API and worker, a headless Chromium renderer, Postgres, RabbitMQ and Redis on one instance, so Standard_B4ms (4 vCPU, 16 GB RAM) is the recommended starting point. For heavy or highly concurrent crawling use a larger size (for example Standard_D4s_v5 or Standard_D8s_v5). NSG inbound: allow 22/tcp from your management network and 80/tcp for the API from the networks that call it.

Step 1: Deploy from the Azure Portal

Sign in to the Azure Portal, choose Create a resource, search the Marketplace for Firecrawl by cloudimg, and select Create. On Basics pick your subscription, resource group, region and size (Standard_B4ms or larger); under Administrator account choose SSH public key and paste your key; under Inbound port rules allow SSH (22) and HTTP (80). Then Review + create and Create.

Step 2: Deploy from the Azure CLI

RG="firecrawl-prod"; LOCATION="eastus"; VM_NAME="firecrawl-01"
GALLERY_IMAGE_ID="/subscriptions/<sub-id>/resourceGroups/azure-cloudimg/providers/Microsoft.Compute/galleries/cloudimgGallery/images/firecrawl-ubuntu-24-04/versions/<version>"
SSH_KEY="$(cat ~/.ssh/id_rsa.pub)"
az group create --name "$RG" --location "$LOCATION"
az network vnet create -g "$RG" --name firecrawl-vnet --address-prefix 10.100.0.0/16 --subnet-name firecrawl-subnet --subnet-prefix 10.100.1.0/24
az network nsg create -g "$RG" --name firecrawl-nsg
az network nsg rule create -g "$RG" --nsg-name firecrawl-nsg --name allow-ssh --priority 100 \
  --source-address-prefixes "<your-mgmt-cidr>" --destination-port-ranges 22 --access Allow --protocol Tcp
az network nsg rule create -g "$RG" --nsg-name firecrawl-nsg --name allow-http --priority 110 \
  --destination-port-ranges 80 --access Allow --protocol Tcp
az vm create -g "$RG" --name "$VM_NAME" --image "$GALLERY_IMAGE_ID" \
  --size Standard_B4ms --storage-sku StandardSSD_LRS \
  --admin-username azureuser --ssh-key-values "$SSH_KEY" \
  --vnet-name firecrawl-vnet --subnet firecrawl-subnet --nsg firecrawl-nsg --public-ip-sku Standard

Step 3: Connect to your VM

ssh azureuser@<vm-ip>

Step 4: Confirm the services are running

The stack runs as five containers (the Firecrawl API and worker, the headless Chromium renderer, Postgres, RabbitMQ and Redis) under one firecrawl.service, fronted by nginx. Confirm the services are active, see the containers, and check which ports are bound. nginx listens on :80, the Firecrawl API is published to 127.0.0.1:3002 (loopback only), and Postgres, RabbitMQ and Redis have no host port at all:

sudo systemctl is-active docker firecrawl nginx
sudo docker compose --env-file /etc/firecrawl/images.env --env-file /etc/firecrawl/firecrawl.env -f /etc/firecrawl/compose.yaml ps --format 'table {{.Service}}\t{{.Status}}'
sudo ss -tlnp | grep -E ':80 |:3002 '

You will see all three services report active, the five containers running (the API, Postgres, RabbitMQ and Redis reporting healthy), and nginx bound on :80 with the Firecrawl API bound only on 127.0.0.1:3002. Postgres (5432), RabbitMQ (5672) and Redis (6379) never appear on a host port because they are reachable only inside the private Docker network.

Step 5: Read the per instance credentials

A unique API bearer token, and unique Postgres, Redis, RabbitMQ and admin secrets, were generated for this VM on the first boot, before the port was reachable, and written to a root only file. Read them:

sudo cat /root/firecrawl-credentials.txt

The file (mode 0600 root:root) holds the bootstrap API_TOKEN you authenticate the API with, FIRECRAWL_URL (the address to reach this VM on), and the POSTGRES_PASSWORD, REDIS_PASSWORD, RABBITMQ_PASSWORD and BULL_AUTH_KEY. None of these ship in the image; every value is unique to this VM, so no two instances share a credential and there is no default token to change.

Step 6: Verify the API security model

Self hosted Firecrawl has no built in authentication, so cloudimg fronts it with nginx and enforces the per instance bearer token. The health endpoint is public, but every other endpoint requires the token. Confirm that an unauthenticated or wrong token request is rejected with 401, while the per instance token is accepted:

FC_TOKEN=$(sudo grep '^API_TOKEN=' /root/firecrawl-credentials.txt | cut -d= -f2-)
curl -s -o /dev/null -w 'health   (no auth):      %{http_code}\n' http://localhost/health
curl -s -o /dev/null -w 'scrape   (no token):     %{http_code}\n' -X POST http://localhost/v1/scrape -H 'Content-Type: application/json' -d '{"url":"https://example.com"}'
curl -s -o /dev/null -w 'scrape   (wrong token):  %{http_code}\n' -X POST http://localhost/v1/scrape -H 'Authorization: Bearer wrong-token' -H 'Content-Type: application/json' -d '{"url":"https://example.com"}'
curl -s -o /dev/null -w 'scrape   (with token):   %{http_code}\n' -X POST http://localhost/v1/scrape -H "Authorization: Bearer $FC_TOKEN" -H 'Content-Type: application/json' -d '{"url":"https://example.com","formats":["markdown"]}'

The health check returns 200 with no authentication, the unauthenticated and wrong token scrape requests are rejected with 401 at the nginx front door, and the same call with the per instance token returns 200. This is the whole security model: the health probe is open for load balancers, and everything else is gated by the bearer token.

Step 7: Scrape your first page

Scraping a page returns clean markdown that is ready to feed to a language model. Send a POST /v1/scrape with the per instance token and print the returned markdown:

FC_TOKEN=$(sudo grep '^API_TOKEN=' /root/firecrawl-credentials.txt | cut -d= -f2-)
curl -s -m 120 -X POST http://localhost/v1/scrape \
  -H "Authorization: Bearer $FC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","formats":["markdown"]}' \
  | python3 -c 'import sys,json;d=json.load(sys.stdin);print("success:",d["success"]);print();print(d["data"]["markdown"])'

The response reports success: True and the page rendered to markdown, for example a heading Example Domain followed by the page text. The headless Chromium renderer fetched and rendered the page entirely on the instance; nothing was sent to an external Firecrawl cloud service and no cloud API key is required. Add "html" or "rawHtml" to formats, or "screenshot", to get those representations too.

Step 8: Crawl a whole site

Crawling follows links from a starting URL and scrapes every page it finds, up to the limit you set. It returns a job id immediately and runs in the background; poll the job to watch it complete and collect the markdown for each page:

FC_TOKEN=$(sudo grep '^API_TOKEN=' /root/firecrawl-credentials.txt | cut -d= -f2-)
CRAWL_ID=$(curl -s -X POST http://localhost/v1/crawl \
  -H "Authorization: Bearer $FC_TOKEN" -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","limit":5}' \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
echo "crawl job: $CRAWL_ID"
sleep 12
curl -s http://localhost/v1/crawl/$CRAWL_ID -H "Authorization: Bearer $FC_TOKEN" \
  | python3 -c 'import sys,json;d=json.load(sys.stdin);print("status:",d.get("status"),"| completed:",d.get("completed"),"/",d.get("total"))'

The first call returns a crawl job id and the job runs in the background; polling GET /v1/crawl/<id> reports the status moving to completed with the count of pages scraped. For a larger site raise limit and add options such as includePaths, excludePaths or maxDepth to scope the crawl.

Step 9: Call the API from your own machine

From anywhere that can reach the VM on port 80, use the bootstrap token and the VM public IP. Retrieve the token once (over SSH) and then call the API with your VM address:

curl -X POST http://<vm-ip>/v1/scrape \
  -H "Authorization: Bearer <API_TOKEN>" -H 'Content-Type: application/json' \
  -d '{"url":"https://firecrawl.dev","formats":["markdown"]}'

For production, use the official Firecrawl SDKs (available for Python and Node/TypeScript). Point the SDK at your instance by setting the API URL to http://<vm-ip>/ (or your own domain, see Step 11) and pass the bootstrap token as the API key, then call scrape, crawl and the other endpoints from your application code exactly as the curl calls above do. When connecting an SDK to a self hosted instance the API key is simply the per instance token this image generated.

Step 10: Optional integrations

Basic scrape and crawl to markdown work out of the box with no external keys. Two optional features can be enabled by adding your own third party keys to /etc/firecrawl/firecrawl.env and running sudo systemctl restart firecrawl:

• LLM extraction (the json format and /v1/extract): set OPENAI_API_KEY (or point OLLAMA_BASE_URL at your own Ollama) to have Firecrawl extract structured data with a model.
• Web search (/v1/search): point SEARXNG_ENDPOINT at a SearXNG instance you run.

These are optional. The self hosted image does not include Firecrawl's cloud only anti bot engine, which does not affect standard scraping and crawling.

Step 11: Use your own domain and HTTPS (production)

The image serves the API over plain HTTP on port 80, which is convenient behind a load balancer or private network. For production you should terminate TLS in front of Firecrawl so the token and payloads travel encrypted:

• Front the VM with Azure Application Gateway, a load balancer with a managed certificate, or a CDN, and forward to nginx on port 80.
• Or install your own certificate: add a TLS server block to /etc/nginx/sites-available/firecrawl referencing your certificate and key, open 443/tcp on the NSG, then sudo systemctl reload nginx.
• Point a DNS name you control at the VM public IP and use that name (and https://) as the API URL in your Firecrawl SDK.

Step 12: Server components

Step 13: Managing the Firecrawl service

sudo systemctl status firecrawl --no-pager | head -12
sudo docker compose --env-file /etc/firecrawl/images.env --env-file /etc/firecrawl/firecrawl.env -f /etc/firecrawl/compose.yaml logs --tail 40 api

Restart the whole stack with sudo systemctl restart firecrawl, follow the API logs live with sudo docker compose --env-file /etc/firecrawl/images.env --env-file /etc/firecrawl/firecrawl.env -f /etc/firecrawl/compose.yaml logs -f api, and view configuration in /etc/firecrawl/firecrawl.env and /etc/firecrawl/compose.yaml. After changing the environment file, restart the service to apply it.

Step 14: Security recommendations

• Restrict the NSG. Allow TCP 80 (and 22 for admin) only from the networks that call the API.
• Terminate TLS in front of Firecrawl (Step 11) so the bearer token and payloads are encrypted in transit.
• Treat the token like a password. It is stored in /root/firecrawl-credentials.txt (root only) and enforced at the nginx front door; rotate it by editing /etc/nginx/conf.d/firecrawl-auth.conf if it is ever exposed.
• Keep the bundled database, broker and cache private. Postgres, RabbitMQ and Redis are on a private Docker network with no host port; keep them that way.
• Keep the OS patched. Unattended security upgrades remain enabled on the running VM.

Step 15: Support and Licensing

Firecrawl is distributed under the GNU Affero General Public License v3.0 (AGPL-3.0). This cloudimg image bundles the unmodified official open source Firecrawl release; cloudimg provides the packaging, the bundled renderer, queue, broker and cache, the per instance secret generation and the nginx enforced bearer token, the paired deploy guide, and 24/7 support with a guaranteed 24 hour response SLA. Firecrawl is an independent open source project and this image is not affiliated with or endorsed by Firecrawl.

Deploy on Azure

Find Firecrawl on Ubuntu 24.04 on the Azure Marketplace, published by cloudimg. Deploy from the Portal or the Azure CLI as shown above.

Need Help?

Email support@cloudimg.co.uk for deployment help, configuration questions, or licensing enquiries.