Crawl4AI on Ubuntu 24.04 on Azure User Guide

Overview

Crawl4AI is a fast, open source web crawler and scraper that converts web pages into clean, LLM ready Markdown and structured data through a simple REST API. It drives a headless Chromium browser, so it renders JavaScript heavy pages the way a real browser sees them, and returns Markdown tuned for large language model context along with screenshots, PDF capture, on page JavaScript execution and streaming or asynchronous job endpoints.

The cloudimg image runs the official Crawl4AI server (unclecode/crawl4ai:0.9.2) as a Docker container bound to loopback 127.0.0.1:11235, behind nginx on port 80. nginx is ready for your TLS certificate.

Secure by default, no open crawler: an unauthenticated web crawler is a server side request forgery and denial of service risk, so this image never exposes an open crawl endpoint. Crawl4AI binds to host loopback only and natively enforces a bearer API token on every API endpoint. A crawl4ai-firstboot.service oneshot generates a unique API token on each VM's first boot, sets it on the container and writes it to a root only file, resolves the instance URL and writes a root only info note. No two VMs share a token and none is baked into the image. An unauthenticated /healthz endpoint stays open for monitoring.

What is included:

• Crawl4AI 0.9.2 as the official server Docker container (headless Chromium bundled)
• docker.service running the container bound to 127.0.0.1:11235
• nginx reverse proxy on port 80, ready for TLS
• crawl4ai-firstboot.service for the first boot API token, container creation and info note
• A unique per VM API token generated on first boot, in a root only 0600 file
• Ubuntu 24.04 LTS base, fully patched
• 24/7 cloudimg support, 24h response SLA

Prerequisites

An active Azure subscription, an SSH key pair, and a VNet with a subnet. Crawl4AI drives a real headless Chromium, which is memory sensitive. Recommended VM size: Standard_B2s (2 vCPU, 4 GB RAM) for evaluation and light crawling, or a Standard_D2s_v5 or larger for concurrent or heavy crawling. The crawler needs outbound internet access to fetch the pages you ask it to crawl.

Step 1: Deploy from the Azure Portal

Search the Marketplace for Crawl4AI on Ubuntu 24.04, choose your VM size, and attach an NSG that allows TCP 22 (SSH) from your management network and TCP 80 / 443 (the crawl API) from the networks and applications that will call it. Front port 80 with TLS in production (see the HTTPS section below).

Step 2: Deploy from the Azure CLI

az vm create \
  --resource-group my-rg \
  --name crawl4ai-01 \
  --image <marketplace-image-urn> \
  --size Standard_B2s \
  --admin-username azureuser \
  --generate-ssh-keys \
  --public-ip-sku Standard

# Open the crawl API port to the callers that need it
az vm open-port --resource-group my-rg --name crawl4ai-01 --port 80

Step 3: Verify the services and retrieve your API token

SSH in as azureuser. First boot creates the container, generates your per VM API token and brings the stack up. Confirm the services are healthy:

systemctl is-active docker nginx
docker ps --format '{{.Names}}\t{{.Status}}'
curl -s -o /dev/null -w 'healthz: HTTP %{http_code}\n' http://127.0.0.1/healthz

Expected output:

active
active
crawl4ai    Up 3 minutes (healthy)
healthz: HTTP 200

Crawl4AI listens only on host loopback 127.0.0.1:11235; nginx fronts it on port 80.

Read your unique API token from the root only info note:

sudo cat /root/crawl4ai-info.txt

The file (mode 0600, owned by root) contains your CRAWL4AI_URL and CRAWL4AI_TOKEN. This token is unique to this VM and was generated on first boot; keep it secret.

Step 4: Secure by default

Crawl4AI enforces your API token on every crawl request. An unauthenticated or wrong token request is rejected with HTTP 401; only your per VM token is accepted. You can prove this from the VM:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
echo "no token   -> HTTP $(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1/crawl -H 'Content-Type: application/json' -d '{"urls":["https://example.com"]}')"
echo "wrong token-> HTTP $(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1/crawl -H 'Authorization: Bearer wrong' -H 'Content-Type: application/json' -d '{"urls":["https://example.com"]}')"
echo "your token -> HTTP $(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1/md -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' -d '{"url":"https://example.com"}')"

Expected output:

no token   -> HTTP 401
wrong token-> HTTP 401
your token -> HTTP 200

Step 5: Your first crawl

The /md endpoint turns a page into clean Markdown in a single call. Pass your token as a bearer header:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
curl -s -X POST http://127.0.0.1/md \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | jq -r .markdown

Expected output:

# Example Domain
This domain is for use in documentation examples without needing permission. Avoid use in operations.
[Learn more](https://iana.org/domains/example)

From a remote machine, replace 127.0.0.1 with your VM's address and use the same token. For production put TLS in front (see below) and call https://your-domain/md.

Step 6: Full crawl with structured output

The /crawl endpoint returns the full result set, including the raw HTML, links, media and several Markdown variants. Use it when you want more than the plain Markdown:

TOKEN=$(sudo grep '^CRAWL4AI_TOKEN=' /root/crawl4ai-info.txt | cut -d= -f2-)
curl -s -X POST http://127.0.0.1/crawl \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com"],"crawler_config":{"type":"CrawlerRunConfig","params":{"cache_mode":"bypass"}}}' \
  | jq -r '.results[0] | "url: " + .url, "status: " + (.status_code|tostring), "links: " + (.links.internal|length|tostring), "markdown chars: " + (.markdown.raw_markdown|length|tostring)'

Expected output:

url: https://example.com
status: 200
links: 0
markdown chars: 166

Crawl4AI also exposes /html (sanitised HTML), /screenshot (a PNG of the rendered page), /pdf (a PDF), /execute_js (run JavaScript on the page) and streaming and job endpoints. See the Crawl4AI documentation for the full API and its rich extraction options.

Step 7: Updates, persistence and patching

The container image is pinned at build time and re-pulled and recreated cleanly on first boot, so every VM starts from a known good image. The container runs with a restart policy so it survives reboots.

docker inspect -f 'image: {{index .Config.Image}}  restart: {{.HostConfig.RestartPolicy.Name}}' crawl4ai
systemctl is-enabled crawl4ai-firstboot.service docker nginx
apt-mark showhold

Expected output:

image: unclecode/crawl4ai:0.9.2  restart: unless-stopped
enabled
enabled
enabled

The empty last line from apt-mark showhold confirms no security updates are held back. The Ubuntu base ships fully patched with unattended security upgrades enabled.

Enabling HTTPS

nginx fronts Crawl4AI on port 80 and is ready for your certificate. For a public domain, point an A record at the VM, then install a certificate with Certbot:

sudo apt-get update
sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx -d your-domain.example.com

Certbot obtains and installs the certificate and reloads nginx for HTTPS on port 443, with automatic renewal. After that, call the API at https://your-domain.example.com/md. Restrict port 80 / 443 with your NSG to the networks that need the crawler, and rotate the API token by recreating the container with a new CRAWL4AI_API_TOKEN (the command is in /root/crawl4ai-info.txt).

Support

Every cloudimg deployment includes 24/7 support with a 24 hour response SLA. Contact support@cloudimg.co.uk for help with deployment, configuration or scaling.

Crawl4AI is open source software licensed under the Apache License 2.0. This is a repackaged open source product with additional charges for cloudimg support services. Crawl4AI is a trademark of its respective owner.