Crawl

Start a crawl job that discovers and scrapes pages starting from a seed URL using BFS, DFS, or best-first strategy. Returns a job ID for tracking progress via SSE or polling.

Execution model

Live request

Runtime depends on endpoint, target, pagination, rendering mode, and active plan limits.

Credit weight

Live catalog

Current weights are managed from the Data API Weights admin table and shown on pricing before use.

Parameters

Name	Type	Requirement	Description
url	string	Required	The starting URL to crawl.
max_pages	number	Optional	Maximum pages to crawl (1-1000).
max_depth	number	Optional	Maximum link depth from the starting URL (1-10).
concurrency	number	Optional	Number of parallel scrape workers (1-10).
crawl_strategy	string	Optional	Strategy: "bfs" (breadth-first), "dfs" (depth-first), or "bff" (best-first).
allow_external_links	boolean	Optional	Follow links to external domains.
respect_robots_txt	boolean	Optional	Obey the target site's robots.txt rules.
include_paths	string[]	Optional	Only crawl URLs matching these glob path patterns.
exclude_paths	string[]	Optional	Skip URLs matching these glob path patterns.
scrape_options	object	Optional	Options passed to each page scrape: { formats, only_main_content, wait_for, timeout, include_tags, exclude_tags, mobile, extract }.
filter_faceted_urls	boolean	Optional	Deduplicate faceted/navigation URL variations.
webhook_url	string	Optional	Webhook URL for crawl completion notification.
webhook_secret	string	Optional	HMAC secret for webhook signature verification.

Response fields

Field	Type	Description
job_id	string	Poll GET /crawl/{job_id} (or subscribe via SSE) for progress and results.
data[].* (on completion)	object	Each crawled page returns the same shape as a single scrape — markdown/html/links/metadata plus status, quality, time_taken, and depth.

Request and response

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://docs.python.org/3/",
  "max_pages": 50,
  "max_depth": 2,
  "crawl_strategy": "bfs",
  "include_paths": [
    "/3/library/*"
  ]
}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}