Docs / API reference

POST/v1/crawl

Crawl

Start a crawl job that discovers and scrapes pages starting from a seed URL using BFS, DFS, or best-first strategy. Returns a job ID for tracking progress via SSE or polling.

Execution model

Live request

Runtime depends on endpoint, target, pagination, rendering mode, and active plan limits.

Credit weight

Live catalog

Current weights are managed from the Data API Weights admin table and shown on pricing before use.

Crawl Endpoint

Discover and scrape a bounded set of connected pages from one starting URL.

Getting Started

Send your API key as a bearer token. Start with the smallest request below.

Start a Crawl

Set a page limit and depth boundary before starting the asynchronous job.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 10,
  "max_depth": 2
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Check Crawl Status

Poll the job until status is completed, failed, or cancelled.

curl -X GET "https://api.datablue.dev/v1/crawl/JOB_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "completed",
  "total_pages": 10,
  "completed_pages": 10,
  "total_results": 10,
  "page": 1,
  "per_page": 20,
  "data": [
    {
      "url": "https://example.com/docs",
      "markdown": "# Documentation",
      "links": [
        "https://example.com/docs/start"
      ]
    }
  ]
}

Crawl Boundaries

Pages, Depth, and Concurrency

Control the job with max_pages (1–1000), max_depth (1–10), and concurrency (1–10).

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 50,
  "max_depth": 3,
  "concurrency": 8
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Crawl Strategy

Choose bfs, dfs, or bff for breadth-first, depth-first, or best-first traversal.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 25,
  "crawl_strategy": "bfs"
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Path Filters

Use glob patterns in include_paths and exclude_paths to keep the crawl in scope.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 25,
  "include_paths": [
    "/docs/**"
  ],
  "exclude_paths": [
    "/docs/archive/**"
  ]
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Domains, Robots, and Facets

Keep external links disabled by default, respect robots.txt, and filter repeated faceted URLs.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 25,
  "allow_external_links": false,
  "respect_robots_txt": true,
  "filter_faceted_urls": true
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Page Scraping

Page Outputs

Set scrape_options once to apply the same output and content settings to every page.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 10,
  "scrape_options": {
    "formats": [
      "markdown",
      "links",
      "images"
    ],
    "only_main_content": true,
    "timeout": 30000
  }
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Structured Page Data

Add scrape_options.extract to apply one extraction prompt and schema to every crawled page.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/catalog",
  "max_pages": 10,
  "scrape_options": {
    "formats": [
      "markdown"
    ],
    "extract": {
      "prompt": "Extract the page title and category",
      "schema": {
        "type": "object",
        "properties": {
          "title": {
            "type": "string"
          },
          "category": {
            "type": "string"
          }
        }
      }
    }
  }
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Manage Jobs

Webhook

Set webhook_url and an optional secret to receive the terminal crawl event.

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/docs",
  "max_pages": 10,
  "webhook_url": "https://api.example.net/webhooks/datablue",
  "webhook_secret": "WEBHOOK_SECRET"
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}

Cancel a Crawl

Cancel a pending or running crawl by job ID.

curl -X POST "https://api.datablue.dev/v1/jobs/JOB_ID/cancel" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "cancelled"
}

Retry a Failed Crawl

Retry a failed crawl with its saved request configuration.

curl -X POST "https://api.datablue.dev/v1/jobs/JOB_ID/retry" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "pending"
}

Export Results

Download completed crawl results as JSON. CSV and ZIP are also supported.

curl -X GET "https://api.datablue.dev/v1/crawl/JOB_ID/export?format=json" \
  -H "Authorization: Bearer YOUR_API_KEY"
Example response
[
  {
    "url": "https://example.com/docs",
    "markdown": "# Documentation",
    "links": [
      "https://example.com/docs/start"
    ]
  }
]

Response

The start response contains a job ID. Status responses add progress and paginated page results as they become available.

Error Handling

Handle 401 for authentication, 422 for invalid input, 429 for limits, and retry only transient failures.

Crawl vs Map

Use Crawl to retrieve page content. Use Map when you only need a fast URL inventory.

Parameters

NameTypeRequirementDescription
urlstringRequiredThe starting URL to crawl.
max_pagesnumberOptionalMaximum pages to crawl (1-1000).
max_depthnumberOptionalMaximum link depth from the starting URL (1-10).
concurrencynumberOptionalNumber of parallel scrape workers (1-10).
crawl_strategystringOptionalStrategy: "bfs" (breadth-first), "dfs" (depth-first), or "bff" (best-first).
allow_external_linksbooleanOptionalFollow links to external domains.
respect_robots_txtbooleanOptionalObey the target site's robots.txt rules.
include_pathsstring[]OptionalOnly crawl URLs matching these glob path patterns.
exclude_pathsstring[]OptionalSkip URLs matching these glob path patterns.
scrape_optionsobjectOptionalOptions passed to each page scrape: { formats, only_main_content, wait_for, timeout, include_tags, exclude_tags, mobile, extract }.
filter_faceted_urlsbooleanOptionalDeduplicate faceted/navigation URL variations.
webhook_urlstringOptionalWebhook URL for crawl completion notification.
webhook_secretstringOptionalHMAC secret for webhook signature verification.

Response fields

FieldTypeDescription
job_idstringPoll GET /crawl/{job_id} (or subscribe via SSE) for progress and results.
data[].* (on completion)objectEach crawled page returns the same shape as a single scrape — markdown/html/links/metadata plus status, quality, time_taken, and depth.

Request and response

curl -X POST "https://api.datablue.dev/v1/crawl" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://docs.python.org/3/",
  "max_pages": 50,
  "max_depth": 2,
  "crawl_strategy": "bfs",
  "include_paths": [
    "/3/library/*"
  ]
}'
Example response
{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "started",
  "message": "Crawl job started"
}