Scrape

Scrape Endpoint

Scrape one URL into clean Markdown, HTML, links, images, screenshots, or structured data.

Getting Started

Send your API key as a bearer token. Start with the smallest request below.

Scrape a Page

Send one URL and request only the outputs your application needs.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "markdown",
    "links"
  ]
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Hacker News\n\n1. Show HN: I built an open-source web scraper with strategy caching\n2. Why Rust is eating the world...",
    "links": [
      "https://news.ycombinator.com/item?id=39912345",
      "https://news.ycombinator.com/item?id=39912346",
      "https://news.ycombinator.com/newest"
    ],
    "status": "success",
    "quality": {
      "markdown_len": 12340,
      "word_count": 1847,
      "links_count": 42,
      "headings_count": 8,
      "images_count": 3,
      "has_title": true,
      "empty_reason": null
    },
    "metadata": {
      "title": "Hacker News",
      "description": null,
      "language": "en",
      "source_url": "https://news.ycombinator.com",
      "status_code": 200,
      "word_count": 1847,
      "reading_time_seconds": 7,
      "content_length": 12340
    },
    "time_taken": 0.48
  }
}

Output Options

Output Formats

Set formats to return Markdown, HTML, raw HTML, links, images, headings, tables, product data, citations, or screenshots.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/products",
  "formats": [
    "markdown",
    "html",
    "links",
    "images",
    "headings"
  ]
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Products\n\nBrowse the current catalog.",
    "html": "<main><h1>Products</h1></main>",
    "links": [
      "https://example.com/products/widget"
    ],
    "images": [
      {
        "src": "https://example.com/widget.jpg",
        "alt": "Widget"
      }
    ],
    "headings": [
      {
        "level": 1,
        "text": "Products"
      }
    ],
    "metadata": {
      "source_url": "https://example.com/products",
      "status_code": 200
    }
  }
}

Markdown Mode

Set format_options.markdown.mode to reader, raw, or prune.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/article",
  "formats": [
    "markdown"
  ],
  "format_options": {
    "markdown": {
      "mode": "reader"
    }
  }
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Example Article\n\nClean article content.",
    "metadata": {
      "source_url": "https://example.com/article",
      "status_code": 200
    }
  }
}

Screenshot

Request a viewport, full-page, or after-actions PNG screenshot.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "screenshot"
  ],
  "format_options": {
    "screenshot": {
      "capture": "full_page",
      "full_page": true,
      "width": 1440,
      "height": 900
    }
  }
}'

Example response

{
  "success": true,
  "data": {
    "screenshot": "data:image/png;base64,iVBORw0KGgo...",
    "metadata": {
      "source_url": "https://example.com",
      "status_code": 200
    }
  }
}

Content Selection

Main Content Only

Set only_main_content to remove navigation, sidebars, and footers. It defaults to true.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/article",
  "formats": [
    "markdown"
  ],
  "only_main_content": true
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Article\n\nMain article text.",
    "metadata": {
      "source_url": "https://example.com/article",
      "status_code": 200
    }
  }
}

CSS and XPath Targets

Use one target with css_selector or xpath, or return named targets with selectors.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/products/widget",
  "formats": [
    "markdown"
  ],
  "selectors": {
    "title": {
      "css": "h1"
    },
    "price": {
      "xpath": "//*[@data-price]"
    }
  }
}'

Example response

{
  "success": true,
  "data": {
    "selector_data": {
      "title": "Widget",
      "price": "$49.00"
    },
    "metadata": {
      "source_url": "https://example.com/products/widget",
      "status_code": 200
    }
  }
}

Include and Exclude Tags

Use include_tags and exclude_tags to control which HTML elements reach extraction.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/article",
  "formats": [
    "markdown"
  ],
  "include_tags": [
    "article",
    "table"
  ],
  "exclude_tags": [
    "aside",
    "footer"
  ]
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Article\n\nSelected article and table content.",
    "metadata": {
      "source_url": "https://example.com/article",
      "status_code": 200
    }
  }
}

Page Behavior

Wait and Timeout

Set wait_for for delayed content and timeout for the request deadline, both in milliseconds.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/app",
  "formats": [
    "markdown"
  ],
  "wait_for": 1500,
  "timeout": 45000
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Loaded Application",
    "metadata": {
      "source_url": "https://example.com/app",
      "status_code": 200
    }
  }
}

Mobile View

Set mobile and an optional device preset to extract the mobile page state.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "markdown",
    "screenshot"
  ],
  "mobile": true,
  "mobile_device": "iphone_14"
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Example Mobile",
    "screenshot": "data:image/png;base64,iVBORw0KGgo...",
    "metadata": {
      "source_url": "https://example.com",
      "status_code": 200
    }
  }
}

Headers and Cookies

Send authenticated or localized page state with custom headers and cookies.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/account",
  "formats": [
    "markdown"
  ],
  "headers": {
    "Accept-Language": "en-US"
  },
  "cookies": {
    "session": "SESSION_VALUE"
  }
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Account",
    "metadata": {
      "source_url": "https://example.com/account",
      "status_code": 200
    }
  }
}

Advanced Workflows

Browser Actions

Run clicks, waits, scrolls, typing, key presses, selections, form fills, hover, navigation, or JavaScript before extraction.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/catalog",
  "formats": [
    "markdown",
    "screenshot"
  ],
  "actions": [
    {
      "type": "click",
      "selector": "button.load-more"
    },
    {
      "type": "wait",
      "milliseconds": 1000
    },
    {
      "type": "scroll",
      "direction": "down",
      "amount": 2
    }
  ]
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Expanded Catalog",
    "screenshot": "data:image/png;base64,iVBORw0KGgo...",
    "metadata": {
      "source_url": "https://example.com/catalog",
      "status_code": 200
    }
  }
}

Structured Extraction

Add extract.prompt and an optional JSON Schema to return typed fields with the page result.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/products/widget",
  "formats": [
    "markdown"
  ],
  "extract": {
    "prompt": "Extract the product name and price",
    "schema": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string"
        },
        "price": {
          "type": "string"
        }
      }
    }
  }
}'

Example response

{
  "success": true,
  "data": {
    "extract": {
      "name": "Widget",
      "price": "$49.00"
    },
    "metadata": {
      "source_url": "https://example.com/products/widget",
      "status_code": 200
    }
  }
}

Network Capture

Set capture_network to include page request and response details.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com/app",
  "formats": [
    "markdown"
  ],
  "capture_network": true
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Application",
    "network_data": {
      "requests": [],
      "responses": []
    },
    "metadata": {
      "source_url": "https://example.com/app",
      "status_code": 200
    }
  }
}

Completion

Webhook

Set webhook_url to receive completion or failure events. Set webhook_secret to verify X-DataBlue-Signature.

curl -X POST "https://api.datablue.dev/v1/scrape" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://example.com",
  "formats": [
    "markdown"
  ],
  "webhook_url": "https://api.example.net/webhooks/datablue",
  "webhook_secret": "WEBHOOK_SECRET"
}'

Example response

{
  "success": true,
  "data": {
    "markdown": "# Example",
    "metadata": {
      "source_url": "https://example.com",
      "status_code": 200
    }
  },
  "job_id": "550e8400-e29b-41d4-a716-446655440001"
}

Response

The response includes only requested outputs plus page metadata, quality information, status, and a job ID when available.

Error Handling

Handle 401 for authentication, 422 for invalid input, 429 for limits, and retry only transient failures.

Scrape vs Crawl

Use Scrape for one known page. Use Crawl when you need content from multiple connected pages.

Name	Type	Requirement	Description
url	string	Required	The URL to scrape. Protocol is auto-prepended if missing.
formats	Array<string \| object>	Optional	Outputs to return: markdown, html, raw_html, links, images, screenshot, structured_data, product_data, tables, headings, citations, or fit_markdown. Object configs set Markdown mode or screenshot capture.
format_options	object	Optional	Per-format settings for Markdown mode and screenshot capture.
only_main_content	boolean	Optional	Extract only the main content, removing navs, footers, and sidebars.
mobile	boolean	Optional	Emulate a mobile device viewport.
mobile_device	string	Optional	Device preset name: "iphone_14", "pixel_7", "ipad_pro".
timeout	number	Optional	Request timeout in milliseconds.
wait_for	number	Optional	Wait this many ms after page load before extracting.
css_selector	string	Optional	Only extract content matching this CSS selector.
xpath	string	Optional	XPath expression for targeted extraction.
selectors	object	Optional	Extract multiple named CSS or XPath targets.
include_tags	string[]	Optional	Only include these HTML tags in extraction.
exclude_tags	string[]	Optional	Exclude these HTML tags from extraction.
headers	object	Optional	Custom HTTP headers to send (e.g. { "Cookie": "session=abc" }).
cookies	object	Optional	Custom cookies to send as name/value pairs.
actions	ActionStep[]	Optional	Browser actions to perform before extraction: click, wait, scroll, type, screenshot, hover, press, select, fill_form, evaluate.
extract	object	Optional	LLM extraction config: { prompt: string, schema: JSONSchema }.
webhook_url	string	Optional	Webhook URL for job completion notification.
webhook_secret	string	Optional	HMAC secret for webhook signature verification.
capture_network	boolean	Optional	Capture browser network requests/responses.

Field	Type	Description
data.status	string	"success", "empty", "thin", or "blocked" — the outcome classification for this result.
data.quality	object	Content quality metrics: markdown_len, word_count, links_count, headings_count, images_count, has_title, and empty_reason (set when the result is empty/thin/blocked/skipped).
data.time_taken	number	Elapsed processing time in seconds for this page result.

Scrape Endpoint

Getting Started

Scrape a Page

Output Options

Output Formats

Markdown Mode

Screenshot

Content Selection

Main Content Only

CSS and XPath Targets

Include and Exclude Tags

Page Behavior

Wait and Timeout

Mobile View

Headers and Cookies

Advanced Workflows

Browser Actions

Structured Extraction

Network Capture

Completion

Webhook

Response

Error Handling

Scrape vs Crawl

Parameters

Response fields

Request and response