First Request

Make one scrape request and verify its markdown, links, images, and metadata.

Before You Begin

For cURL, install curl and optionally jq. For Python, install requests. JavaScript examples use the built-in fetch available in Node.js 18 and newer.
Use a public URL that can be opened without signing in. This guide uses https://example.com for predictable output.
Keep your API key in an environment variable. Do not paste it directly into source files.

Step 1: Create an Account

Create an account at datablue.dev or register via the API:

Example

curl -X POST "https://api.datablue.dev/v1/auth/register" \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com", "password": "YourStrongPassword1", "name": "Your Name"}'

Step 2: Generate an API Key

Log into the Dashboard and navigate to API Keys. Click Create New Key to generate a persistent API key with the wh_ prefix.

Step 3: Configure Your Shell

Example

export DATABLUE_API_KEY="wh_your_api_key"
export DATABLUE_BASE_URL="https://api.datablue.dev/v1"

test -n "$DATABLUE_API_KEY" && echo "DataBlue credentials are available"

Step 4: Make Your First Request

curl --fail-with-body --silent --show-error \
  -X POST "$DATABLUE_BASE_URL/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "images"]
  }'

Understand every value in this request

Value	Required	Meaning
`POST /v1/scrape`	Yes	Processes one known URL and returns the requested page outputs directly.
`Authorization`	Yes	Sends your API key as a Bearer credential. Keep this request on a trusted backend, worker, terminal, or server runtime.
`Content-Type`	Yes	Declares that the request body is JSON.
`url`	Yes	The public page DataBlue should retrieve and process.
`formats`	No	Selects the output fields. This example requests markdown, links, and images.

Additional Scrape controls

Option	Type	Use it when
`only_main_content`	boolean	You want the primary readable region without navigation, footer, or sidebar content.
`timeout`	number	You need to set the maximum request time in milliseconds.
`wait_for`	number	The page reveals necessary content shortly after load.
`mobile` / `mobile_device`	boolean / string	You need the page's mobile layout or a supported device preset.
`css_selector` / `xpath`	string	You want to target one known part of the document.
`include_tags` / `exclude_tags`	string[]	You need tag-level control over retained markup.
`headers` / `cookies`	object	An authorized source requires request-specific headers or cookies.
`actions`	object[]	The browser must click, wait, scroll, type, hover, press, select, fill a form, evaluate, or capture before extraction.
`extract`	object	You want prompt- or schema-guided structured extraction with the page result.
`capture_network`	boolean	You explicitly need supported browser network capture.
`webhook_url` / `webhook_secret`	string	Your integration uses the documented completion notification flow.

Example Response

Example

{
  "success": true,
  "data": {
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "links": ["https://www.iana.org/domains/example"],
    "images": [],
    "metadata": {
      "title": "Example Domain",
      "language": "en",
      "source_url": "https://example.com",
      "status_code": 200,
      "word_count": 28
    }
  }
}

Response anatomy

Field	Type	How to use it
`success`	boolean	Confirms whether the API operation succeeded. Also check the HTTP status before reading page data.
`data.markdown`	string	LLM-ready readable content when `markdown` was requested.
`data.html`	string	Normalized markup when `html` was requested.
`data.raw_html`	string	Original page markup when `raw_html` was requested.
`data.links`	array	Discovered page links when `links` was requested.
`data.images`	array	Image records and available labels when `images` was requested.
`data.headings`	array	The document outline when `headings` was requested.
`data.structured_data`	object / array	Embedded structured metadata when requested and available.
`data.screenshot`	string	The screenshot result when screenshot output was requested.
`data.metadata`	object	Source URL, title, language, source status, word count, reading time, and content length when available.
`data.status`	string	Classifies the public content result as successful, empty, thin, or blocked.
`data.quality`	object	Contains public content counts and an empty reason when the result is not usable.
`data.time_taken`	number	Measured page-processing time in seconds for this request.

Output fields are conditional. Your integration should require the fields it requested, tolerate documented optional metadata, and avoid assuming every format appears when it was not requested.

Step 5: Inspect the Response Safely

Save the complete response during development so filters do not hide useful errors.

Example

curl --fail-with-body --silent --show-error \
  -X POST "$DATABLUE_BASE_URL/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links", "images"]
  }' | tee scrape-response.json

jq '.success' scrape-response.json
jq '.data.metadata' scrape-response.json
jq -r '.data.markdown' scrape-response.json
jq '.data.links | length' scrape-response.json
jq '.data.images | length' scrape-response.json

Turn success into a shell assertion

Example

jq -e '.success == true and (.data.markdown | length > 0)' \
  scrape-response.json > /dev/null \
  && echo "Scrape succeeded" \
  || echo "Scrape failed or returned no markdown"

Select Only the Outputs You Need

Format	Returned field	Use it for
`markdown`	`data.markdown`	LLM context, indexing, reading, summarization, and document pipelines
`html`	`data.html`	Normalized HTML when structure and markup remain important
`raw_html`	`data.raw_html`	Source-level inspection and custom parsing
`links`	`data.links`	Discovery, graph building, and follow-up requests
`images`	`data.images`	Image URLs, alt text, and visual-content inventory
`headings`	`data.headings`	Outline reconstruction and section-aware processing
`structured_data`	`data.structured_data`	JSON-LD, Open Graph, and other embedded structured metadata
`screenshot`	`data.screenshot`	Visual evidence, QA, and rendered-page archives

Large formats increase response size. Request raw_html or screenshots only when the consumer actually needs them.

Request a full-page screenshot

Example

curl --fail-with-body --silent --show-error \
  -X POST "$DATABLUE_BASE_URL/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": [
      "markdown",
      {
        "type": "screenshot",
        "capture": "full_page",
        "full_page": true,
        "width": 1440,
        "height": 900
      }
    ]
  }' > screenshot-response.json

Reader Content and Full-Page Content

only_main_content chooses focused reading content or broader page content.

Goal	Recommended settings
Articles, guides, and LLM context	`only_main_content: true` with markdown mode `reader`
Navigation, headers, footers, and complete page context	`only_main_content: false` with markdown mode `raw`
Smaller relevance-focused markdown	Markdown mode `prune`

Example

curl --fail-with-body --silent --show-error \
  -X POST "$DATABLUE_BASE_URL/scrape" \
  -H "Authorization: Bearer $DATABLUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "only_main_content": false,
    "formats": [
      {"type": "markdown", "mode": "raw"},
      "links",
      "images"
    ]
  }'

Common First-Request Failures

The request returns 401

Check the environment variable, Bearer header, and key status. Then test GET /v1/auth/me.

The request returns 422

The JSON body failed validation. Confirm that url is a string, formats is an array, and the request uses Content-Type: application/json.

The response is successful but a field is absent

Optional fields may be absent. Check the requested formats before treating that as an error.

The page returns little or no useful content

Check data.metadata, public reachability, content mode, and wait_for.

The request times out

Use a stable page and request only the outputs you need. See Production Readiness for retries.

Move From Test to Integration

Continue with PythonInstall datablue==2.1.0 and use typed sync or async clients. Continue with Node.jsInstall @datablue/sdk@1.1.0 for a typed async client. Connect an AI agentRun @datablue/mcp@latest locally over stdio and expose explicit DataBlue tools. Collect multiple pagesMove to a bounded asynchronous crawl after validating one-page output.