Docs / API reference

POST/v1/search

Search

Discover ranked organic results with Google, Bing, or DuckDuckGo, then optionally scrape every result page into LLM-ready content. Search runs asynchronously and preserves result rank across the full job lifecycle.

Execution model

Live request

Runtime depends on endpoint, target, pagination, rendering mode, and active plan limits.

Credit weight

Live catalog

Current weights are managed from the Data API Weights admin table and shown on pricing before use.

Search Endpoint

Search the web for ranked organic results with Google, Bing, or DuckDuckGo. Return discovery metadata, or scrape every result into LLM-ready content.

Google uses DataBlue's rendered Google search engine by default. Bing and DuckDuckGo are no-key selectable alternatives, and Search may use an eligible fallback when the preferred engine returns no usable results. Each response identifies the requested engine, engine used, and fallback chain.

Getting Started

Send your API key as a bearer token. Search runs asynchronously, so start a job and then read its status.

Start a Search

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"best web scraping tools","engine":"google","num_results":10,"formats":[]}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"best web scraping tools\""
}

Check Search Status

Use the returned job_id until the status is completed, failed, or cancelled.

curl "https://api.datablue.dev/v1/search/JOB_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "completed",
  "query": "best web scraping tools",
  "engine_requested": "google",
  "engine_used": "google",
  "provider_used": "google_rendered",
  "fallback_used": false,
  "fallback_chain": [
    {
      "provider": "google_rendered",
      "status": "success",
      "results_count": 1,
      "elapsed_ms": 1840
    }
  ],
  "total_results": 1,
  "completed_results": 1,
  "data": [
    {
      "url": "https://example.com/guide",
      "rank": 1,
      "title": "Web Scraping Guide",
      "snippet": "A practical guide to web scraping.",
      "success": true,
      "scraped": false
    }
  ]
}

Result Types

Discovery Results

Set formats to [] to return ranked organic URLs, titles, and snippets without scraping destination pages.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "latest Python releases",
    "engine": "google",
    "num_results": 10,
    "formats": [],
    "language": "en",
    "country": "US",
    "safe_search": true
  }'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"latest Python releases\""
}

Scraped Results

Request one or more formats to process every discovered URL with DataBlue's single-page Scrape engine. Search defaults to the markdown format.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "web scraping documentation",
    "engine": "duckduckgo",
    "num_results": 10,
    "formats": ["markdown", "links", "images"],
    "only_main_content": true
  }'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"web scraping documentation\""
}

Choosing a result type.

Need	Formats	Returned data
Find and rank sources	`[]`	Rank, URL, title, snippet
Build a research corpus	`["markdown"]`	Discovery metadata plus requested page formats
Extract typed facts	One or more formats	Page content plus `extract` output

Search Options

Use these fields to control how many results Search returns and where those results come from.

Options at a Glance

Option	Default	Purpose
`query`	required	Organic search terms
`num_results`	`10`	Total unique organic results, from 1 through 50
`engine`	`google`	Preferred engine: `google`, `bing`, or `duckduckgo`
`formats`	`["markdown"]`	Use `[]` for discovery only, or request page content formats
`language`	`en`	Preferred result language
`country`	none	Optional two-letter geographic context
`safe_search`	`false`	Enable filtered search results

Result Count

Set num_results from 1 to 50. Search stops after collecting enough unique organic URLs and may return fewer when fewer valid results are available.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"web scraping tools","num_results":20,"formats":[]}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"web scraping tools\""
}

Language and Country

Set language for the preferred result language and country for two-letter geographic context, such as en with IN.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"best laptops","language":"en","country":"IN","formats":[]}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"best laptops\""
}

Safe Search

Set safe_search to true to request filtered Google results.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"family travel videos","safe_search":true,"formats":[]}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"family travel videos\""
}

Content Scraping

These options apply when one or more destination-page formats are requested.

Output Formats

Request only the outputs you need. Supported formats are markdown, html, raw_html, links, images, headings, structured_data, and screenshot. Unknown formats such as json receive 422. data[].screenshot is a Base64-encoded PNG string. Screenshot options support capture (viewport or full_page), full_page, width, and height.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "web scraping documentation",
    "formats": ["markdown", "links", "screenshot"],
    "format_options": {
      "markdown": {"mode": "reader"},
      "screenshot": {"capture": "full_page", "full_page": true}
    }
  }'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"web scraping documentation\""
}

Main Content Only

only_main_content defaults to true. Set it to false when navigation, footers, and surrounding page content are required.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"company documentation","only_main_content":false}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"company documentation\""
}

Mobile View

Set mobile to true to scrape mobile page variants. Add mobile_device when a specific device preset is needed.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"mobile shopping pages","mobile":true,"mobile_device":"iphone_14"}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"mobile shopping pages\""
}

Headers and Cookies

Use headers and cookies for the destination pages scraped from the results. They do not change search-result discovery.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "product documentation",
    "headers": {"Accept-Language": "en-IN"},
    "cookies": {"region": "IN"}
  }'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"product documentation\""
}

Structured Extraction

Provide extract.prompt to transform each successfully scraped page into focused JSON. Add extract.schema to request a stable shape. The result is returned in each item's extract field.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Python web frameworks",
    "num_results": 10,
    "formats": ["markdown"],
    "extract": {
      "prompt": "Extract the framework name and its main use case",
      "schema": {
        "type": "object",
        "properties": {
          "name": {"type": "string"},
          "use_case": {"type": "string"}
        },
        "required": ["name", "use_case"]
      }
    }
  }'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "completed",
  "query": "Python web frameworks",
  "total_results": 1,
  "completed_results": 1,
  "time_taken": 7.106,
  "data": [
    {
      "id": "18c8bb75-32dc-48e0-a2c1-d973e2a41a62",
      "url": "https://example.com/framework",
      "rank": 1,
      "title": "Example Framework",
      "success": true,
      "scraped": true,
      "extract": {
        "name": "Example Framework",
        "use_case": "Building web applications"
      }
    }
  ]
}

Response

Job and Result Fields

The status response contains the job lifecycle and the results collected so far. Destination-page fields that were not requested are omitted.

Field	Meaning
`status`	Current job state
`total_results`	Unique organic results accepted for the job
`completed_results`	Result records completed so far
`data[]`	Ranked result items in organic order
`data[].success`	Whether this result completed without an item-level error
`data[].scraped`	Whether the destination page was processed for requested formats
`data[].error`	Destination-page failure for this result, when present
`time_taken`	Measured job processing time in seconds, when available
`error`	Top-level job failure, when present

Result items. Each data[] item contains id and url and can include rank, title, snippet, requested output fields, extract, metadata, and error.

Partial Failures

If one destination page cannot be scraped, its search result is retained with success: false, scraped: false, and an error. Other results continue processing.

Example

{
  "url": "https://example.com/unavailable",
  "rank": 3,
  "title": "Unavailable Result",
  "snippet": "The organic search snippet is preserved.",
  "success": false,
  "scraped": false,
  "error": "Page could not be scraped"
}

Webhooks

Set webhook_url to receive job.completed or job.failed. Add webhook_secret to sign deliveries with X-DataBlue-Signature. See the Webhooks guide for verification and retries.

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "web scraping documentation",
    "num_results": 10,
    "formats": [],
    "webhook_url": "https://your-app.example/webhooks/datablue",
    "webhook_secret": "YOUR_WEBHOOK_SECRET"
  }'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search started for \"web scraping documentation\""
}

Manage Jobs

Cancel a Search

Cancel a pending or running job with POST /v1/jobs/{job_id}/cancel. Cancelling an already-cancelled job returns success.

curl -X POST "https://api.datablue.dev/v1/jobs/JOB_ID/cancel" \
  -H "Authorization: Bearer YOUR_API_KEY"

Example response

{
  "success": true,
  "message": "Job cancelled",
  "job_id": "550e8400-e29b-41d4-a716-446655440001"
}

Retry a Failed Search

Retry a failed or cancelled job with POST /v1/jobs/{job_id}/retry. The retry creates a new job with the same configuration.

curl -X POST "https://api.datablue.dev/v1/jobs/JOB_ID/retry" \
  -H "Authorization: Bearer YOUR_API_KEY"

Example response

{
  "success": true,
  "message": "Job retried successfully",
  "original_job_id": "550e8400-e29b-41d4-a716-446655440001",
  "new_job_id": "550e8400-e29b-41d4-a716-446655440002",
  "type": "search"
}

Export Results

JSON, CSV, and ZIP. Download completed Search results from GET /v1/search/{job_id}/export?format=json. The supported export values are json, csv, and zip.

curl "https://api.datablue.dev/v1/search/JOB_ID/export?format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -o search-results.json

Example response

{
  "query": "latest Python releases",
  "results": [
    {
      "url": "https://www.python.org/downloads/",
      "title": "Download Python",
      "snippet": "Download the latest version of Python.",
      "success": true
    }
  ]
}

Change format to csv or zip and use the matching output filename to download those export types. ZIP exports include an index, full JSON data, and per-result folders for available content and metadata.

Retention. Raw Search results are available for 48 hours. Export before that window closes. After expiry, the status response returns an empty data array with result_expired: true, retention_hours, and expired_at. An expired export returns 410; a job with no exportable results returns 404.

Error Handling

Signal	Meaning	Client action
`400`	Cancel or retry is invalid for the current job state	Read status before changing job state
`401`	Missing or invalid bearer credential	Replace the API key; do not retry unchanged credentials
`404`	Job is missing, belongs to another user, or has no exportable results	Verify the job ID and authenticated account
`422`	Request validation failed	Correct the query, engine, result count, formats, or field types
`429`	Quota or active-job concurrency limit reached	Honor `Retry-After` and avoid duplicate submissions
`410`	Raw export data passed its retention window	Run a new Search job
`status: failed`	Search discovery or the job failed	Read the top-level `error`; retry only when appropriate
`data[].success: false`	One requested destination page failed	Keep successful items and inspect the item-level `error`

Search vs Google SERP API

Use Search for optional page content. Use Search when you need organic result discovery and may also need markdown, HTML, links, images, screenshots, or extraction from the destination pages.

Use Google SERP for search-page data. Use the dedicated Google SERP API when you need its richer Google search response contract instead of optional destination-page scraping.

Parameters

Name	Type	Requirement	Description
query	string	Required	The search query.
num_results	number	Optional	Total organic results to return (1-50).
engine	"google" \| "bing" \| "duckduckgo"	Optional	Preferred discovery engine. Search may use an eligible fallback when it returns no usable results.
language	string	Optional	Preferred search-result language.
country	string	Optional	Optional two-letter country code for geographic result context.
safe_search	boolean	Optional	Request filtered search results.
formats	string[]	Optional	Destination-page outputs: markdown, html, raw_html, links, screenshot, structured_data, headings, and images. Use [] for discovery only.
format_options	object	Optional	Per-format settings, such as markdown.mode or screenshot.capture/full_page/width/height.
only_main_content	boolean	Optional	Extract only the primary content from each result page.
mobile	boolean	Optional	Emulate a mobile device viewport for scraping.
mobile_device	string	Optional	Mobile device preset name.
headers	object	Optional	Custom HTTP headers sent while scraping result pages.
cookies	object	Optional	Custom cookies sent while scraping result pages.
extract	object	Optional	LLM extraction config applied to each result: { prompt, schema, provider, model }.
webhook_url	string	Optional	Webhook URL for search completion notification.
webhook_secret	string	Optional	HMAC secret for webhook signature verification.

Response fields

Field	Type	Description
success	boolean	Whether the API response itself was produced successfully.
job_id	string	Search job identifier used by status, export, cancel, and retry endpoints.
status	string	Current job status: started, pending, running, completed, failed, or cancelled.
query	string	Original Search query.
engine_requested	string	Preferred engine from the request.
engine_used	string	Search engine that produced the accepted discovery rows.
provider_used	string	Concrete provider adapter that produced the accepted rows.
fallback_used	boolean	Whether a provider after the preferred path produced the rows.
fallback_chain	object[]	Safe provider-attempt statuses, counts, and timings in execution order.
total_results	number	Number of unique organic results accepted for the job.
completed_results	number	Number of result records completed so far.
time_taken	number	Measured job processing time in seconds, when available.
data[].id	string	Persisted result identifier.
data[].rank	number	One-based organic result rank.
data[].url	string	Canonical result URL.
data[].title	string	Organic result title.
data[].snippet	string	Organic result snippet.
data[].success	boolean	Whether this result completed without an item-level error.
data[].scraped	boolean	Whether the destination page was processed for requested formats.
data[].requested outputs	object	Requested page formats, extraction output, and page metadata added when destination scraping is requested. Screenshot is a Base64-encoded PNG string.
data[].error	string	Item-level destination-page failure, when present.
error	string	Top-level job failure, when present.

Request and response

curl -X POST "https://api.datablue.dev/v1/search" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "query": "best python web scraping libraries 2026",
  "engine": "google",
  "num_results": 10,
  "formats": [
    "markdown"
  ]
}'

Example response

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440001",
  "status": "started",
  "message": "Search job started"
}