Map

Discover all URLs on a website by combining sitemap.xml parsing, robots.txt discovery, and link crawling. Returns a flat list of URLs with metadata.

Execution model

Live request

Runtime depends on endpoint, target, pagination, rendering mode, and active plan limits.

Credit weight

Live catalog

Current weights are managed from the Data API Weights admin table and shown on pricing before use.

Parameters

Name	Type	Requirement	Description
url	string	Required	The website URL to map.
search	string	Optional	Filter URLs matching this search string.
limit	number	Optional	Maximum number of URLs to return.
include_subdomains	boolean	Optional	Include URLs from subdomains.
use_sitemap	boolean	Optional	Parse sitemap.xml for URL discovery.
respect_robots_txt	boolean	Optional	Honor robots exclusions during rendered discovery.

Response fields

Field	Type	Description
status	string	"success" or "empty" (with empty_reason) — a zero-link map is reported as empty, and a failed root fetch as failed, rather than a silent empty success.
links[].source	string	How each URL was discovered: "sitemap", "robots", or "crawl".

Code example

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "url": "https://docs.python.org",
  "limit": 200,
  "include_subdomains": false,
  "respect_robots_txt": true
}'

System responses

200 OK

Request processed successfully.

401 UNAUTHORIZED

Missing or invalid API key.

429 RATE LIMIT

System capacity exceeded.

500 SYSTEM FAILURE

Internal core exception.

Example response

{
  "success": true,
  "total": 187,
  "status": "success",
  "links": [
    {
      "url": "https://docs.python.org/3/",
      "title": "Python 3 Documentation",
      "lastmod": "2026-03-28",
      "source": "sitemap"
    },
    {
      "url": "https://docs.python.org/3/tutorial/index.html",
      "title": "The Python Tutorial",
      "source": "sitemap"
    },
    {
      "url": "https://docs.python.org/3/library/index.html",
      "title": "The Python Standard Library",
      "source": "crawl"
    },
    {
      "url": "https://docs.python.org/3/reference/index.html",
      "title": "The Python Language Reference",
      "source": "robots"
    }
  ]
}