Map

Map Endpoint

Map builds a normalized inventory of URLs for one website. It returns links and available sitemap metadata without scraping the content of every page.

What Map Returns

Each result contains a normalized url. Sitemap records may also include lastmod and priority; page-link discovery may provide a title or description. The source field identifies how the URL was discovered.

Map does not return page content. Use Scrape for one page or Crawl for content from many pages.

Authentication

Bearer API key. Send Authorization: Bearer YOUR_API_KEY with every Map request. Missing, malformed, or invalid credentials return 401. Keep API keys in server-side environment variables.

Quickstart

Start a Map

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":100}'

{
  "success": true,
  "total": 0,
  "links": [],
  "job_id": "550e8400-e29b-41d4-a716-446655440002"
}

Check Map Status

A cold request returns a job_id. Poll the status route while status is pending or running, then stop on completed, failed, or cancelled.

curl "https://api.datablue.dev/v1/map/JOB_ID" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "job_id": "550e8400-e29b-41d4-a716-446655440002",
  "status": "completed",
  "map_status": "success",
  "url": "https://example.com",
  "total": 2,
  "completed_pages": 2,
  "total_pages": 2,
  "time_taken": 2.418,
  "links": [
    {
      "url": "https://example.com/",
      "source": "sitemap"
    },
    {
      "url": "https://example.com/docs",
      "lastmod": "2026-07-01",
      "priority": 0.8,
      "source": "sitemap"
    }
  ]
}

Map Options

In the Playground, open Map Configuration to set these options. The table shows the matching API field for each control.

Options at a Glance

Limit is a maximum. A site can return fewer URLs when its eligible sitemap tree is exhausted, its pages expose fewer links, or the selected scope and filter remove candidates.

Normalization. Map removes fragments, rejects unrelated domains, and deduplicates equivalent URLs before returning results.

The option examples below show the completed link response. A cold request first returns a job_id; use Check Map Status from Quickstart to retrieve the completed response.

Limit Results

Set limit to the maximum number of unique URLs you need. Map stops discovery as soon as it has enough matching results.

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":3}'

{
  "success": true,
  "total": 3,
  "map_status": "success",
  "links": [
    {
      "url": "https://example.com/",
      "source": "sitemap"
    },
    {
      "url": "https://example.com/docs",
      "source": "sitemap"
    },
    {
      "url": "https://example.com/pricing",
      "source": "sitemap"
    }
  ]
}

Filter URLs

The Playground control Filter (optional) maps to the API field search. It keeps URLs whose path, available title, or description matches the phrase or its words. It does not search inside page content or submit a query to the target website.

The limit applies to matching results. Non-matching URLs are removed, matching URLs are ranked by relevance, and discovery continues until Map reaches the limit or exhausts the available sources.

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://docs.example.com","search":"/guides/","limit":2}'

{
  "success": true,
  "total": 2,
  "map_status": "success",
  "links": [
    {
      "url": "https://docs.example.com/guides/getting-started",
      "source": "sitemap"
    },
    {
      "url": "https://docs.example.com/guides/deployment",
      "source": "sitemap"
    }
  ]
}

A compatible saved inventory can return filtered links immediately, as shown above. A cold request returns a job_id; poll it with the status request from Quickstart to receive the same filtered link shape when the job completes.

Use Sitemap

Set use_sitemap to true to read sitemap locations declared by the site and follow nested sitemap indexes. Set it to false to skip sitemap parsing and use saved inventory plus page-link discovery.

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "limit": 2,
    "use_sitemap": true
  }'

{
  "success": true,
  "total": 2,
  "map_status": "success",
  "links": [
    {
      "url": "https://docs.example.com/",
      "source": "sitemap"
    },
    {
      "url": "https://docs.example.com/api",
      "lastmod": "2026-07-01",
      "source": "sitemap"
    }
  ]
}

Include Subdomains

Set include_subdomains to true to keep URLs from matching subdomains such as blog.example.com. Set it to false to keep results on the requested hostname.

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":2,"include_subdomains":true}'

{
  "success": true,
  "total": 2,
  "map_status": "success",
  "links": [
    {
      "url": "https://example.com/",
      "source": "sitemap"
    },
    {
      "url": "https://blog.example.com/updates",
      "source": "sitemap"
    }
  ]
}

Respect Robots.txt

Set respect_robots_txt to true to apply robots exclusions during page discovery and result filtering. Turning it off does not enable sitemap discovery; use_sitemap controls sitemap parsing independently.

curl -X POST "https://api.datablue.dev/v1/map" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","limit":2,"use_sitemap":false,"respect_robots_txt":true}'

{
  "success": true,
  "total": 2,
  "map_status": "success",
  "links": [
    {
      "url": "https://example.com/",
      "source": "http"
    },
    {
      "url": "https://example.com/docs",
      "source": "http"
    }
  ]
}

Discovery Sources

Discovery Order

1. Read a compatible cached map or persistent URL inventory.
2. Read declared sitemap locations when sitemap discovery is enabled.
3. Traverse nested sitemap indexes until page URLs or the requested limit are reached.
4. Use homepage link discovery only when the cheaper sources are insufficient.
5. Normalize, filter, deduplicate, and return in-scope URLs.

Index and Sitemaps

Persistent inventory. Successful discovery can populate a domain URL inventory. A later compatible request can return immediately when that inventory already satisfies the requested scope and limit. If it is insufficient, Map continues into live discovery.

Declared sitemap locations. With use_sitemap: true, Map reads sitemap URLs published through robots.txt. Sites can declare multiple regional, language, product, or content roots.

Nested sitemap indexes. A sitemap index can point to more indexes or final URL sets. Map follows declared children, tracks visited files to avoid loops, and collects each safe loc it reaches.

Homepage Discovery

If the inventory and sitemap tree cannot answer the request, Map extracts internal links from the homepage. It starts with lightweight page acquisition and uses rendered discovery only when the page is blocked, empty, or client-rendered.

Same-session traversal. When the homepage still exposes too few links, Map may visit selected internal pages in the same session. Bounded workers share one discovery set and stop when the limit is reached.

Sitemap Support

Nested and Modern Sitemaps

Standard XML sitemapindex and urlset documents are parsed incrementally. Map also recognizes common JSON URL fields such as urls, routes, loc, url, and path, resolving relative paths against the sitemap URL.

Streamed gzip content is supported, with bounded decoding for ZIP, BZ2, XZ, Zstandard when available, TAR, and TAR.GZ sitemap archives.

Healthy Exhaustion

When a healthy sitemap tree is fully traversed but contains fewer URLs than limit, Map returns the URLs it found. It does not start an expensive browser merely to fill a quota the declared tree cannot satisfy.

A failed child sitemap does not discard URLs from healthy siblings. Map can keep usable discoveries while continuing through the remaining declared branches.

Response

Job and Link Fields

Immediate, Empty, and Expired Results

Immediate result. A compatible cached map can return populated links directly from POST /v1/map without a job_id.

Empty result. A completed job with no usable links returns map_status: empty, empty_reason, and an empty links array. This is different from status: failed.

Retention. Raw Map results are available for 48 hours. After expiry, status returns an empty links array with result_expired: true, retention_hours, and expired_at. An expired export returns 410.

Performance

Early Stopping and Browser Use

Every discovery stage observes limit. Once enough unique, in-scope URLs are collected, pending sitemap and page-discovery work is stopped.

Persistent inventory and declared sitemaps are normally fastest. Rendered discovery is reserved for sites where those sources and lightweight homepage acquisition remain insufficient. Request only the number of URLs your downstream workflow will consume.

Safety

URL and Archive Boundaries

Targets, redirects, and nested sitemap locations are restricted to public HTTP and HTTPS destinations. Local, private, loopback, link-local, cloud metadata, and unsupported-protocol targets are rejected.

Redirect hops, raw bytes, decompressed bytes, archive members, recursion depth, and visited sitemap files are bounded. XML records are processed incrementally so large sitemap trees do not require an unbounded in-memory document.

Manage Jobs

Cancel a Map

Cancel pending or running work with POST /v1/jobs/{job_id}/cancel. Cancelling an already-cancelled job returns success.

curl -X POST "https://api.datablue.dev/v1/jobs/JOB_ID/cancel" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "message": "Job cancelled",
  "job_id": "550e8400-e29b-41d4-a716-446655440002"
}

Retry a Failed Map

Retry a failed or cancelled job with POST /v1/jobs/{job_id}/retry. The retry creates a new Map job with the original configuration.

curl -X POST "https://api.datablue.dev/v1/jobs/JOB_ID/retry" \
  -H "Authorization: Bearer YOUR_API_KEY"

{
  "success": true,
  "message": "Job retried successfully",
  "original_job_id": "550e8400-e29b-41d4-a716-446655440002",
  "new_job_id": "550e8400-e29b-41d4-a716-446655440003",
  "type": "map"
}

Export Results

Export a completed map as JSON or CSV through GET /v1/map/{job_id}/export. JSON contains the complete link-object array. CSV contains URL, title, description, last-modified date, and priority columns.

curl "https://api.datablue.dev/v1/map/JOB_ID/export?format=json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -o map-results.json

[
  {
    "url": "https://example.com/",
    "source": "sitemap"
  },
  {
    "url": "https://example.com/docs",
    "lastmod": "2026-07-01",
    "priority": 0.8,
    "source": "sitemap"
  }
]

Change format to csv and use a .csv output filename for spreadsheet workflows. A completed job with no exportable links returns 404.

Error Handling

One unavailable nested sitemap does not automatically discard URLs collected from healthy siblings. A top-level job failure is reported only when Map cannot complete through its permitted discovery paths.

Map vs Crawl

Use Map for URL inventory. It is the lower-cost choice for planning a crawl, finding a subset of paths, monitoring site structure, or exporting known URLs.

Use Crawl for page content. Crawl visits and scrapes pages, applies depth and path rules, and returns content from each completed page. A common workflow is Map first, then Crawl only the paths your application needs.