Map URLs

POST /v1/map discovers URLs under a base URL by combining sitemap parsing with HTML link extraction from the seed page. Use it when you want to find candidate URLs before deciding which pages to extract.

The map endpoint discovers URLs only. It does not extract page content, render JavaScript, or crawl every discovered page. Discovery can time out on large sites; if the request takes too long, the API returns 408.

Request

This example maps URLs from quotes.toscrape.com and filters the discovered URLs to ones containing author.

export SCRAPER_API_BASE_URL="https://scraper.geonode.io"
export GEONODE_SCRAPER_API_KEY="YOUR_API_KEY"

curl -X POST "$SCRAPER_API_BASE_URL/v1/map" \
  -H "X-Api-Key: $GEONODE_SCRAPER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://quotes.toscrape.com/",
    "search": "author",
    "include_subdomains": false,
    "ignore_query_parameters": true
  }'

The search parameter is a case-insensitive substring filter applied to discovered URLs. It does not query a search engine.

Request Body

The map request has fewer options than extraction because it only discovers URLs. Start with url, then add search if you want to narrow the discovered URLs.

Response

A successful request returns 200 with discovered links and their discovery source.

{
  "success": true,
  "links": [
    {
      "url": "https://quotes.toscrape.com/author/Albert-Einstein",
      "source": "html"
    }
  ],
  "warning": null
}

An empty links array can still be a successful map response. In that case, check warning for a non-fatal explanation.