Understanding Map
The Map API helps you discover URLs across a website without extracting the content of each page.
Instead of downloading and processing every page, the Map API discovers URLs from a website and returns them as a structured list. This makes it useful for exploring websites, planning scraping workflows, and identifying the pages you want to process later.
How the Map API Works
The Map API starts with a single website URL and discovers additional URLs from the website.
During discovery, the Map API can collect URLs from:
- Website sitemaps.
- Links discovered in HTML pages.
Each discovered URL includes information about how it was found, such as sitemap or html.
Map vs Crawl vs Extraction
Although these APIs work together, they solve different problems.
| Feature | Map | Crawl | Extraction |
|---|---|---|---|
| Discover website URLs | ✓ | ✓ | — |
| Extract page content | — | ✓ | ✓ |
| Process a single page | — | — | ✓ |
| Process multiple pages | ✓ | ✓ | — |
| Return a list of discovered URLs | ✓ | — | — |
Use the Map API to discover pages first. Once you've identified the pages you need, use the Extraction API to extract individual pages or the Crawl API to process larger sections of the website.
Common Use Cases
The Map API is useful for:
- Building a list of URLs from a website.
- Discovering documentation, blog, or support pages.
- Preparing URLs before extraction or crawling.
- Exploring the content available on a website.
- Identifying pages for further processing or analysis.
When Should You Use the Map API?
Choose the Map API when your goal is to discover where content exists, rather than extracting the content itself.
| Goal | Recommended API |
|---|---|
| Discover available pages | Map API |
| Extract content from one page | Extraction API |
| Extract content from many connected pages | Crawl API |
Next Steps
Now that you understand what the Map API does, continue to Map Workflows to learn how to use the Map endpoint as part of a complete URL discovery workflow.