Map Workflows

The Map API is typically the first step in a scraping workflow. It helps you discover URLs from a website so you can decide which pages to process next.

Instead of extracting page content, the Map API returns a list of discovered URLs that can be used with other Scraper API endpoints.

Typical Workflow

Most applications use the Map API as part of the following workflow.

The workflow starts with a website URL. After reviewing the discovered URLs, you can decide whether to extract specific pages, crawl larger sections of the website, or export the URL list for further processing.

Step 1 — Submit a Website

Start by sending a request to the Map endpoint.

POST /v1/map

Provide the website URL that you want to discover.

The Map API returns a response containing the discovered URLs.

Step 2 — Review the Results

The response includes a list of discovered URLs.

Each URL also indicates how it was discovered, such as:

• html
• sitemap

Review the results before deciding which pages should be processed further.

Step 3 — Choose Your Next Step

After reviewing the discovered URLs, choose the workflow that best fits your application.

Each option serves a different purpose:

Example Workflow

The following example shows how the Map API fits into a typical scraping pipeline.

Rather than processing every page, the application first discovers all available URLs, filters the pages it needs, and then extracts content only from the relevant pages.

Building a Processing Pipeline

Many applications use the Map API as the first stage of a larger workflow.

Separating URL discovery from content extraction gives you greater control over what your application processes.

Best Practices

• Start with the Map API before extracting or crawling a large website.
• Review the discovered URLs before processing them.
• Use the search parameter to narrow the discovered URLs when appropriate.
• Use the Extraction API for individual pages.
• Use the Crawl API when processing larger sections of a website.
• Export or store discovered URLs if they will be reused later.

End-to-End Workflow

Next Steps

Now that you understand how the Map API fits into a complete workflow, continue to Your First Map to create your first request and explore the request and response in detail.