Understanding Crawl
The Crawl API collects content from multiple pages across a website, starting from a single URL. It automatically discovers linked pages, extracts their content, and processes them as an asynchronous crawl job.
What is Crawl?
Crawl is a website traversal and content extraction service. Starting from a single seed URL, it visits the page, discovers additional links, and extracts content from every eligible page it crawls.
Unlike extracting a single webpage, Crawl automatically continues exploring connected pages until the crawl is complete.
Good to know
Crawl combines link discovery and content extraction into a single automated workflow.
At a Glance
- π Start from a single seed URL.
- π Discover linked pages automatically.
- π Extract content from each page.
- βοΈ Process pages as an asynchronous crawl job.
- π¦ Retrieve crawl results when processing completes.
How Crawl Works
Every crawl starts from a single seed URL. As each page is processed, the crawler extracts content, discovers new links, and continues exploring eligible pages until the crawl is complete.
The crawl follows a simple cycle:
- Start from a seed URL.
- Visit and process the page.
- Extract the page content.
- Discover additional links.
- Continue crawling eligible pages.
- Finish when the crawl completes.
Crawl Workflow
From your application's perspective, every crawl follows the same lifecycle.
Once a crawl job is created, pages are processed in the background. You can then monitor the job and retrieve its results when processing finishes.
Crawl vs. Map vs. Extraction
Each Scraper API endpoint is designed for a different task.
| Feature | Extraction | Map | Crawl |
|---|---|---|---|
| Extract page content | β | β | β |
| Discover URLs | β | β | β |
| Follow links automatically | β | β | β |
| Process multiple pages | β | β | β |
When should you use each endpoint?
| Use Case | Recommended Endpoint |
|---|---|
| Extract content from one page | Extraction |
| Discover URLs on a website | Map |
| Collect content across multiple pages | Crawl |
Common Use Cases
Crawl is commonly used for:
- Documentation websites
- Knowledge bases
- Blogs and news sites
- Product catalogs
- Company websites
- AI and RAG data collection
Best Practices
To build efficient crawls:
- Start with a small crawl before scaling up.
- Crawl only the sections you need.
- Review crawl results before downstream processing.
- Configure crawl requests based on your use case.
Next Steps
Now that you understand how Crawl works, you're ready to create your first crawl job.
Continue to Your First Crawl to learn how to submit your first crawl request and understand the initial API response.