Overview
Crawl API
Discover and extract linked webpages by starting with a website URL.
At a glance
From request to clean content
Submit a request, let Geonode handle proxies and rendering, and get structured output for your apps and pipelines.
Capabilities
What is Crawl?
Crawl starts from a URL and follows links across a site.
It discovers pages and extracts content in one asynchronous job.
You can control depth, limits, and domain scope.
Fit
When to use it
Documentation websites
Multi-page website extraction
Website content collection
Workflows where you do not already know every URL
Architecture
How it works
A typical crawl request follows these steps.
Provide a starting URL.
Configure depth, limits, and domain options.
Start the crawl and receive a job ID.
Crawl processes linked pages.
Review status and retrieve results.
Documentation
Explore Crawl guides
Jump into setup, workflows, jobs, and troubleshooting.
Was this page helpful?