Configuring Crawl Requests

The Crawl API provides several request parameters that let you control how websites are crawled. This guide explains each parameter and shows how to configure common crawl scenarios.

Request Body

Every crawl request requires a starting URL. All other parameters are optional.

{
  "url": "https://docs.geonode.com/"
}

Required Parameter

url

The starting URL for the crawl.

Example:

{
  "url": "https://docs.geonode.com/"
}

Crawl Depth

The depth parameter controls how many levels deep the crawler follows links from the starting URL.

Example:

{
  "url": "https://docs.geonode.com/",
  "depth": 2
}

If the requested depth exceeds your plan limit, the API returns a validation error.

{
  "code": "VALIDATION_ERROR",
  "message": "Crawl depth limit exceeded: requested 3, plan allows at most 2",
  "correlation_id": "5d2012da-9bf9-4224-978b-b4337ab7026f",
  "retryable": false
}

Page Limit

The limit parameter specifies the maximum number of pages to crawl.

Example:

{
  "url": "https://docs.geonode.com/",
  "limit": 5
}

Domain Scope

same_domain_only

Limits crawling to pages within the same domain as the starting URL.

Example:

{
  "url": "https://docs.geonode.com/",
  "same_domain_only": true
}

include_subdomains

Controls whether subdomains are included during the crawl.

Example:

{
  "url": "https://docs.geonode.com/",
  "include_subdomains": false
}

Additional Request Options

The following parameters can be combined to further customize crawl requests.

Example:

{
  "url": "https://docs.geonode.com/",
  "formats": [
    "markdown"
  ],
  "render_js": true,
  "same_domain_only": true,
  "include_subdomains": false,
  "proxy": {
    "country": "us"
  },
  "wait_config": {
    "wait_for": 3000
  }
}

For detailed information about these parameters, see the dedicated guides for output formats, JavaScript rendering, proxy configuration, and browser wait configuration.

Complete Example

The following request combines multiple crawl configuration options.

{
  "url": "https://docs.geonode.com/",
  "depth": 2,
  "limit": 5,
  "formats": [
    "markdown"
  ],
  "render_js": true,
  "same_domain_only": true,
  "include_subdomains": false,
  "proxy": {
    "country": "us"
  },
  "wait_config": {
    "wait_for": 3000
  }
}

If the request is accepted, the API returns a response similar to the following:

{
  "job_id": "7344caad-297d-4eac-99ac-198a8c8e6233",
  "url": "https://docs.geonode.com/",
  "status": "queued",
  "status_url": "/v1/crawl/7344caad-297d-4eac-99ac-198a8c8e6233",
  "estimated_pages": 5
}

Next Steps

Continue to Understanding Crawl Results to learn how crawl results are structured and how to interpret the returned data.