Your First Crawl

In this guide, you'll create your first crawl job using the Crawl API.

The Crawl API accepts a starting URL and returns a crawl job that runs asynchronously. Once the job is created, you can monitor its progress and retrieve the results using the job ID.

Before You Begin

Before creating a crawl job, make sure you have:

A GeoNode API key.
The Scraper API base URL.

If you haven't completed the initial setup, see Before You Start.

Create a Crawl Job

Create a crawl by sending a POST request to the following endpoint.

POST /v1/crawl

For your first crawl, you only need a starting URL and can optionally specify the output format and page limit.

Example Request

curl --request POST \
  --url https://api.geonode.com/v1/crawl \
  --header "Authorization: Bearer <YOUR_API_KEY>" \
  --header "Content-Type: application/json" \
  --data '{
    "url": "https://docs.geonode.com/",
    "formats": [
      "markdown"
    ],
    "limit": 5
  }'

You can also send the request body as JSON.

{
  "url": "https://docs.geonode.com/",
  "formats": [
    "markdown"
  ],
  "limit": 5
}

Example Response

If the request is accepted, the API returns a 202 Accepted response similar to the following.

{
  "job_id": "8c8f8ad4-a0a0-46f8-92d5-253025c6e19f",
  "url": "https://docs.geonode.com/",
  "status": "queued",
  "status_url": "/v1/crawl/8c8f8ad4-a0a0-46f8-92d5-253025c6e19f",
  "estimated_pages": 5
}

Understanding the Response

Field	Description
`job_id`	Unique identifier for the crawl job.
`url`	The seed URL used to start the crawl.
`status`	Current status of the crawl job.
`status_url`	Endpoint for checking the crawl job status.
`estimated_pages`	Estimated number of pages to be processed.

The Crawl API processes requests asynchronously. A successful request creates a crawl job rather than returning the crawl results immediately.

What Happens Next?

After your crawl job is created, you can use the returned job ID to monitor its progress.

The next guide explains how to configure crawl requests with additional options, while Managing Crawl Jobs covers how to monitor job progress and retrieve results.

Continue to Configuring Crawl Requests to learn about all available request options, including crawl depth, page limits, output formats, JavaScript rendering, proxy settings, and browser wait configuration.