Managing Crawl Jobs

Crawl jobs are processed asynchronously. After creating a crawl job, you can retrieve its status, monitor its progress, or list previous crawl jobs.

This guide explains how to work with crawl jobs throughout their lifecycle.

List Crawl Jobs

Retrieve a list of your crawl jobs.

GET /v1/crawl/jobs

The response includes your crawl jobs along with their current status, crawl statistics, configuration, and pagination information.

Example:

{
  "jobs": [
    {
      "job_id": "453bd7d7-5355-4d6d-a38e-d9e7eb218c3f",
      "url": "string",
      "status": "queued",
      "total_pages": 0,
      "completed_pages": 0,
      "failed_pages": 0,
      "config": {
        "render_js": true,
        "formats": [
          "markdown"
        ],
        "same_domain_only": true,
        "include_subdomains": true,
        "proxy": {
          "country": "string",
          "type": "datacenter"
        },
        "wait_config": {
          "wait_until": "commit",
          "wait_for": "string",
          "wait_timeout": 30000
        }
      },
      "created_at": "2019-08-24T14:15:22Z",
      "completed_at": "2019-08-24T14:15:22Z"
    }
  ],
  "page": 0,
  "page_size": 0,
  "page_count": 0
}

Retrieve a Crawl Job

Retrieve the latest status and results for a specific crawl job.

GET /v1/crawl/{job_id}

Once the crawl completes, this endpoint also returns the extracted page results.

Job Status

Each crawl job includes a status field that indicates its current state.

Status	Description
`queued`	The crawl job has been accepted and is waiting to start.
`running`	The crawler is actively processing pages.
`completed`	The crawl has finished and the results are available.

Example:

{
  "job_id": "453bd7d7-5355-4d6d-a38e-d9e7eb218c3f",
  "status": "queued"
}

Monitor Crawl Progress

Each job includes progress information that can be used to track the crawl.

Field	Description
`total_pages`	Total number of pages included in the crawl.
`completed_pages`	Number of pages successfully processed.
`failed_pages`	Number of pages that failed during crawling.

Example:

{
  "total_pages": 20,
  "completed_pages": 15,
  "failed_pages": 1
}

Retrieve the job periodically to monitor its progress until the status becomes completed.

Crawl Configuration

Each listed job includes the configuration used when the crawl was created.

Field	Description
`render_js`	Indicates whether JavaScript rendering was enabled.
`formats`	Requested output formats.
`same_domain_only`	Indicates whether crawling was limited to the starting domain.
`include_subdomains`	Indicates whether subdomains were included.
`proxy`	Proxy configuration used for the crawl.
`wait_config`	Browser wait configuration used during extraction.

Pagination

The response includes pagination information for the job list.

Field	Description
`page`	Current page number.
`page_size`	Number of jobs returned per page.
`page_count`	Total number of available pages.

Example:

{
  "page": 0,
  "page_size": 10,
  "page_count": 3
}

Typical Workflow

Next Steps

If you no longer need an active crawl job, continue to Cancelling Crawl Jobs to learn how to stop a running crawl.

Managing Crawl Jobs

On this page