Handling Crawl API Errors

The Crawl API uses standard HTTP status codes together with structured JSON error responses.

The exact response depends on the endpoint and the reason the request could not be completed.

Standard Error Response

Most Crawl endpoints return the following error format:

{
  "code": "string",
  "message": "string",
  "correlation_id": "string",
  "retryable": false
}

Fields

Field	Description
`code`	Machine-readable error code.
`message`	Human-readable description of the error.
`correlation_id`	Request correlation identifier when available.
`retryable`	Indicates whether retrying the request may succeed.

HTTP Status Codes

`POST /v1/crawl`

Status Code	Description
`202 Accepted`	Crawl job accepted successfully.
`401 Unauthorized`	Authentication failed.
`402 Payment Required`	Insufficient balance.
`422 Validation Error`	The request failed validation.
`429 Too Many Requests`	Request rate limit exceeded.
`503 Service Unavailable`	Service is temporarily unavailable.

`GET /v1/crawl/jobs`

Status Code	Description
`200 OK`	Crawl jobs retrieved successfully.
`401 Unauthorized`	Authentication failed.
`422 Validation Error`	Invalid query parameters.

`GET /v1/crawl/{job_id}`

Status Code	Description
`200 OK`	Crawl job retrieved successfully.
`401 Unauthorized`	Authentication failed.
`404 Not Found`	Crawl job was not found.
`422 Validation Error`	Invalid request parameters.

`DELETE /v1/crawl/{job_id}`

Status Code	Description
`202 Accepted`	Cancellation request accepted.
`401 Unauthorized`	Authentication failed.
`404 Not Found`	Crawl job was not found.
`409 Conflict`	The crawl job cannot be cancelled in its current state.
`422 Validation Error`	Invalid request parameters.

Validation Errors

Requests that fail validation return an HTTPValidationError response.

The response contains a detail array describing one or more validation issues.

{
  "detail": [
    {
      "loc": [
        "string"
      ],
      "msg": "string",
      "type": "string"
    }
  ]
}

Validation Fields

Field	Description
`loc`	Location of the validation error.
`msg`	Description of the validation error.
`type`	Validation error type.

Example: Invalid Job State

If a crawl job cannot be cancelled because of its current state, the API returns a 409 Conflict response.

Example:

{
  "code": "INVALID_STATE",
  "message": "Crawl job 9293f045-0523-409e-a4e9-874abc663a9e is already completed and cannot be cancelled",
  "retryable": false
}

Error Fields

Field	Description
`code`	Machine-readable error code.
`message`	Explains why the request could not be completed.
`retryable`	Indicates whether retrying the request may succeed.

The Crawl API returns structured error responses together with standard HTTP status codes. Validation errors include additional details about invalid request parameters, while endpoint-specific errors provide information about why an operation could not be completed. Reviewing both the HTTP status code and the response body can help identify and resolve issues more efficiently.