Understanding Crawl Results

After creating a crawl job, use the job ID to retrieve its current status and results.

The response includes information about the crawl job, the configuration used, crawl statistics, and the extracted content for each crawled page.

Get Crawl Results

Retrieve the current status and results of a crawl job.

GET /v1/crawl/{job_id}

Once the crawl completes, the response includes the extracted pages and their associated metadata.

Job Information

The top-level response provides general information about the crawl job.

Field	Description
`job_id`	Unique identifier for the crawl job.
`url`	Starting URL used for the crawl.
`status`	Current status of the crawl job.
`created_at`	Time the crawl job was created.
`completed_at`	Time the crawl job completed.

Example:

{
  "job_id": "9293f045-0523-409e-a4e9-874abc663a9e",
  "url": "https://docs.geonode.com/",
  "status": "completed",
  "created_at": "2026-07-23T16:55:51.603677Z",
  "completed_at": "2026-07-23T16:56:06.735093Z"
}

Crawl Configuration

The crawl_config object shows the configuration that was used when the crawl job was created.

Field	Description
`render_js`	Indicates whether JavaScript rendering was enabled.
`formats`	Output formats requested for extracted content.
`same_domain_only`	Indicates whether crawling was limited to the starting domain.
`include_subdomains`	Indicates whether subdomains were included.
`proxy`	Proxy configuration used during the crawl.
`wait_config`	Browser wait configuration, if specified.

Example:

{
  "crawl_config": {
    "render_js": false,
    "formats": [
      "markdown"
    ],
    "same_domain_only": true,
    "include_subdomains": false,
    "proxy": {
      "country": null,
      "type": "residential"
    },
    "wait_config": null
  }
}

The returned configuration reflects the settings used for the crawl job.

Crawl Statistics

The response includes statistics that summarize the crawl.

Field	Description
`total_pages`	Total number of pages included in the crawl.
`completed_pages`	Number of pages successfully processed.
`failed_pages`	Number of pages that failed to process.
`cancelled_pages`	Number of pages that were cancelled.

Example:

{
  "total_pages": 5,
  "completed_pages": 5,
  "failed_pages": 0,
  "cancelled_pages": 0
}

Token Summary

The token_summary object provides information about token usage for the crawl job.

Field	Description
`tokens_charged_total`	Total tokens charged for the crawl.
`tokens_reserved`	Tokens reserved for the crawl job.

Example:

{
  "token_summary": {
    "tokens_charged_total": 5,
    "tokens_reserved": 0
  }
}

Understanding the Results Array

The results array contains one object for each crawled page.

Page Information

Each object in the results array describes a single crawled page.

Field	Description
`url`	URL of the crawled page.
`parent_url`	URL where this page was discovered.
`depth`	Crawl depth of the page.
`status`	Crawl status for the page.
`error_code`	Error code if the page failed.
`error_message`	Error details if the page failed.

Example:

{
  "url": "https://docs.geonode.com/",
  "parent_url": null,
  "depth": 0,
  "status": "completed",
  "error_code": null,
  "error_message": null
}

Extracted Data

The data object contains the extracted content for the page.

Depending on the requested output formats, it may include Markdown, HTML, or both.

Example:

{
  "data": {
    "markdown": "# Welcome to Geonode\n\nFind exactly what you need...",
    "html": null
  }
}

Page Metadata

Each page includes metadata describing the crawl operation.

Field	Description
`http_status`	HTTP response status received for the page.
`duration_ms`	Time taken to process the page.
`tokens_charged`	Tokens charged for processing the page.

Example:

{
  "metadata": {
    "http_status": 200,
    "duration_ms": 957,
    "tokens_charged": 1
  }
}

Discovered Links

The links array contains the URLs discovered on the crawled page.

Example:

{
  "links": [
    "https://docs.geonode.com/",
    "https://docs.geonode.com/docs/api-reference",
    "https://docs.geonode.com/docs/guides",
    "https://docs.geonode.com/docs/scraper-api",
    "https://docs.geonode.com/docs/changelog",
    "https://geonode.com/contact"
  ]
}

Complete Response Example

The following example shows a completed crawl job response.

{
  "job_id": "9293f045-0523-409e-a4e9-874abc663a9e",
  "url": "https://docs.geonode.com/",
  "status": "completed",
  "crawl_config": {
    "render_js": false,
    "formats": [
      "markdown"
    ],
    "same_domain_only": true,
    "include_subdomains": false,
    "proxy": {
      "country": null,
      "type": "residential"
    },
    "wait_config": null
  },
  "token_summary": {
    "tokens_charged_total": 5,
    "tokens_reserved": 0
  },
  "total_pages": 5,
  "completed_pages": 5,
  "failed_pages": 0,
  "cancelled_pages": 0,
  "created_at": "2026-07-23T16:55:51.603677Z",
  "completed_at": "2026-07-23T16:56:06.735093Z",
  "results": [
    {
      "url": "https://docs.geonode.com/",
      "parent_url": null,
      "depth": 0,
      "status": "completed"
    }
  ]
}

Next Steps

Now that you understand the structure of crawl results, continue to Managing Crawl Jobs to learn how to list crawl jobs, monitor their progress, and retrieve specific jobs.

Understanding Crawl Results

On this page