Understanding Batch
Batch extraction lets you process multiple URLs in a single asynchronous request.
Instead of sending a separate request for every URL, you submit all URLs together as one batch job. Geonode processes each URL in the background and lets you retrieve the results after the job has completed.
Batch jobs are asynchronous
The Batch endpoint immediately returns a job ID instead of the extracted content. Use the job ID to monitor progress and retrieve the results when processing finishes.
When to Use Batch
Batch extraction is useful when you need to process many webpages at once.
Common use cases include:
- Extracting product pages from an e-commerce website
- Processing blog articles or news posts
- Scraping documentation pages
- Monitoring multiple websites
- Running scheduled extraction jobs
If you only need to extract content from a single webpage, use the Extraction endpoint instead.
Choosing the Right Endpoint
Geonode provides different endpoints depending on the task you want to perform.
| If you want to... | Use |
|---|---|
| Extract content from a single URL | Extraction |
| Extract content from multiple URLs | Batch |
| Crawl an entire website by following links | Crawl |
Why Use Batch?
Without batch extraction, every URL requires its own API request.
Batch extraction groups multiple URLs into a single request, making it easier to process large collections of webpages.
How Batch Works
Batch extraction follows a simple asynchronous workflow.
Unlike the Extraction endpoint, Batch does not return the extracted content immediately. Instead, it creates a job and returns a job ID that you can use to monitor progress and retrieve the results once processing has finished.
Batch Workflow
A typical batch request follows these steps:
- Submit one or more URLs to the Batch endpoint.
- Geonode creates a new batch job.
- The API immediately returns a unique job ID.
- Each URL is processed independently.
- Retrieve the completed results using the job ID.
Batch vs Extraction
Although both endpoints extract webpage content, they are designed for different workloads.
| Feature | Extraction | Batch |
|---|---|---|
| Number of URLs | One | Multiple |
| Processing | Synchronous or asynchronous | Asynchronous |
| Best for | Individual webpages | Large collections of webpages |
| Initial response | Extracted content or Job ID | Job ID |
| Final result | Single extraction | Collection of extraction results |
Key Concepts
Before using the Batch endpoint, keep the following in mind:
- Each URL is processed independently.
- Some URLs may succeed while others fail within the same batch.
- Batch requests return a job ID instead of the extracted content.
- Results can be retrieved after processing has completed.
Next Steps
Now that you understand how batch extraction works, continue to Your First Batch to submit your first batch extraction request.