Tools Core

geonode-scraper-tools-core gives you named scraper operations and JSON-friendly dictionaries. Call them via ScraperToolService, or select a subset with get_operations(). LangChain and CrewAI packages use this layer under the hood.

Responses below are from live runs against https://scraper.geonode.io.

Setup

pip install geonode-scraper-tools-core python-dotenv

Create a .env file:

GEONODE_SCRAPER_API_KEY=your_geonode_key
SCRAPER_API_BASE_URL=https://scraper.geonode.io

import os
from dotenv import load_dotenv
from geonode_scraper_tools_core import ScraperToolSettings, ScraperToolService

load_dotenv()

settings = ScraperToolSettings(
    host=os.environ.get("SCRAPER_API_BASE_URL", "https://scraper.geonode.io"),
    api_key=os.environ["GEONODE_SCRAPER_API_KEY"],
)
service = ScraperToolService(settings)

Host

Use https://scraper.geonode.io for production.

ScraperToolSettings fields

Field	Type	Notes
`host`	`str`	Required. API base URL.
`api_key`	`str`	Required. Geonode Scraper API key.
`verify_ssl`	`bool`	Default `True`.
`request_timeout`	timeout	Optional HTTP timeout.
`max_retries`	`int`	Default `0`.
`retry_backoff_seconds`	`float`	Default `1.0`.
`poll_interval_seconds`	`float`	Default `3.0` (used by wait helpers).
`poll_timeout_seconds`	`float`	Default `60.0` (used by wait helpers).

Response shape

Every service method returns a dict like:

{
    "ok": True,
    "operation": "extract",
    "attempts": 1,
    "result": { ... },  # real payload
}

Wait helpers also include poll_attempts. Read the payload from response["result"].

Operations overview

Group	Operations
Extraction	`extract`, `get_job_result`, `wait_for_job`, `list_jobs`
Batch	`create_batch`, `get_batch_status`, `wait_for_batch`, `list_batch_jobs`
Crawl	`create_crawl`, `get_crawl_status`, `wait_for_crawl`, `list_crawl_jobs`
Map	`map_urls`, `list_map_jobs`, `get_map_job`
Search	`search`, `list_search_jobs`, `get_search_job`
Account	`get_statistics`, `get_concurrency_usage`, `health_check`

wait_for_job, wait_for_batch, and wait_for_crawl poll until a job finishes, so you do not write poll loops yourself.

Extraction

Operations: extract, get_job_result, wait_for_job, list_jobs

extract (sync)

Scrape one URL and wait for content in the same call.

Main inputs: url, formats (markdown / html), processing_mode (sync / async), render_js, proxy_country, proxy_type, headers, wait_for, wait_timeout, wait_until

response = service.extract(
    url="https://docs.geonode.com/docs/scraper-api/quick-start",
    formats=["markdown"],
    processing_mode="sync",
)

result = response["result"]
markdown = (result.get("data") or {}).get("markdown") or ""
print("ok:", response["ok"])
print("tokens:", result.get("tokens_charged"))
print("markdown_len:", len(markdown))
print("preview:", markdown[:200])

Response (live run):

ok: True
tokens: 1
markdown_len: 15871
preview: ---
canonical: https://docs.geonode.com/docs/scraper-api/quick-start
meta-description: Get your API key, authenticate requests, and choose the right API for your use case.
...

extract (async)

Same method with processing_mode="async". Returns a job_id immediately.

response = service.extract(
    url="https://docs.geonode.com/docs/scraper-api/quick-start",
    formats=["markdown"],
    processing_mode="async",
)

result = response["result"]
print("job_id:", result["job_id"])
print("status:", result["status"])

Response (live run):

job_id: 8a2bbbd5-7034-412b-acdf-e30226ae32b6
status: queued

wait_for_job

Poll an async extract job until it finishes (or times out).

Inputs: job_id, optional timeout_seconds, poll_interval_seconds

response = service.wait_for_job(
    job_id="8a2bbbd5-7034-412b-acdf-e30226ae32b6",
    timeout_seconds=120,
)

result = response["result"]
markdown = (result.get("data") or {}).get("markdown") or ""
print("status:", result["status"])
print("markdown_len:", len(markdown))
print("poll_attempts:", response.get("poll_attempts"))

Response (live run):

status: completed
markdown_len: 15871
poll_attempts: 2

get_job_result

Fetch the current state or final result for one extract job (no waiting).

Inputs: job_id

response = service.get_job_result(
    job_id="8a2bbbd5-7034-412b-acdf-e30226ae32b6",
)

result = response["result"]
print("status:", result["status"])
print("tokens:", result.get("tokens_charged"))

Response (live run):

status: completed
tokens: 1

list_jobs

List past extract jobs.

Inputs: optional job_id, url, status, output, start_date, end_date, page, page_size

response = service.list_jobs(page=1, page_size=3)

result = response["result"]
print("page:", result["page"], "page_size:", result["page_size"])
for job in result.get("jobs") or []:
    print(job["job_id"], job["status"], job.get("url"))

Response (live run):

page: 1 page_size: 3
752b8599-5915-441c-bc1f-b9fb0d938f72 completed ...
0bcd424e-b5ab-4b85-9cbd-5b44f7ed433c completed http://example.com/
0dbb126c-4331-4b4b-9d55-0c380ba69ae7 completed http://example.com/

Batch

Operations: create_batch, get_batch_status, wait_for_batch, list_batch_jobs

create_batch

Submit many URLs as one job.

Inputs: urls, formats, optional render_js, proxy_country, proxy_type, headers

response = service.create_batch(
    urls=[
        "https://docs.geonode.com/docs/scraper-api/quick-start",
        "https://docs.geonode.com/docs/scraper-api",
    ],
    formats=["markdown"],
)

result = response["result"]
print("job_id:", result["job_id"])
print("accepted_urls:", result["accepted_urls"])
print("status:", result["status"])

Response (live run):

job_id: 55b11790-5c7f-4945-bc13-bd9a365a1835
accepted_urls: 2
status: queued

get_batch_status

Poll progress and partial results (paginated).

Inputs: job_id, page, page_size

response = service.get_batch_status(
    job_id="55b11790-5c7f-4945-bc13-bd9a365a1835",
    page=1,
    page_size=10,
)

result = response["result"]
print(
    result["status"],
    result["completed_urls"],
    "/",
    result["total_urls"],
)

Response (live run, mid-job):

processing 0 / 2

wait_for_batch

Poll until the batch finishes.

Inputs: job_id, optional timeout_seconds, poll_interval_seconds

response = service.wait_for_batch(
    job_id="55b11790-5c7f-4945-bc13-bd9a365a1835",
    timeout_seconds=120,
)

result = response["result"]
print(
    result["status"],
    result["completed_urls"],
    "/",
    result["total_urls"],
)
print("poll_attempts:", response.get("poll_attempts"))

Response (live run):

completed 2 / 2
poll_attempts: 3

list_batch_jobs

Inputs: optional status, start_date, end_date, page, page_size

response = service.list_batch_jobs(page=1, page_size=3)

for job in (response["result"].get("jobs") or []):
    print(
        job["job_id"],
        job["status"],
        job["completed_urls"],
        "/",
        job["accepted_urls"],
    )

Response (live run):

55b11790-5c7f-4945-bc13-bd9a365a1835 completed 2 / 2
d8e92d3a-c939-4ec6-b133-3e04c6de71d1 completed 2 / 2
883bd4ad-cb9c-49bc-99f5-d007fce5a217 completed 0 / 1

Crawl

Operations: create_crawl, get_crawl_status, wait_for_crawl, list_crawl_jobs

create_crawl

Start from a seed URL.

Inputs: url, depth, limit, formats, same_domain_only, include_subdomains, optional render_js, proxy_country, proxy_type

response = service.create_crawl(
    url="https://docs.geonode.com/docs/scraper-api",
    depth=2,
    limit=3,
    formats=["markdown"],
    same_domain_only=True,
)

result = response["result"]
print("job_id:", result["job_id"])
print("estimated_pages:", result["estimated_pages"])
print("status:", result["status"])

Response (live run):

job_id: 9f447ced-4aa3-45a3-b9af-421f751b8cec
estimated_pages: 3
status: queued

get_crawl_status

Inputs: job_id, page, page_size

response = service.get_crawl_status(
    job_id="9f447ced-4aa3-45a3-b9af-421f751b8cec",
    page=1,
    page_size=10,
)

result = response["result"]
print(
    result["status"],
    result.get("completed_pages"),
    "/",
    result.get("total_pages"),
)

Response (live run, early poll):

processing 0 / 1

wait_for_crawl

Inputs: job_id, optional timeout_seconds, poll_interval_seconds

response = service.wait_for_crawl(
    job_id="9f447ced-4aa3-45a3-b9af-421f751b8cec",
    timeout_seconds=180,
)

result = response["result"]
print(
    result["status"],
    result["completed_pages"],
    "/",
    result["total_pages"],
)
print("poll_attempts:", response.get("poll_attempts"))

Response (live run):

completed 3 / 3
poll_attempts: 19

list_crawl_jobs

Inputs: optional url, status, start_date, end_date, page, page_size

response = service.list_crawl_jobs(page=1, page_size=2)

for job in (response["result"].get("jobs") or []):
    print(
        job["job_id"],
        job["status"],
        job["completed_pages"],
        "/",
        job["total_pages"],
    )

Response (live run):

9f447ced-4aa3-45a3-b9af-421f751b8cec completed 3 / 3
d8e5dec2-7450-4023-8fba-858243ac5339 completed 0 / 1

Map

Operations: map_urls, list_map_jobs, get_map_job

map_urls

Discover URLs under a base URL (sitemap + HTML links). Does not scrape page content.

Inputs: url, optional search, include_subdomains, ignore_query_parameters

response = service.map_urls(
    url="https://docs.geonode.com/docs/scraper-api",
)

result = response["result"]
links = result.get("links") or []
print("link_count:", result.get("links_count") or len(links))
for link in links[:5]:
    print(link.get("source"), link.get("url"))

Response (live run):

link_count: 112
sitemap https://docs.geonode.com/docs/scraper-api
sitemap https://docs.geonode.com/docs/scraper-api/quick-start
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/choosing_scraper_api_plan
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/faq
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/pricing-and-requests

list_map_jobs

Inputs: optional url, status, start_date, end_date, page, page_size

response = service.list_map_jobs(page=1, page_size=2)

for job in (response["result"].get("jobs") or []):
    print(job["job_id"], job["status"], job.get("links_count"), job.get("url"))

Response (live run):

c7cb6e40-ddfa-416a-b255-00dfdfd717bc completed 112 https://docs.geonode.com/docs/scraper-api
b38bdc1c-2b53-4ccb-8947-6673cf5b53e0 completed 112 https://docs.geonode.com/docs/scraper-api

get_map_job

Inputs: job_id

response = service.get_map_job(
    job_id="c7cb6e40-ddfa-416a-b255-00dfdfd717bc",
)

result = response["result"]
print("status:", result["status"])
print("link_count:", result.get("links_count"))
for link in (result.get("links") or [])[:3]:
    print(link.get("source"), link.get("url"))

Response (live run):

status: completed
link_count: 112
sitemap https://docs.geonode.com/docs/scraper-api
sitemap https://docs.geonode.com/docs/scraper-api/quick-start
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/choosing_scraper_api_plan

Search

Operations: search, list_search_jobs, get_search_job

search

Run a web search and get ranked hits (also returns a job_id).

Inputs: query, optional locale, page, safe (off / moderate / strict), time_range (day / week / month / year)

response = service.search(query="geonode scraper api")

result = response["result"]
print("job_id:", result["job_id"])
print("hit_count:", result.get("results_count") or len(result.get("results") or []))
for hit in (result.get("results") or [])[:5]:
    print(hit["position"], hit["title"], hit["url"])

Response (live run):

job_id: 29809a61-49b6-4bf6-925b-aa32dd91e441
hit_count: 15
1 Wholesale proxies & web data infrastructure | Geonode https://geonode.com/
2 Geonode - PyPI https://pypi.org/user/Geonode/
3 pavel.s - PyPI https://pypi.org/user/pavel.s/
4 Geonode Documentation | Geonode https://docs.geonode.com/
5 GeoNode https://geonode.org/

list_search_jobs

Inputs: optional query, status, start_date, end_date, page, page_size

response = service.list_search_jobs(page=1, page_size=2)

for job in (response["result"].get("jobs") or []):
    print(job["job_id"], job.get("query"), job["status"], job.get("results_count"))

Response (live run):

29809a61-49b6-4bf6-925b-aa32dd91e441 geonode scraper api completed 15
6aef3e93-f714-4eb5-ab7a-c0b00fee5dea geonode scraper api completed 15

get_search_job

Inputs: job_id

response = service.get_search_job(
    job_id="29809a61-49b6-4bf6-925b-aa32dd91e441",
)

result = response["result"]
print("status:", result["status"])
print("hit_count:", result.get("results_count"))
for hit in (result.get("results") or [])[:3]:
    print(hit["position"], hit["title"], hit["url"])

Response (live run):

status: completed
hit_count: 15
1 Wholesale proxies & web data infrastructure | Geonode https://geonode.com/
2 Geonode - PyPI https://pypi.org/user/Geonode/
3 pavel.s - PyPI https://pypi.org/user/pavel.s/

Statistics, usage, and health

Operations: get_statistics, get_concurrency_usage, health_check

get_statistics

Inputs: optional start_date, end_date

response = service.get_statistics()

result = response["result"]
print("extraction_count:", result.get("extraction_count"))
print("success_rate:", result.get("success_rate"))

Response (live run):

extraction_count: 4383
success_rate: 0.95117

get_concurrency_usage

No inputs. Checks live concurrency slots vs your plan limit.

response = service.get_concurrency_usage()

result = response["result"]
print(
    result["work_concurrency_in_use"],
    "/",
    result["work_concurrency_limit"],
)

Response (live run):

0 / 50

health_check

No inputs. Confirms the Scraper API is up.

response = service.health_check()

result = response["result"]
print(result.get("service"), result.get("status"), result.get("version"))

Response (live run):

Scraper API ok 0.1.0

Selecting a subset of operations

Use get_operations() when you only want some tools (for example before wiring into an agent framework).

from geonode_scraper_tools_core import get_operations

ops = get_operations(["extract", "map_urls", "create_batch", "wait_for_batch"])
print([op.key for op in ops])

Response:

['extract', 'map_urls', 'create_batch', 'wait_for_batch']

OPERATIONS is the full registry (all 21). Each entry has key, tool_name, description, args_schema, and service_method.

Used by LangChain and CrewAI

Most agent users install:

geonode-scraper-langchain
geonode-scraper-crewai

Those packages wrap this core service. Use tools-core directly when you want named operations and plain dicts in your own Python code, without an agent framework.

For package versions and changelog, see geonode-scraper-tools-core on PyPI.

Tools Core

On this page