LangChain

geonode-scraper-langchain builds LangChain StructuredTool objects you can call with tool.invoke(...) or pass into an agent. Each tool returns a JSON-friendly dict.

Responses below are from live runs against https://scraper.geonode.io.

Setup

pip install geonode-scraper-langchain python-dotenv

Create a .env file:

GEONODE_SCRAPER_API_KEY=your_geonode_key
SCRAPER_API_BASE_URL=https://scraper.geonode.io

import os
from dotenv import load_dotenv
from geonode_scraper_langchain import build_langchain_tools
from geonode_scraper_tools_core import ScraperToolSettings

load_dotenv()

settings = ScraperToolSettings(
    host=os.environ.get("SCRAPER_API_BASE_URL", "https://scraper.geonode.io"),
    api_key=os.environ["GEONODE_SCRAPER_API_KEY"],
)

tools = build_langchain_tools(settings=settings)
by_name = {tool.name: tool for tool in tools}

Host

Use https://scraper.geonode.io for production.

Response shape

Every tool returns a dict like:

{
    "ok": True,
    "operation": "extract",
    "attempts": 1,
    "result": { ... },
}

Read the payload from response["result"].

Tools overview

Group	Tool names
Extraction	`scraper_extract_content`, `scraper_get_job_result`, `scraper_wait_for_job`, `scraper_list_jobs`
Batch	`scraper_create_batch`, `scraper_get_batch_status`, `scraper_wait_for_batch`, `scraper_list_batch_jobs`
Crawl	`scraper_create_crawl`, `scraper_get_crawl_status`, `scraper_wait_for_crawl`, `scraper_list_crawl_jobs`
Map	`scraper_map_urls`, `scraper_list_map_jobs`, `scraper_get_map_job`
Search	`scraper_search`, `scraper_list_search_jobs`, `scraper_get_search_job`
Account	`scraper_get_statistics`, `scraper_get_concurrency_usage`, `scraper_check_health`

Extraction

scraper_extract_content (sync)

response = by_name["scraper_extract_content"].invoke(
    {
        "url": "https://docs.geonode.com/docs/scraper-api/quick-start",
        "formats": ["markdown"],
        "processing_mode": "sync",
    }
)

result = response["result"]
markdown = (result.get("data") or {}).get("markdown") or ""
print("ok:", response["ok"])
print("tokens:", result.get("tokens_charged"))
print("markdown_len:", len(markdown))
print("preview:", markdown[:200])

Response (live run):

ok: True
tokens: 1
markdown_len: 15871
preview: ---
canonical: https://docs.geonode.com/docs/scraper-api/quick-start
meta-description: Get your API key, authenticate requests, and choose the right API for your use case.
...

scraper_extract_content (async)

response = by_name["scraper_extract_content"].invoke(
    {
        "url": "https://docs.geonode.com/docs/scraper-api/quick-start",
        "formats": ["markdown"],
        "processing_mode": "async",
    }
)

result = response["result"]
print("job_id:", result["job_id"])
print("status:", result["status"])

Response (live run):

job_id: 2ac66637-905c-4892-a6b9-5ace58b2ffe9
status: queued

scraper_wait_for_job

response = by_name["scraper_wait_for_job"].invoke(
    {
        "job_id": "2ac66637-905c-4892-a6b9-5ace58b2ffe9",
        "timeout_seconds": 120,
    }
)

result = response["result"]
markdown = (result.get("data") or {}).get("markdown") or ""
print("status:", result["status"])
print("markdown_len:", len(markdown))
print("poll_attempts:", response.get("poll_attempts"))

Response (live run):

status: completed
markdown_len: 15871
poll_attempts: 3

scraper_get_job_result

response = by_name["scraper_get_job_result"].invoke(
    {"job_id": "2ac66637-905c-4892-a6b9-5ace58b2ffe9"}
)

result = response["result"]
print("status:", result["status"])
print("tokens:", result.get("tokens_charged"))

Response (live run):

status: completed
tokens: 1

scraper_list_jobs

response = by_name["scraper_list_jobs"].invoke({"page": 1, "page_size": 3})

result = response["result"]
print("page:", result["page"], "page_size:", result["page_size"])
for job in result.get("jobs") or []:
    print(job["job_id"], job["status"], job.get("url"))

Response (live run):

page: 1 page_size: 3
752b8599-5915-441c-bc1f-b9fb0d938f72 completed ...
0bcd424e-b5ab-4b85-9cbd-5b44f7ed433c completed http://example.com/
0dbb126c-4331-4b4b-9d55-0c380ba69ae7 completed http://example.com/

Batch

scraper_create_batch

response = by_name["scraper_create_batch"].invoke(
    {
        "urls": [
            "https://docs.geonode.com/docs/scraper-api/quick-start",
            "https://docs.geonode.com/docs/scraper-api",
        ],
        "formats": ["markdown"],
    }
)

result = response["result"]
print("job_id:", result["job_id"])
print("accepted_urls:", result["accepted_urls"])
print("status:", result["status"])

Response (live run):

job_id: 600bb35a-d6fc-4e05-be49-816b8f3ad5d7
accepted_urls: 2
status: queued

scraper_get_batch_status

response = by_name["scraper_get_batch_status"].invoke(
    {
        "job_id": "600bb35a-d6fc-4e05-be49-816b8f3ad5d7",
        "page": 1,
        "page_size": 10,
    }
)

result = response["result"]
print(result["status"], result["completed_urls"], "/", result["total_urls"])

Response (live run, mid-job):

processing 0 / 2

scraper_wait_for_batch

response = by_name["scraper_wait_for_batch"].invoke(
    {
        "job_id": "600bb35a-d6fc-4e05-be49-816b8f3ad5d7",
        "timeout_seconds": 120,
    }
)

result = response["result"]
print(result["status"], result["completed_urls"], "/", result["total_urls"])
print("poll_attempts:", response.get("poll_attempts"))

Response (live run):

completed 2 / 2
poll_attempts: 3

scraper_list_batch_jobs

response = by_name["scraper_list_batch_jobs"].invoke({"page": 1, "page_size": 3})

for job in (response["result"].get("jobs") or []):
    print(
        job["job_id"],
        job["status"],
        job["completed_urls"],
        "/",
        job["accepted_urls"],
    )

Response (live run):

600bb35a-d6fc-4e05-be49-816b8f3ad5d7 completed 2 / 2
55b11790-5c7f-4945-bc13-bd9a365a1835 completed 2 / 2
d8e92d3a-c939-4ec6-b133-3e04c6de71d1 completed 2 / 2

Crawl

scraper_create_crawl

response = by_name["scraper_create_crawl"].invoke(
    {
        "url": "https://docs.geonode.com/docs/scraper-api",
        "depth": 2,
        "limit": 3,
        "formats": ["markdown"],
        "same_domain_only": True,
    }
)

result = response["result"]
print("job_id:", result["job_id"])
print("estimated_pages:", result["estimated_pages"])
print("status:", result["status"])

Response (live run):

job_id: ab83f113-5086-46d3-a045-3a42263b5c87
estimated_pages: 3
status: queued

scraper_get_crawl_status

response = by_name["scraper_get_crawl_status"].invoke(
    {
        "job_id": "ab83f113-5086-46d3-a045-3a42263b5c87",
        "page": 1,
        "page_size": 10,
    }
)

result = response["result"]
print(
    result["status"],
    result.get("completed_pages"),
    "/",
    result.get("total_pages"),
)

Response (live run, early poll):

processing 0 / 1

scraper_wait_for_crawl

response = by_name["scraper_wait_for_crawl"].invoke(
    {
        "job_id": "ab83f113-5086-46d3-a045-3a42263b5c87",
        "timeout_seconds": 180,
    }
)

result = response["result"]
print(result["status"], result["completed_pages"], "/", result["total_pages"])

Response (live run):

completed 3 / 3

scraper_list_crawl_jobs

response = by_name["scraper_list_crawl_jobs"].invoke({"page": 1, "page_size": 2})

for job in (response["result"].get("jobs") or []):
    print(
        job["job_id"],
        job["status"],
        job["completed_pages"],
        "/",
        job["total_pages"],
    )

Response (live run):

ab83f113-5086-46d3-a045-3a42263b5c87 completed 3 / 3
9f447ced-4aa3-45a3-b9af-421f751b8cec completed 3 / 3

Map

scraper_map_urls

response = by_name["scraper_map_urls"].invoke(
    {"url": "https://docs.geonode.com/docs/scraper-api"}
)

result = response["result"]
links = result.get("links") or []
print("link_count:", result.get("links_count") or len(links))
for link in links[:5]:
    print(link.get("source"), link.get("url"))

Response (live run):

link_count: 112
sitemap https://docs.geonode.com/docs/scraper-api
sitemap https://docs.geonode.com/docs/scraper-api/quick-start
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/choosing_scraper_api_plan
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/faq
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/pricing-and-requests

scraper_list_map_jobs

response = by_name["scraper_list_map_jobs"].invoke({"page": 1, "page_size": 2})

for job in (response["result"].get("jobs") or []):
    print(job["job_id"], job["status"], job.get("links_count"), job.get("url"))

Response (live run):

3abfc672-2aff-4daf-9f68-f033978ddfbd completed 112 https://docs.geonode.com/docs/scraper-api
c7cb6e40-ddfa-416a-b255-00dfdfd717bc completed 112 https://docs.geonode.com/docs/scraper-api

scraper_get_map_job

response = by_name["scraper_get_map_job"].invoke(
    {"job_id": "3abfc672-2aff-4daf-9f68-f033978ddfbd"}
)

result = response["result"]
print("status:", result["status"])
print("link_count:", result.get("links_count"))
for link in (result.get("links") or [])[:3]:
    print(link.get("source"), link.get("url"))

Response (live run):

status: completed
link_count: 112
sitemap https://docs.geonode.com/docs/scraper-api
sitemap https://docs.geonode.com/docs/scraper-api/quick-start
sitemap https://docs.geonode.com/docs/scraper-api/additional-resources/choosing_scraper_api_plan

Search

scraper_search

response = by_name["scraper_search"].invoke({"query": "geonode scraper api"})

result = response["result"]
print("job_id:", result["job_id"])
print("hit_count:", result.get("results_count") or len(result.get("results") or []))
for hit in (result.get("results") or [])[:5]:
    print(hit["position"], hit["title"], hit["url"])

Response (live run):

job_id: f80269e1-1911-4fd4-b1a1-fc0f7bbae7f9
hit_count: 15
1 Wholesale proxies & web data infrastructure | Geonode https://geonode.com/
2 Geonode - PyPI https://pypi.org/user/Geonode/
3 pavel.s - PyPI https://pypi.org/user/pavel.s/
4 Geonode Documentation | Geonode https://docs.geonode.com/
5 GeoNode https://geonode.org/

scraper_list_search_jobs

response = by_name["scraper_list_search_jobs"].invoke({"page": 1, "page_size": 2})

for job in (response["result"].get("jobs") or []):
    print(job["job_id"], job.get("query"), job["status"], job.get("results_count"))

Response (live run):

f80269e1-1911-4fd4-b1a1-fc0f7bbae7f9 geonode scraper api completed 15
29809a61-49b6-4bf6-925b-aa32dd91e441 geonode scraper api completed 15

scraper_get_search_job

response = by_name["scraper_get_search_job"].invoke(
    {"job_id": "f80269e1-1911-4fd4-b1a1-fc0f7bbae7f9"}
)

result = response["result"]
print("status:", result["status"])
print("hit_count:", result.get("results_count"))
for hit in (result.get("results") or [])[:3]:
    print(hit["position"], hit["title"], hit["url"])

Response (live run):

status: completed
hit_count: 15
1 Wholesale proxies & web data infrastructure | Geonode https://geonode.com/
2 Geonode - PyPI https://pypi.org/user/Geonode/
3 pavel.s - PyPI https://pypi.org/user/pavel.s/

Statistics, usage, and health

scraper_get_statistics

response = by_name["scraper_get_statistics"].invoke({})

result = response["result"]
print("extraction_count:", result.get("extraction_count"))
print("success_rate:", result.get("success_rate"))

Response (live run):

extraction_count: 4392
success_rate: 0.95128

scraper_get_concurrency_usage

response = by_name["scraper_get_concurrency_usage"].invoke({})

result = response["result"]
print(
    result["work_concurrency_in_use"],
    "/",
    result["work_concurrency_limit"],
)

Response (live run):

0 / 50

scraper_check_health

response = by_name["scraper_check_health"].invoke({})

result = response["result"]
print(result.get("service"), result.get("status"), result.get("version"))

Response (live run):

Scraper API ok 0.1.0

Selecting a subset of tools

tools = build_langchain_tools(
    settings=settings,
    operations=["extract", "map_urls", "create_batch", "wait_for_batch"],
)

print([tool.name for tool in tools])

Response (live run):

['scraper_extract_content', 'scraper_map_urls', 'scraper_create_batch', 'scraper_wait_for_batch']

Use tools with an agent

Pass the tool list into your LangChain agent or bind them to a chat model. Example with bind_tools:

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini")
llm_with_tools = llm.bind_tools(tools)

# The model can request tool calls; your agent loop then runs tool.invoke(...)

Or build an agent that owns the tools (exact API depends on your LangChain version):

# Pseudocode: wire `tools` into your preferred LangChain agent helper
# agent = create_agent(model=llm, tools=tools)
# agent.invoke({"messages": [("user", "Extract markdown from https://example.com")]})

The tool calls themselves match the tool.invoke({...}) examples above.

Toolkit helper

You can also use the toolkit class:

from geonode_scraper_langchain import ScraperLangChainToolkit
from geonode_scraper_tools_core import ScraperToolSettings

settings = ScraperToolSettings(
    host="https://scraper.geonode.io",
    api_key=os.environ["GEONODE_SCRAPER_API_KEY"],
)

toolkit = ScraperLangChainToolkit.from_settings(settings)
tools = toolkit.get_tools()

For package versions and changelog, see geonode-scraper-langchain on PyPI.

LangChain

On this page