Code Examples
cURL, Python, and Node.js examples for calling the Scraper API.
These examples call POST /v1/extract in synchronous mode and print the extracted Markdown. Before running them, set your Scraper API base URL and API key.
export SCRAPER_API_BASE_URL="https://scraper.geonode.io"
export GEONODE_SCRAPER_API_KEY="YOUR_API_KEY"cURL
Use this example when you want to test the API from a terminal before writing application code.
curl -X POST "$SCRAPER_API_BASE_URL/v1/extract" \
-H "X-Api-Key: $GEONODE_SCRAPER_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"render_js": false,
"processing_mode": "sync"
}'Python
This example uses Python's standard library so you do not have to install any extra packages.
import json
import os
from urllib import request, error
base_url = os.environ["SCRAPER_API_BASE_URL"]
api_key = os.environ["GEONODE_SCRAPER_API_KEY"]
payload = {
"url": "https://example.com",
"formats": ["markdown"],
"render_js": False,
"processing_mode": "sync",
}
req = request.Request(
f"{base_url}/v1/extract",
data=json.dumps(payload).encode("utf-8"),
headers={
"X-Api-Key": api_key,
"Content-Type": "application/json",
},
method="POST",
)
try:
with request.urlopen(req, timeout=60) as response:
result = json.loads(response.read().decode("utf-8"))
print(result["data"]["markdown"])
except error.HTTPError as exc:
body = exc.read().decode("utf-8")
print(f"Request failed with HTTP {exc.code}: {body}")Node.js
This example uses the built-in fetch API available in current Node.js versions.
const baseUrl = process.env.SCRAPER_API_BASE_URL;
const apiKey = process.env.GEONODE_SCRAPER_API_KEY;
const response = await fetch(`${baseUrl}/v1/extract`, {
method: "POST",
headers: {
"X-Api-Key": apiKey,
"Content-Type": "application/json",
},
body: JSON.stringify({
url: "https://example.com",
formats: ["markdown"],
render_js: false,
processing_mode: "sync",
}),
});
const result = await response.json();
if (!response.ok) {
throw new Error(`Request failed with HTTP ${response.status}: ${JSON.stringify(result)}`);
}
console.log(result.data.markdown);