Geonode Logo light

Keep Headers

keep_headers: boolean | false;

What's This?

When the scraper requests data from a website, it sends along "headers" – bits of information about the request. The keep_headers option lets you decide if the scraper should use the original headers you provided or its default ones.

How Does it Work?

  • Default (keep_headers=false): The scraper uses its default headers when requesting data from the website.

  • With (keep_headers=true):The scraper includes the headers you provided, along with any additional ones it might add.

When Should I Use This?

Turn on keep_headers if the website you're trying to access requires specific headers to respond correctly. It ensures consistency and can meet specific requirements of the site being scraped.

Methods

useKeepHeaders(boolean) 

Configuration Name: keep_headers

Initializing Scraper

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.useKeepHeaders(true);
scraper.scrape('https://example.com/');

Using Configuration Object

const config = { keep_headers: true };
scraper.scrape('https://example.com/', config);