Geonode Logo light

Callback Mode

scrapeCallbackMode(url: string, callbackUrl: string, configurations?: IConfigurations)

Parameters

  • Name
    url
    Type
    string
    Description

    The URL of the webpage you want to scrape.

  • Name
    callbackUrl
    Type
    string
    Description

    The URL where you want to receive the scraped data.

  • Name
    configurations
    Type
    IConfigurations, optional
    Description

    The URL of the webpage you want to scrape.

Description

The .scrapeCallbackMode() method operates in Callback Mode. You get an immediate response with a request ID to track your scraping task. The result is sent back to your specified callback URL via a POST request.

Usage

  • Basic: Provide the URL you want to scrape and the callback URL where you want to receive the results.
scraper.scrapeCallbackMode('https://example.com/' , ‘https://exampleCallback.com/`)
  .then(res => {
    console.log('Response:', res?.data);
  })
  .catch(err => {
    console.error('Error:', err);
  });

 
Response: {
  response: { requestId: 'd836h452-a92a-63f9-4673-0e454aab73c3' },
  message: 'scrape'
}
  • Using Custom Configuration: Customize the scraping process by providing a configuration object to override default settings.
const customConfig: IConfigurations = {
  js_render: true, 
  response_format: json,
  block_resources: true, 
  device_type: 'desktop, 
  country_code: 'us',  
};
scraper.scrapeCallbackMode('https://example.com/' , ‘https://exampleCallback.com/, customConfig)
 .then(res => {
  console.log('JSON response:', res?.data);
 })
 .catch(error => {
  console.error('Error:', error.message);
 });

Response: {
  response: { requestId: 'd836h452-a92a-63f9-4673-0e454aab73c3' },
  message: 'scrape'
}
  • Using SDK Methods: Use the SDK's built-in methods to set specific options for the scraping process. These methods can be chained for multiple configurations.
scraper
 .useJsRendering(true)
 .setResponseFormat('json')
 .setBlockResources(true) 
 .setDeviceType('desktop') 
 .setCountryCode('us')


scraper.scrapeCallbackMode('https://example.com/' , ‘https://exampleCallback.com/)
 .then(res => {
  console.log('JSON response:', res?.data);
 })
 .catch(error => {
  console.error('Error:', error.message);
 });

Response: {
  response: { requestId: 'd836h452-a92a-63f9-4673-0e454aab73c3' },
  message: 'scrape'
}