Javascript Scenario

js_scenario: Object | {};

What's This?

The js_scenario option empowers you to simulate user interactions on the target webpage. By defining a sequence of actions, you can click buttons, type into fields, scroll, and more, ensuring the page is in the desired state before scraping.

Properties

actions: An array of action objects that dictate how the Scraper interacts with the target document. Each action object can have the following properties:
- action: The kind of interaction (e.g., click, type, scroll).
- value: (Optional) Relevant value for the action, like the duration to wait.
- selector: (Optional) Specifies which HTML element the action targets.
- options: (Optional) Fine-tunes the action with additional settings.

Supported Actions

Action Name	Description	Required Properties	Optional Properties
click	Simulates a mouse click on a part of the webpage.	selector: Where to click.	delay: Time before clicking, button:'left', clickCount: Number of clicks.
wait	Pauses the Scraper for a specified time.	value: How long to wait (in milliseconds).	None.
waitFor	Waits until a specific element appears or disappears on the webpage.	Waits until a specific element appears or disappears on the webpage.	visible: Wait until the element is visible. hidden: Wait until the element is hidden. timeout: Maximum wait time.
click_and_wait_for_navigation	Clicks on an element and waits for the webpage to navigate to a new page.	selector: Where to click.	delay: Time before clicking, button: 'left', clickCount: Number of clicks
scroll_x	Scrolls horizontally within an element.	selector: Which element to scroll within. value: How long to wait (in milliseconds).	None.
scroll_y	Scrolls vertically within an element.	selector: Which element to scroll within. value: How long to wait (in milliseconds).	None.
type	Types text into a field, like filling out a form.	selector: Where to type, value: What text to type.	delay: Time between keystrokes.
scroll_on_list_and_get_items	Scrolls through a list and retrieves items from it.	scrollDelay: Time to wait between scrolls.	list_selector: Specify the list. items_selector: Items to retrieve. num_of_items: Number of items.
evaluate	Executes a JavaScript function on the webpage.	value: The JavaScript function to run.	None.

How Does it Work?

Example: js_scenario= { actions: [ {action: 'click', selector: '#submitButton'}, {action: 'wait', value: 5000} ] }, you're instructing the Scraper to visit the webpage and click on the element with the ID 'submitButton'.

When Should I Use This?

Utilize js_scenario when you need the Scraper to interact with the webpage to access the desired information. This is especially useful for pages that require user actions to display content.

Methods

addScenario.onClick(string, object?)

Clicks ֱthe target item.

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

const onClickOptions = { delay: 3000, button: 'left', clickCount: 3 };
scraper.addScenario.onClick('.your-selector', onClickOptions);
scraper.scrape('https://example.com/');

Using Configuration Object

const config = {
  js_scenario: {
    actions: [
      {
        action: 'click',
        selector: 'your-selector',
        options: {
          delay: 3000,
          button: 'left',
          clickCount: 3,
        },
      },
    ],
  },
};
scraper.scrape('https://example.com/', config);

addScenario.onWait(number)

Waits for a fixed amount of milliseconds.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.addScenario.onWait(2);
scraper.scrape('https://example.com/');

Using Configuration Object

const config = { js_scenario: { actions: [{ action: 'wait', value: 2 }] } };
scraper.scrape('https://example.com/', config);

addScenario.onWaitFor(string, object?)

Waits for a target HTML element to appear or disappear.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.addScenario.onWaitFor('.your-selector');
scraper.scrape('https://example.com/');

Using Configuration Object

const config = { js_scenario: { actions: [{ action: 'wait_for', selector: '.your-selector' }] } };
scraper.scrape('https://example.com/', config);

addScenario.onClickAndWaitForNavigation(string, object?)

Clicks on the target item and waits for the navigation triggered by the click. Only use if the click triggers a navigation.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.addScenario.onClickAndWaitForNavigation('.your-selector');
scraper.scrape('https://example.com/');

Using Configuration Object

const config = { js_scenario: { actions: [{ action: click_and_wait_for_navigation, selector: '.your-selector' }] } };
scraper.scrape('https://example.com/', config);

addScenario.onScrollY(string, number)

Scrolls within the target element along the y-axis.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.addScenario.onScrollY('.your-selector', 100);
scraper.scrape('https://example.com/');

Using Configuration Object

const config = {
  js_scenario: {
    actions: [{ action: 'scroll_y', selector: '.your-selector', value: 100 }],
  },
};
scraper.scrape('https://example.com/', config);

addScenario.onScrollX(string, object?)

Scrolls within the target element along the x-axis.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.addScenario.onScrollX('.your-selector', 100);
scraper.scrape('https://example.com/');

Using Configuration Object

const config = {
  js_scenario: {
    actions: [{ action: 'scroll_x', selector: '.your-selector', value: 100 }],
  },
};
scraper.scrape('https://example.com/', config);

addScenario.onScrollOnlListAndGetItems(object)

Scrolls within the target element along the x-axis.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

const options = {
  list_selector: '.your-selector',
  items_selector: '#items-selector',
  num_of_items: 50,
};
scraper.addScenario.onScrollOnlListAndGetItems(options);
scraper.scrape('https://example.com/');

Using Configuration Object

const config = {
  js_scenario: {
    actions: [
      {
        action: 'scroll_on_list_and_get_items',
        options: { list_selector: '.your-selector', items_selector: '#items-selector', num_of_items: 50 },
      },
    ],
  },
};
scraper.scrape('https://example.com/', config);

addScenario.onEvaluate(string)

Scrolls within the target element along the x-axis.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

scraper.addScenario.onEvaluate('7 * 8');
scraper.scrape('https://example.com/');

Using Configuration Object

const config = { js_scenario: { actions: [{ action: 'evaluate', selector: '7 * 8' }] } };
scraper.scrape('https://example.com/', config);

addScenario.onType(string, string, object?)

Scrolls within the target element along the x-axis.

Configuration Name: js_scenario

Initializing Scraper:

const GeonodeScraperApi = require('geonode-scraper-api');
const scraper = new GeonodeScraperApi('<Your_username>', '<Your_password>');

Using Method

const typingOptions = { delay: 0.5 };
scraper.addScenario.onType('.your-selector', 'example text', typingOptions);
scraper.scrape('https://example.com/');

Using Configuration Object

const typingOptions = { delay: 0.5 };
const config = {
  js_scenario: {
    actions: [
      {
        action: 'type',
        selector: '.your-selector',
        value: 'example text',
        options: typingOptions,
      },
    ],
  },
};
scraper.scrape('https://example.com/', config);

Core Concepts

Let's Scrape: Scraping Methods

Configurations

API

Javascript Scenario

What's This?

Properties

Supported Actions

How Does it Work?

When Should I Use This?

Methods

addScenario.onClick(string, object?)

Using Method

Using Configuration Object

addScenario.onWait(number)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onWaitFor(string, object?)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onClickAndWaitForNavigation(string, object?)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onScrollY(string, number)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onScrollX(string, object?)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onScrollOnlListAndGetItems(object)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onEvaluate(string)

Configuration Name: js_scenario

Using Method

Using Configuration Object

addScenario.onType(string, string, object?)

Configuration Name: js_scenario

Using Method

Using Configuration Object