How to scrape all the page infinity scroll

Scraping only 1st page 12 data I need to scrape all the profile

Url: Staff Information - University of Johannesburg

Sitemap:
{"_id":"sss","startUrl":["https://www.uj.ac.za/staff-members/"],"selectors":[{"id":"pro","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":".staff-horizontal-layout a","type":"SelectorLink"},{"id":"name","multiple":false,"parentSelectors":["pro"],"regex":"","selector":"h1 a","type":"SelectorText"},{"id":"dep","multiple":false,"parentSelectors":["pro"],"regex":"","selector":".wpb_wrapper p:nth-of-type(1) a:nth-of-type(1)","type":"SelectorText"},{"id":"mail","multiple":false,"parentSelectors":["pro"],"regex":"","selector":"div.wpb_text_column:nth-of-type(2) p:nth-of-type(3)","type":"SelectorText"},{"id":"phn","multiple":false,"parentSelectors":["pro"],"regex":"","selector":"div.wpb_text_column:nth-of-type(2) p:nth-of-type(2)","type":"SelectorText"}]}

There are 3,5k of profiles, 12 of them are shown at page loading... So you need to add scroll element to reach the Bottom of the page... or....I can give you an advice, use sitemaps to get all profile URLs, it's easy ))
image

is there any way adding all the profile url (3.5 k ) at a time .

of course there is a way )))))
you can use this website to add all your links and preprocess them at first

How can I processed them? I am trying like this Pastelink.net - Publish Hyperlinks

use this URL as a start page, select all links as multiple and follow them.... easy ! ))

1 Like

wow man. Thanks what an idea :heart_eyes: