Does anybody have experience with Linkedin Search pagination links? Has anybody managed to automate their clicking? As far as I understand the situation, after the page is loaded there is no URL for pagination links, but the URL is generated dynamically. In other words, when you click on the pagination link then the website generates the URL.
I went through the Web Scraper pagination & tutorial and didn't have much success with implementing the right solution to the problem.
Also, I have the same problem while trying to scrape search results with Linkedin Sales Navigator.
At the moment, I am adding pagination links manually as a "Start URL" and it would be great to have it automated.
URL example of Linkedin Search: https://www.linkedin.com/search/results/people/?facetCurrentCompany=["1033"]&origin=FACETED_SEARCH&page=1
This is the Sitemap that I use for scraping Linkedin Sales Navigator:
{"_id":"lsn-fraudscope","startUrl":["https://www.linkedin.com/sales/search/people?company=FraudScope%3A16236221&companyTimeScope=CURRENT&logHistory=true&page=1&searchSessionId=Xxja1ikoR9e7ymqIy%2FkRHQ%3D%3D"],"selectors":[{"id":"profile","type":"SelectorLink","selector":"dt.result-lockup__name a.ember-view","parentSelectors":["profile-scrolldown"],"multiple":false,"delay":0},{"id":"profile-name","type":"SelectorText","selector":".profile-topcard-person-entity__name","parentSelectors":["profile"],"multiple":false,"regex":"","delay":0},{"id":"profile-position","type":"SelectorText","selector":"dt.profile-position__title","parentSelectors":["profile"],"multiple":false,"regex":"","delay":0},{"id":"profile-position-description","type":"SelectorText","selector":"dd.profile-position__description","parentSelectors":["profile"],"multiple":false,"regex":"","delay":0},{"id":"profile-position-location","type":"SelectorText","selector":"div.profile-topcard__location-data","parentSelectors":["profile"],"multiple":false,"regex":"","delay":0},{"id":"profile-position-when_profile_locked","type":"SelectorText","selector":".profile-topcard-person-entity__content dd","parentSelectors":["profile"],"multiple":false,"regex":"","delay":0},{"id":"profile-scrolldown","type":"SelectorElementScroll","selector":"li.search-results__result-item","parentSelectors":["_root"],"multiple":true,"delay":0}]}