Following links after pagination

How to?
#1

Hello, thank you for your efforts, this is a great extension! I am having issues trying to follow links to extract data with pagination, I am able to grab the initial links, but after adding the pagination, I get no data and the scraper quits immediately. I am trying to grab the information for the doctors(below). Thanks!

Url: https://www.beaconhealthsystem.org/find-a-doctor/find-a-doctor?type=1

Sitemap:
{"_id":"find-a-doctor","startUrl":["https://www.beaconhealthsystem.org/find-a-doctor/find-a-doctor?type=1"],"selectors":[{"id":"doctor","type":"SelectorLink","selector":"div.doctorInformation > a","parentSelectors":["_root","pagination"],"multiple":true,"delay":"1000"},{"id":"image","type":"SelectorImage","selector":"img.imgphoto","parentSelectors":["doctor"],"multiple":false,"delay":0},{"id":"about-text","type":"SelectorHTML","selector":"div.ajax__tab_panel p","parentSelectors":["doctor"],"multiple":false,"regex":"","delay":0},{"id":"request-appointment","type":"SelectorElement","selector":"input.viewProfileBtn","parentSelectors":["doctor"],"multiple":false,"delay":0},{"id":"pagination","type":"SelectorElementClick","selector":"div.doctorResultContainer","parentSelectors":["_root","pagination"],"multiple":true,"delay":"5000","clickElementSelector":"li.page-button-item a","clickType":"clickOnce","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"}]}

#2

It is not possible to scrape a site with this type of pagination. The page reloads when you click on next page, but link doesn't change so element click selector and link selector doesn't work.

#3

I've not figured out how to do this yet but I'm close. You can set it to display all records on one screen and then scrape each link

This is how I worked it - It's a work around but it gets you all 900

{"_id":"drtemp","startUrl":["https://www.beaconhealthsystem.org/find-a-doctor/find-a-doctor?type=1"],"selectors":[{"id":"Container","type":"SelectorElement","selector":"div.findADocContainer","parentSelectors":["MainLinkClick"],"multiple":false,"delay":0},{"id":"Name","type":"SelectorText","selector":"h1.doctorName","parentSelectors":["Container"],"multiple":false,"regex":"","delay":0},{"id":"Type","type":"SelectorText","selector":"span.doctorSpecialty","parentSelectors":["Container"],"multiple":false,"regex":"","delay":0},{"id":"Med Degree","type":"SelectorText","selector":"div#ctl00_cphContent_ctl01_TabContainer1_tbpnloverview_divMedicalDegree li","parentSelectors":["Container"],"multiple":false,"regex":"","delay":0},{"id":"Residency ","type":"SelectorText","selector":"div#ctl00_cphContent_ctl01_TabContainer1_tbpnloverview_divResidency li","parentSelectors":["Container"],"multiple":false,"regex":"","delay":0},{"id":"Bio","type":"SelectorText","selector":"div.ajax__tab_panel p","parentSelectors":["Container"],"multiple":false,"regex":"","delay":0},{"id":"MainPageContaned","type":"SelectorElement","selector":"div.doctorResultContainer:nth-of-type(n+2)","parentSelectors":["_root"],"multiple":true,"delay":0},{"id":"MainLinkClick","type":"SelectorLink","selector":"div.doctorInformation > a","parentSelectors":["MainPageContaned"],"multiple":false,"delay":0}]}

#4

It is still not possible to select "All" while scraping. Copying URL after you click "All" doesn't work either.