Been struggling with pagination. With different settings I've tried, the window will either close after paginating (only scraping one page of data), or with other settings it will paginate properly but not scrape anything.
The data is loaded dynamically after clicking the next button (replacing the data on the page), and there's nothing unique about the next button going from page to page. There's also a long delay before the new page of data is loaded, so I've been setting the page load delay to 40000. Tried to adapt a few different site maps from related issues here, but have not been able to get this to work properly. Seems to be close, but not quite there.
Unfortunately this site is only accessible with login info, but here is the sitemap
Sitemap:
{"_id":"mysiteurl","startUrl":["http://www.notrealsite.com"],"selectors":[{"id":"Paginator","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.pageArea","multiple":false,"delay":"3500","clickElementSelector":"div.menuPagerBar:nth-of-type(1)","clickType":"clickMore","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Row wrappers","type":"SelectorElement","parentSelectors":["Paginator"],"selector":"tr","multiple":true,"delay":0},{"id":"company","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"td:eq(1)","multiple":false,"regex":"","delay":0},{"id":"name","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"td:eq(2)","multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"td:eq(3)","multiple":false,"regex":"","delay":0},{"id":"location","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"td:eq(4)","multiple":false,"regex":"","delay":0},{"id":"zipcode","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"td:eq(5)","multiple":false,"regex":"","delay":0},{"id":"phone","type":"SelectorText","parentSelectors":["Row wrappers"],"selector":"td:eq(6)","multiple":false,"regex":"","delay":0}]}
Also, if I change the Paginator selector from "div.menuPagerBar:nth-of-type(1)" to ".next:eq(0)", the pagination seems more reliable, however no data is scraped when doing this. And if I leave the delay at 3500, it goes through the pages pretty fast as expected. I did try raising that to be longer than the page loan time of 40000 that I'm using, but that did not fix the non-data gathering problem.