Very tricky pagination issue

How to?
#1

Hello,

I have trouble to scrap this page with pagination. I can only scrap first page. Much appreciated for your help.

https://www.sharedividends.com.au/yield-scanner/

Sitemap:
{"_id":"all-dividends","startUrl":["https://www.sharedividends.com.au/yield-scanner/"],"selectors":[{"id":"element-click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"table","multiple":true,"delay":"500","clickElementSelector":"div:nth-of-type(5) a[onclick='repage(2 )']","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"links","type":"SelectorLink","parentSelectors":["element-click"],"selector":"td:nth-of-type(1) a","multiple":true,"delay":0}]}

#2

Yes, "tricky" indeed.
The problem is:

  • pagination is JS controlled
  • a new page doesn't change the CSS, html or url of the "next page" link

Similar to Problems with js pagination where [KristapsWS] concluded "You won't be able to scrape this pagination with current web scraper selectors because the page reloads after you change page but the URL stays the same."
Now, they were able to get a workaround because of the page being aspx in the background, but this is not true here.

Eventually I used a different scraping tool which does not care about the uniqueness of the "next page" button like WS does and was able to get you your data.

You can find it here: https://pastebin.com/MyUMhjCq

#3

thanks for your reply, may I ask what do you use?