Problems with js pagination

How to?
#1

I'm trying to get the details from each linked item on each page of a paginated list, and I am having trouble - getting the first page was easy, but when trying to add pagination in I am now just getting the second page - not the first or any other.

The pagination is via javascript so I am using the element click selector - the items themselves are links though, so I am using the link selector for those - not sure if this presents a problem.

Url: https://www.fdrr.ag.gov.au/Search.aspx?search_String=&locality=&postcode=&state=&PageIndex=0&SearchAll=true

Sitemap:
{"_id":"fdrr2","startUrl":["https://www.fdrr.ag.gov.au/Search.aspx?search_String=&locality=&postcode=&state=&PageIndex=0&SearchAll=true"],"selectors":[{"id":"pagination","type":"SelectorElementClick","selector":"tr.searchRow, tr.searchRowAlternate","parentSelectors":["_root"],"multiple":true,"delay":"3000","clickElementSelector":"tr.searchPager:nth-of-type(1) td:nth-of-type(n+5):not(:last-of-type) a","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"outlet","type":"SelectorLink","selector":"div.labelTitle a","parentSelectors":["_root","pagination"],"multiple":true,"delay":"1000"},{"id":"title","type":"SelectorText","selector":"div#ctl00_ContentPlaceHolder1_pnlTitle.row span","parentSelectors":["outlet"],"multiple":false,"regex":"","delay":0},{"id":"phone","type":"SelectorText","selector":"div#ctl00_ContentPlaceHolder1_pnlPhone.row span","parentSelectors":["outlet"],"multiple":false,"regex":"","delay":0},{"id":"street address","type":"SelectorText","selector":"div#ctl00_ContentPlaceHolder1_pnlPhysicalAddress.row div.majorItem","parentSelectors":["outlet"],"multiple":false,"regex":"","delay":0},{"id":"postal address","type":"SelectorText","selector":"div#ctl00_ContentPlaceHolder1_pnlPostalAddress.row span","parentSelectors":["outlet"],"multiple":false,"regex":"","delay":0},{"id":"email","type":"SelectorText","selector":"div#ctl00_ContentPlaceHolder1_pnlEmail.row span","parentSelectors":["outlet"],"multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorText","selector":"div#ctl00_ContentPlaceHolder1_pnlWebsite.row span","parentSelectors":["outlet"],"multiple":false,"regex":"","delay":0}]}

Any help greatly appreciated. I'm guessing I am just missing something obvious.

Very tricky pagination issue
#2

You won't be able to scrape this pagination with current web scraper selectors because the page reloads after you change page but the URL stays the same.

#3

Damn. I'm guessing you're right and know more about this than me - but just to confirm the URL does actually appear to change when the pagination links are clicked - the PageIndex GET parameter in the URL goes up

#4

FYI for anyone else hitting this problem, I solved this by just generating the URL's for each page (just changing out the pageindex), removing the click selector and adding them all as start URL's. A bit hacky but it works :smiley:

#6

But you can create one link for each pagination, like:

https://www.fdrr.ag.gov.au/Search.aspx?search_String=&locality=&postcode=&state=&PageIndex=0&SearchAll=true
https://www.fdrr.ag.gov.au/Search.aspx?search_String=&locality=&postcode=&state=&PageIndex=1&SearchAll=true

......

https://www.fdrr.ag.gov.au/Search.aspx?search_String=&locality=&postcode=&state=&PageIndex=119&SearchAll=true

save it as html and scrape your site.

I made it and this is your links:

http://listofico.eu/links.html

enjoy :slight_smile: