Pagination - consistently stops after 2 complete pages

How to?
#1

Hi there!

I tried to look for solutions in the forum, but the situations I find all seem a bit different.To be honest, I'm totally new to scraping and I find it rather difficult to understand how proposed solutions work and 'tweak' them to fit my situation.

Goal: The goal is to open all items (organisations) and scrape the website URL of each organisation. The search results are paginated and the search result URL does not change.

Problem: It consistently stops after page 2, giving 20 complete results.

The scrape starts at https://www.advieskeuze.nl/aankoopmakelaar/zoekresultaat?ps=DKQAdAAAAAAA#ps

This is the sitemap I made

{"_id":"advieskeuze-makelaars","startUrl":["https://www.advieskeuze.nl/aankoopmakelaar/zoekresultaat?ps=DKQAdAAAAAAA#ps"],"selectors":[{"id":"1-zoekresultaat-link","type":"SelectorLink","parentSelectors":["_root","pagination1"],"selector":"a.blauw","multiple":true,"delay":"500"},{"id":"2makelaar-website","type":"SelectorText","parentSelectors":["1-zoekresultaat-link"],"selector":"span a.blauw","multiple":false,"regex":"","delay":0},{"id":"3-terug-zoekresultaten","type":"SelectorLink","parentSelectors":["1-zoekresultaat-link"],"selector":"div.mt1.hide-mobile a","multiple":false,"delay":"200"},{"id":"pagination1","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".paginationVolgende.menuLink","multiple":false,"delay":"250","clickElementSelector":".paginationVolgende.menuLink","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

Very curious to hear your thoughts!

Best,
Petja

#2

try this

It will scroll through all 200 records before it clicks into each one. It currently get's you name and web address

{"_id":"advieskeuze-makelaars","startUrl":["https://www.advieskeuze.nl/aankoopmakelaar/zoekresultaat?ps=DKQAdAAAAAAA#ps"],"selectors":[{"id":"1-zoekresultaat-link","type":"SelectorLink","parentSelectors":["pagination1"],"selector":"a.blauw","multiple":false,"delay":""},{"id":"2makelaar-website","type":"SelectorText","parentSelectors":["1-zoekresultaat-link"],"selector":"span a.blauw","multiple":false,"regex":"","delay":0},{"id":"pagination1","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.zoekregel","multiple":true,"delay":"0","clickElementSelector":"span.paginationVolgende.menuLink","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"}]}
#3

Thank you Bretfeig! It gets to 200 results now. Any idea on why it stops there?

#4

The search results only come back with top 200 elements. Limitation on web page

"191 tot 200 van top 200 (3491 totaal) "

#5

Thanks for your quick reply! There are 200 pages with each a maximum of 20 results. The total number of results is 3491. It comes back with 200. Perhaps it only opens one result per page?

#6

There are not 200 pages, there are 200 results, each page has 10 and there are 20 page .

Count the first page, I get 10 results

#7

Ah sorry, got it! Thanks!