Recursive pagination doesn't work

Web Scraper version: 0.3.8
Chrome version: Version 68.0.3440.106 (Official Build) (64-bit)
OS: MacOS High Sierra 10.13.6

If the pagination link selector is made a child to itself it will recursively discover all pagination pages. Figure 3 shows a selector graph where you can see how pagination links discover more pagination links and more data.

My sitemap (see below) has indeed a 'pagination' which is it's own child, and it looks recursive when seeing the selector graph.

However, the scraping stops after the second page.

Now, after pasting the sitemap here, I can see that the 'child' pagination doesn't really exist in the sitemap json, even if it shows in the graph. Maybe that's where the bug is :slight_smile:

Thank you for this excellent tool!
Florin

Sitemap:

{"_id":"agencies_clutch","startUrl":["https://clutch.co/agencies/social-media-marketing/facebook"],"selectors":[{"id":"pagination","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.pager-next a","multiple":true,"delay":0},{"id":"item","type":"SelectorElement","parentSelectors":["pagination"],"selector":"li.provider-row","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["item"],"selector":"span.field-content a","multiple":false,"regex":"","delay":0},{"id":"descr","type":"SelectorText","parentSelectors":["item"],"selector":"p.tagline","multiple":false,"regex":"","delay":0},{"id":"location","type":"SelectorText","parentSelectors":["item"],"selector":"div.list-item:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"employees","type":"SelectorText","parentSelectors":["item"],"selector":"div.list-item:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"pagination1","type":"SelectorLink","parentSelectors":["pagination1"],"selector":"li.pager-next a","multiple":false,"delay":0}]}

Hi there!

In order for your pagination selector to work recursively, you should set both itself and _root as a parent selector. Otherwise it won't work.

image

Got it!!

It both makes sense and it works :smiley:

Thank you, iconoclast!
Florin