Web Scraper version: 0.3.8
Chrome version: Version 68.0.3440.106 (Official Build) (64-bit)
OS: MacOS High Sierra 10.13.6
If the pagination link selector is made a child to itself it will recursively discover all pagination pages. Figure 3 shows a selector graph where you can see how pagination links discover more pagination links and more data.
My sitemap (see below) has indeed a 'pagination' which is it's own child, and it looks recursive when seeing the selector graph.
However, the scraping stops after the second page.
Now, after pasting the sitemap here, I can see that the 'child' pagination doesn't really exist in the sitemap json, even if it shows in the graph. Maybe that's where the bug is
Thank you for this excellent tool!
Florin
Sitemap:
{"_id":"agencies_clutch","startUrl":["https://clutch.co/agencies/social-media-marketing/facebook"],"selectors":[{"id":"pagination","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.pager-next a","multiple":true,"delay":0},{"id":"item","type":"SelectorElement","parentSelectors":["pagination"],"selector":"li.provider-row","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["item"],"selector":"span.field-content a","multiple":false,"regex":"","delay":0},{"id":"descr","type":"SelectorText","parentSelectors":["item"],"selector":"p.tagline","multiple":false,"regex":"","delay":0},{"id":"location","type":"SelectorText","parentSelectors":["item"],"selector":"div.list-item:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"employees","type":"SelectorText","parentSelectors":["item"],"selector":"div.list-item:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"pagination1","type":"SelectorLink","parentSelectors":["pagination1"],"selector":"li.pager-next a","multiple":false,"delay":0}]}