Selector Element Scroll for automatic 'load-more' on website

How to?
#1

I want to fetch all comments from the forum. The page loads automatically as the user scrolls. The sitemap is working well, except that it only loads more a few times, scrapes all the data and stops. It is not scrolling until the end of the forum.

Url: https://www.cryptocompare.com/coins/btc/forum/USD

Sitemap:
{"_id":"cryptocompare","startUrl":["https://www.cryptocompare.com/coins/btc/forum/USD"],"selectors":[{"id":"box","type":"SelectorElement","selector":"div.post-content","parentSelectors":["loader"],"multiple":true,"delay":0},{"id":"content","type":"SelectorText","selector":"p.ng-binding","parentSelectors":["box"],"multiple":false,"regex":"","delay":0},{"id":"name","type":"SelectorText","selector":"span.username","parentSelectors":["box"],"multiple":false,"regex":"","delay":0},{"id":"time","type":"SelectorText","selector":"div.item-ago","parentSelectors":["box"],"multiple":false,"regex":"","delay":0},{"id":"loader","type":"SelectorElementScroll","selector":"forum-container.ng-isolate-scope","parentSelectors":["_root"],"multiple":true,"delay":"2000"}]}

Please help :slight_smile:

#2

Remove the second element selector and make the element scroll selector select the box elements.

Tthe element scroll selector will scroll down the page until there are no new elements loaded. In this case it is selecting just one element.

1 Like