Infinite Scroll Loop problem

How to?
#1

Describe the problem.
Hey guys, currently I am trying to scrape a blog site that has infinite scroll.
I was therefore using the element scroll selector in order to make the scraper scroll down to the bottom of the dynamic webpage prior to clicking the links. For some reason it just does not scrape the complete data, but returns appox. 230 links only. Could you tell me what I am doing wrong?

Url: https://futurism.com/

Sitemap:
{"_id":"forum_futurism_all","startUrl":["https://futurism.com/"],"selectors":[{"id":"blogsfromscrolldown","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.pages__LatestPostsInitialColumn-sc-1cvwks4-4 a.FuturismPostListing3__StyledLinkContent-h29va7-0, div.PageContent div div.PageLayoutRow__Wrapper-ie22rx-0 a.FuturismPostListing3__StyledLinkContent-h29va7-0","multiple":true,"delay":"5000"},{"id":"linktoblog","type":"SelectorLink","parentSelectors":["blogsfromscrolldown"],"selector":"parent","multiple":false,"delay":0},{"id":"blog_element","type":"SelectorElement","parentSelectors":["linktoblog"],"selector":"div.PageContent > div > div > div","multiple":false,"delay":0},{"id":"blog_title","type":"SelectorText","parentSelectors":["blog_element"],"selector":"h1.Type","multiple":false,"regex":"","delay":0},{"id":"blog_subtitle","type":"SelectorText","parentSelectors":["blog_element"],"selector":"h4.Type","multiple":false,"regex":"","delay":0},{"id":"blog_editor","type":"SelectorText","parentSelectors":["blog_element"],"selector":"div.Type span a","multiple":false,"regex":"","delay":0},{"id":"blog_date","type":"SelectorText","parentSelectors":["blog_element"],"selector":"span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"blog_text","type":"SelectorText","parentSelectors":["blog_element"],"selector":"div.Type > div:nth-of-type(n+2)","multiple":false,"regex":"","delay":0}]}

#2

Hey,

The reason for that is the selectors that you have selected are constant, so when the scraper discovers new 'cards' that do not posses any of the selectors that you have selected (and it does) it stops scraping.

Here is an updated version of the sitemap that should scroll until there is no more new elements to be found or you run out of RAM.

{"_id":"forum_futurism_all","startUrl":["https://futurism.com/"],"selectors":[{"id":"blogsfromscrolldown","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".PageContent > div a:has(.PostInfoColumn)","multiple":true,"delay":"3000"},{"id":"linktoblog","type":"SelectorLink","parentSelectors":["blogsfromscrolldown"],"selector":"_parent_","multiple":false,"delay":0},{"id":"blog_element","type":"SelectorElement","parentSelectors":["linktoblog"],"selector":"div.PageContent > div > div > div","multiple":false,"delay":0},{"id":"blog_title","type":"SelectorText","parentSelectors":["blog_element"],"selector":"h1.Type","multiple":false,"regex":"","delay":0},{"id":"blog_subtitle","type":"SelectorText","parentSelectors":["blog_element"],"selector":"h4.Type","multiple":false,"regex":"","delay":0},{"id":"blog_editor","type":"SelectorText","parentSelectors":["blog_element"],"selector":"div.Type span a","multiple":false,"regex":"","delay":0},{"id":"blog_date","type":"SelectorText","parentSelectors":["blog_element"],"selector":"span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"blog_text","type":"SelectorText","parentSelectors":["blog_element"],"selector":"div.Type > div:nth-of-type(n+2)","multiple":false,"regex":"","delay":0}]}
#3

Awesome! Thanks a lot, I will try it and give you a feedback.