Infinite scrolling stops after 2nd scroll

[therocsta]

Hi. I have created a sitemap to scrape this page. The page is an infinitely scrolling page, loading items each time the browser gets to the bottom. I have the scrolling working, but the problem is the scrolling stops after the 2nd scroll.
I have tried different timeouts etc but just can't get the page to keep scrolling past the 3rd load of results (ie. it loads first 10 results, then scrolls, then loads 2nd set of results, then scrolls, then loads 3rd set of results and then stops scrolling.

If I manually scroll while the scraper is running, I get more results. I need to get this to happen automatically though, and I cannot see what is causing the auto scrolling to stop.

Url: Discover

Sitemap:

{"_id":"gumroad-animation","startUrl":["https://discover.gumroad.com/discover?category=animation&sort=price_desc"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.product-card__grid.js-product-card-holder.product-card-grid","multiple":true,"delay":10000,"scrollElementSelector":"main.discover-page.js-discover-product-ui.js-discover-category-cards-holder"},{"delay":10000,"id":"link","multiple":true,"parentSelectors":["_root"],"selector":"a.product-info-link","type":"SelectorLink"}]}

[leemeng]

Just need to tweak the scroller selector to
div > article.product.card

You can reduce delay to 3000. Also use the zoom out trick.

[therocsta]

Hi leemeng

Thanks for coming back to me. The scroller will only scroll when I set the scroller selector as follows:

"scrollElementSelector":"main.discover-page.js-discover-product-ui.js-discover-category-cards-holder"

Setting this to any other selector doesn't trigger the scroller (it took me a while to find the right trigger element to get the scroller to scroll ...)

Unfortunately, the zoom out trick doesn't work on this site - even zoomed out to 25% it only loads the first set of results.

I just can't see why the scroller stops after 2 scrolls ...

So at most using the following sitemap I get 27 results and technically there are like 4000+ possible listings:

{"_id":"gumroad-animation-2","startUrl":["https://discover.gumroad.com/discover?category=animation&sort=price_desc"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.article.product.card","multiple":true,"delay":3000,"scrollElementSelector":"main.discover-page.js-discover-product-ui.js-discover-category-cards-holder"},{"delay":3000,"id":"link","multiple":true,"parentSelectors":["_root"],"selector":"a.product-info-link","type":"SelectorLink"}]}