Hi. I have created a sitemap to scrape this page. The page is an infinitely scrolling page, loading items each time the browser gets to the bottom. I have the scrolling working, but the problem is the scrolling stops after the 2nd scroll.
I have tried different timeouts etc but just can't get the page to keep scrolling past the 3rd load of results (ie. it loads first 10 results, then scrolls, then loads 2nd set of results, then scrolls, then loads 3rd set of results and then stops scrolling.
If I manually scroll while the scraper is running, I get more results. I need to get this to happen automatically though, and I cannot see what is causing the auto scrolling to stop.
Url: Discover
Sitemap:
{"_id":"gumroad-animation","startUrl":["https://discover.gumroad.com/discover?category=animation&sort=price_desc"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.product-card__grid.js-product-card-holder.product-card-grid","multiple":true,"delay":10000,"scrollElementSelector":"main.discover-page.js-discover-product-ui.js-discover-category-cards-holder"},{"delay":10000,"id":"link","multiple":true,"parentSelectors":["_root"],"selector":"a.product-info-link","type":"SelectorLink"}]}