Unable to scrape all entries on page because entries only load on scroll

Hi there,

I'm trying to scrape an eCommerce site. An example of the link i am scraping is https://shopee.sg/search?keyword=drill&page=[1]&sortBy=relevancy

My issue is, there are 50 entries on each page. However, only 15 entries load on page load, the others show up only when i scroll down the page, i tried element scroll selector but i am only able to scrape 20 entries. Is there any way to mimic scrolling down a page to the end? ( Correct me if i am wrong but I think element scroll jumps to end of page?)

Has anyone come across a similar issue and how to scrape a page that loads on scrolling?

Thank you guys!

You are probably not using the scroller correctly. There is a very good vid by webscraper.io which covers this step-by-step:

The scroller bit starts at 3:25, but you should watch from the beginning because it shows you how to set up wrapper elements properly (which the scroller needs).

One note though, the delay mentioned in the vid, 500 ms, is probably too short for many sites. I find I need to set it to at least 2000 ms and up to 6000 ms (2-6 sec) for it to work properly.