A paginated site that I am trying to scrape requires you to scroll through all of the content on the page for the product thumbnails to load. As I would like to save the thumbnails, I tried to use element scroll.
The steps to my program are:
- Load a page. Scroll to the bottom
- Grab all product thumbnails and titles
- Open each product URL and grab description
- Open next page, and perform step 1 again
The issue is that the scrolling always keeps stopping after a while. This happens after:
- The laptop display turns off for a while
- The scraping window is not visible for a while
I discovered that it stops on step 1 if any of the above happens. For example, if the scraping window is not visible and it is on step 3, it will continue until it gets to the scrolling on step 1 and stops. I need to "view" the window again for the scrolling action to continue, and only then does it start chugging along until the window is out of sight and it goes to the next scrolling action on step 1 again.
Is there any way to continue scrolling without my having to keep the window on-screen all the time?
I am using Brave on a MacBook Pro.
Note: viewing the scraper window in Mission Control (without actually making the window active) also resumes the scrolling action. I don't know why my seeing the window has anything to do whether the scroll will occur, but that's how it works I guess.