Scroller does not work on certain websites

leemeng · October 4, 2019, 1:17pm

Web Scraper version: Dev 0.4.2.3
Chrome version: 77.0.3865.90 (Official Build) (64-bit) (cohort: Stable
OS:

On certain websites with sections, the scroller does not activate, no matter which element is selected.

Examples:
https://partscatalog.deere.com/jdrc/search/type/parts/term/JD10436

https://super.walmart.com.mx/productos?Ntt=chocolates

Sitemap: None, scroller does not work on these sites.

{id:"my sitemap"}

martins · October 5, 2019, 5:45am

Currently scroll element selector only tries to scroll down page body. It doesn't scroll the scroll bar of different elements. For example in the John Deere page you can actually see that the scroll bar isn't full height of the window but only for the list wrapper element.

There is a undocumented workaround that you can try. It is not that easy to do but it will make the scrolling work. In a future release we want to make the element scroll down selector automatically discover the scroll-able element. That is why we didn't publish this feature. Follow these steps:

Generate a CSS selector for the wrapper element. The easiest way is to select the wrapper element while hovering the scroll bar.
Create element scroll down selector
Export the sitemap
Find the element scroll down selector in the exported source
Add an extra parameter to the selector ,"scrollElementSelector":"<CSS SELECTOR HERE>"
Import the sitemap

Here is an example sitemap for the John Deere page:

{"_id":"scroll-example","startUrl":["https://partscatalog.deere.com/jdrc/search/type/parts/term/JD10436"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".linkList a","multiple":true,"delay":2000,"scrollElementSelector":"div.content"}]}

leemeng · October 5, 2019, 6:12am

This works great, thank you.

Marvin · February 14, 2020, 11:09am

Hi @martins - I have a similar issue but my scrollframe is so long that after applying your "fix" it just picks up the last items on the page. Is there any way I can tell it to scroll "step by step"?

Thanks a lot!

bier · March 11, 2021, 10:54pm

Hey @leemeng and @martins, I am having a similar problem but can't manage to figure out which CSS SELECTOR to use for the scroll to work. I'm working on this website: https://shop.weezy.co.uk/en/store/Weezy_SW6/478923

Some help would be very much appreciated!

kwilson · July 11, 2021, 5:45pm

I'm still having this issue in 2021. Perhaps it would be wise to add "scrollElementSelector" to the regular UI so that that users can specify it when building their sitemaps?

martins · July 12, 2021, 10:50am

We will attempt to make this auto-magically work by scanning the page for scroll bars. That's why we have the feature hidden. No ETA for this at the moment.

Our next big release is pagination selector. If you want to bump this up, do the survey within the extension. We aggregate and prioritize problems from the web scraping problem survey.

zewitch · July 17, 2021, 9:52pm

Hello @martins please pardon my intrusion upon this topic, but I have been using this undocumented feature to try and scrape tweetdeck (single column, all other columns removed, column set to my own timeline)

I've had some success (scroll scrolls! Some tweets are grabbed!) but also perplexing problems (tweets are being grabbed chaotically and most of them are lost, process terminates way before all tweets are scrolled through)

if you could maybe give some pointers or maybe take a look at tweetdeck when you have time, it would be very very appreciated

DiogoGranja · July 20, 2021, 5:36pm

I had a case similar to yours, about the scrollbar, I had to put a scroll-element inside another scroll-element, with that he managed to get all the elements on the page, the problem of putting scroll-element inside another scroll, and that increased the time on the page, like 40 seconds, more for my case it met my need, to be able to scrape all the products of an e-commercer. more than 8 hours of scanning. I need to study a way to reduce this time.