Speed on Element Scroll

Feature Requests
#1

Hi,

I'm trying to scrap SoundCloud stats (https://soundcloud.com/PROFILE/stats/). The page have some pop-ups with information I need.
It works fine for the first, but when I try to get the second popup, the element is not loaded, it will load only when I scroll the screen passing the element (the countries data).

I tried to use the "element scroll down", but it get to the bottom of the page (whatever element I select, it alway goes to the bottom) immediately, without time to load the element.
If I scroll manually in the scrap window, it works fine.

It's possible to select the speed of the scroll?

Here is the snippet of a minimum sitemap to do what I want: (currently not working)

{"_id":"teste","startUrl":["https://soundcloud.com/capitalaberto/stats"],"selectors":[{"id":"popup-principais-paises","type":"SelectorElementClick","selector":"div.modal__modal","parentSelectors":["_root"],"multiple":true,"delay":"6000","clickElementSelector":"div.topStatsModule__item:nth-of-type(3) div.topStats__wrapper.topStats__wrapperEligible h3.topStats__title","clickType":"clickOnce","discardInitialElements":true,"clickElementUniquenessType":"uniqueHTMLText"},{"id":"popup-principais-paises-modal","type":"SelectorElement","selector":"div.topStatsFullItem","parentSelectors":["popup-principais-paises"],"multiple":true,"delay":0},{"id":"popup-principais-paises-modal-itens-titulo","type":"SelectorText","selector":"p.topStatsFullItem__text span","parentSelectors":["popup-principais-paises-modal"],"multiple":false,"regex":"","delay":0},{"id":"popup-principais-paises-modal-itens-numero","type":"SelectorText","selector":"span.topStatsFullItem__count","parentSelectors":["popup-principais-paises-modal"],"multiple":true,"regex":"","delay":0},{"id":"scroll-down","type":"SelectorElementScroll","selector":"html","parentSelectors":["_root"],"multiple":true,"delay":"1500"}]}

Many thanks and great job with the extension, is very complete and helpful.

Diego

1 Like
#2

If the site renders only the part that is displayed on screen then before starting the scraping job you can try to zoom out as much as possible so site renders as much as possible.

#3

Good idea, I will try that.

Thanks!

#4

I had a case similar to yours, about the scroll bar, I had to put a scroll-element inside another scroll-element, with that he managed to get all the elements on the page, the problem of putting scroll-element inside another scroll, and that I increase the time on the page, like 40 seconds, more for my case it met my need, to be able to scrape all the products of an e-commercer. more than 8 hours of scanning. I need to study a way to reduce this time.

#5

+1 on this feature. Would be amazing to have