Selection on Scroll Loading

How to?
#1

Hey guys,

Is it possible to take in data as the loading on the page happens when we scroll?

My sitemap is working perfectly fine but it is not able to take more elements since the websites load more products on scrolling down.

You can check out the link here: https://www.mojarto.com/artworks/abstract/s
It is similar how Twitter works. Just loading more products and not tweets.

Sitemap:
{"_id":"mojarto_1","startUrl":["https://www.mojarto.com/artworks/abstract/s"],"selectors":[{"id":"Main_div","type":"SelectorElement","parentSelectors":["_root"],"selector":"li.white-panel","multiple":true,"delay":"1000"},{"id":"painting_link","type":"SelectorLink","parentSelectors":["Main_div"],"selector":"a.mname","multiple":false,"delay":"100"},{"id":"larger_image","type":"SelectorImage","parentSelectors":["painting_link"],"selector":"div.full img.large-artimage","multiple":false,"delay":"100"}]}

#2

You were so close. Instead of Element select, use Element Scroll Select. The trick is to play with the delay setting (both on your scroll select and the main page load delay)

Here is the updated map

{"_id":"forum-broke-autoscroll","startUrl":["https://www.mojarto.com/artworks/abstract/s"],"selectors":[{"id":"Main_div","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"li.white-panel","multiple":true,"delay":"9000"},{"id":"painting_link","type":"SelectorLink","parentSelectors":["Main_div"],"selector":"a.mname","multiple":false,"delay":"100"},{"id":"larger_image","type":"SelectorImage","parentSelectors":["painting_link"],"selector":"div.full img.large-artimage","multiple":false,"delay":"100"}]}
1 Like
#3

Hi!
This worked like a charm. Got over 1.3k paintings. Thank you so much!

I was trying on another website but that one has pagination. Is it possible for you to help me out?
It worked without pagination, but when I added the pagination, it only works on the pagination but does not select any of the paintings.

URL: https://www.saatchiart.com/paintings/abstract

Sitemap:
{"_id":"saatchi_1","startUrl":["https://www.saatchiart.com/paintings/abstract"],"selectors":[{"id":"main_div","type":"SelectorElement","parentSelectors":["pagination"],"selector":"div.polaroid","multiple":true,"delay":0},{"id":"title ","type":"SelectorLink","parentSelectors":["main_div"],"selector":"div.tl62ok-6.gibNzB a","multiple":false,"delay":0},{"id":"picture_link","type":"SelectorLink","parentSelectors":["title "],"selector":"a.fullscreen","multiple":false,"delay":0},{"id":"final_image","type":"SelectorImage","parentSelectors":["picture_link"],"selector":"div.ui-tooltip.ui-tooltip-focus div.ui-tooltip-content img","multiple":false,"delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"a.s1xud8v9-1:nth-of-type(n+2)","multiple":true,"delay":"100"}]}

#4

here you go. It's not 100% as it's going to go into a loop when it reached 990 due to how I configured to the pagination to look for the 2nd to last element which is the (>) next page arrow. However, once you get up to 980+ that goes away and it will likely loop between 998 and 999 indefinitely.

There might be a way to stop that but it would require more playing then I have time for now.

There is another way by using URL ranges. See if you can search the forum and find examples of that. If you do, just delete the pagination element and i will get you those last few without loooping

Fixed Map.

{"_id":"saatchi_1","startUrl":["https://www.saatchiart.com/paintings/abstract"],"selectors":[{"id":"title ","type":"SelectorText","parentSelectors":["Element Select "],"selector":"div.tl62ok-6.gibNzB a","multiple":false,"regex":"","delay":0},{"id":"picture_link","type":"SelectorLink","parentSelectors":["Element Select "],"selector":"a.z4v2fy-0","multiple":false,"delay":0},{"id":"final_image","type":"SelectorElementAttribute","parentSelectors":["picture_link"],"selector":"img.art-original ","multiple":false,"extractAttribute":"src","delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"a.s1xud8v9-1:nth-last-of-type(2)","multiple":false,"delay":""},{"id":"Element Select ","type":"SelectorElement","parentSelectors":["_root","pagination"],"selector":"div.polaroid","multiple":true,"delay":0}]}