Element Scroll Stops Working after 100 Items

How to?
#1

I was able to resolve some of the issues I was having by referencing other topics in here. However I can't seem to figure out why it keeps stopping after 100 items. Anyone have thoughts into why it would stop at 100?

Much appreciated!

##########

Url: http://youmewho.com/l/EtQduNCo2mQCBaqPw

Sitemap:
{"_id":"jamberry-capture","startUrl":["http://youmewho.com/l/EtQduNCo2mQCBaqPw"],"selectors":[{"id":"item-name","type":"SelectorText","selector":"parent","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"item-image","type":"SelectorImage","selector":"span.catalog img","parentSelectors":["item"],"multiple":false,"delay":0},{"id":"item","type":"SelectorElementScroll","selector":"div.reactWrap","parentSelectors":["_root"],"multiple":true,"delay":"3500"}]}

#2

For some reason scroll down won't even go down for me too, so I've found a different solution with two clickers, one calls an item popup, and another constantly clicks for next item.

Check it out:

{"_id":"youmewho2","startUrl":["http://youmewho.com/l/EtQduNCo2mQCBaqPw"],"selectors":[{"id":"click","type":"SelectorElementClick","selector":"div.dialog-root > div > div > div","parentSelectors":["_root"],"multiple":false,"delay":"1000","clickElementSelector":"div.reactWrap:nth-of-type(2)","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"click_next_inside","type":"SelectorElementClick","selector":"div.dialog-root > div > div > div","parentSelectors":["_root"],"multiple":true,"delay":"1500","clickElementSelector":"span.fontAwesome.chevronRightReactElementDetail","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"item_number","type":"SelectorText","selector":"div.textCenter > span","parentSelectors":["click_next_inside"],"multiple":false,"regex":"","delay":0},{"id":"img_src","type":"SelectorImage","selector":"div.textCenter img","parentSelectors":["click_next_inside"],"multiple":false,"delay":0},{"id":"name","type":"SelectorText","selector":"div#menuScrollable3.unselectable","parentSelectors":["click_next_inside"],"multiple":false,"regex":"","delay":0}]}

Yet another problem came up -- it seems they don't want item description to be copy-pasted, so they put it between divs. This can be solved using some CSS customizer extension to enforce proper listing, or either use JS extension to copy all text into one place.
I believe i could have missed something with the text, I'll to try to look into it tomorrow.

P.S. if you want to manually stop the scrape don't close the window, rather close the popup, scraper will successfully end scraping and results will be shown.

#3

Thank you for looking at this iconoclast. I loaded the sitemap and tried running it. It seems to only capture 100 entries too.

I understand what you mean about the description information which is why I went with the other method instead of doing the popup. I can at least grab the title the other way. The difference between the export file size is also huge with the pop up due to the vast amount of data it is collecting from the description.

I just don't understand why it is getting capped at 100 though.

#4

Unfortunately this website built using ReactJS and it force reflows all items even if i call scroll using JQuery extension. This needs a sophisticated workaround.

#5

Thank you iconoclast for looking into this. I agree this will definitely require a more sophisticated solutions.