Load more by scroll

How to?
#1

I try to scrape this site (www.frauimmer-herrewig.de) it all works fine, even subcategories get scraped even though I just selected the parent list.

Anyway, I have a problem getting the items scraped which dynamicly load by scroll. I tried different ways which I found in this forum, but nothing worked for me. The attached sitemap is my latest working sitemap without the try to get the load more items.

Can somebody give me a hint?

Url: https://www.frauimmer-herrewig.de

Sitemap:
{"_id":"frauimmer-herrewig","startUrl":["https://www.frauimmer-herrewig.de/"],"selectors":[{"id":"category-link","type":"SelectorLink","parentSelectors":["_root"],"selector":"ul.menu.categories > li > a","multiple":true,"delay":0},{"id":"card","type":"SelectorLink","parentSelectors":["category-link"],"selector":".company-box > div.details > h2 > a","multiple":true,"delay":0},{"id":"vendor-title","type":"SelectorText","parentSelectors":["card"],"selector":"h3.h5","multiple":false,"regex":"","delay":0},{"id":"vendor-name","type":"SelectorText","parentSelectors":["card"],"selector":"div.info-box p:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"vendor-address","type":"SelectorText","parentSelectors":["card"],"selector":"p span:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"vendor-city","type":"SelectorText","parentSelectors":["card"],"selector":"p span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"vendor-mobile","type":"SelectorText","parentSelectors":["card"],"selector":"div.info-box ul.no-bullet:nth-of-type(1) li:nth-of-type(1) a","multiple":false,"regex":"","delay":0},{"id":"vendor-phone","type":"SelectorText","parentSelectors":["card"],"selector":"div.info-box ul.no-bullet:nth-of-type(1) li:nth-of-type(2) a","multiple":false,"regex":"","delay":0},{"id":"vendor-mail","type":"SelectorText","parentSelectors":["card"],"selector":"ul.no-bullet:nth-of-type(2) li:nth-of-type(1) a","multiple":false,"regex":"","delay":0},{"id":"vendor-website","type":"SelectorText","parentSelectors":["card"],"selector":"ul.no-bullet:nth-of-type(2) li:nth-of-type(2) a","multiple":false,"regex":"","delay":0}]}

#2

Hello Fabio,

In this case, you had to select a selector from the html, as by selecting the selector with a click, the scraper chooses the wrong selector for the scroll to work.

Try this one:

{"_id":"frauimmer-herrewig","startUrl":["https://www.frauimmer-herrewig.de/"],"selectors":[{"id":"category-link","type":"SelectorLink","parentSelectors":["_root"],"selector":"ul.menu.categories > li > a","multiple":true,"delay":0},{"id":"card","type":"SelectorLink","parentSelectors":["scroll"],"selector":"h2.h3 a","multiple":false,"delay":0},{"id":"vendor-title","type":"SelectorText","parentSelectors":["card"],"selector":"h3.h5","multiple":false,"regex":"","delay":0},{"id":"vendor-name","type":"SelectorText","parentSelectors":["card"],"selector":"div.info-box p:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"vendor-address","type":"SelectorText","parentSelectors":["card"],"selector":"p span:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"vendor-city","type":"SelectorText","parentSelectors":["card"],"selector":"p span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"vendor-mobile","type":"SelectorText","parentSelectors":["card"],"selector":"div.info-box ul.no-bullet:nth-of-type(1) li:nth-of-type(1) a","multiple":false,"regex":"","delay":0},{"id":"vendor-phone","type":"SelectorText","parentSelectors":["card"],"selector":"div.info-box ul.no-bullet:nth-of-type(1) li:nth-of-type(2) a","multiple":false,"regex":"","delay":0},{"id":"vendor-mail","type":"SelectorText","parentSelectors":["card"],"selector":"ul.no-bullet:nth-of-type(2) li:nth-of-type(1) a","multiple":false,"regex":"","delay":0},{"id":"vendor-website","type":"SelectorText","parentSelectors":["card"],"selector":"ul.no-bullet:nth-of-type(2) li:nth-of-type(2) a","multiple":false,"regex":"","delay":0},{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["category-link"],"selector":".company-box","multiple":true,"delay":"2000"}]}

#3

Wow! Thank you so much @webber!
I checked your sitemap and think now I got it :slight_smile:

#4