Why is Page Scrolling in my sitemap not working correctly?

How to?
#1

Hi,

For scraping products I need to scroll down a page und click on a button until no more buttons are loaded. But the scraping goes to the next step without scrolling down for every button. What did I do wrong?

Sitemap:
{"_id":"krannich_solarspeicher","startUrl":["https://shop.krannich-solar.com/"],"selectors":[{"id":"scrollen","type":"SelectorElementScroll","parentSelectors":["weiter-button","produkttypen"],"selector":"div.box--content","multiple":true,"delay":"500"},{"id":"weiter-button","type":"SelectorElementClick","parentSelectors":["weiter-button","produkttypen"],"selector":"div.content-main--inner","multiple":true,"delay":2000,"clickElementSelector":"a.is--primary.is--icon-right","clickType":"clickMore","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"produktseite","type":"SelectorLink","parentSelectors":["weiter-button","produkttypen"],"selector":"a.product--title","multiple":true,"delay":0},{"id":"produkttypen","type":"SelectorLink","parentSelectors":["_root"],"selector":".start-row-xs-6 .container > a","multiple":true,"delay":0},{"id":"krannich_artikelname","type":"SelectorText","parentSelectors":["produktseite"],"selector":"[itemprop='offers'] h2","multiple":false,"regex":"","delay":0},{"id":"krannich_hersteller","type":"SelectorText","parentSelectors":["produktseite"],"selector":"[itemprop='offers'] h3:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelnummer","type":"SelectorText","parentSelectors":["produktseite"],"selector":"div.bs-col-xs-6","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelpreis","type":"SelectorText","parentSelectors":["produktseite"],"selector":"span.price--bold","multiple":false,"regex":"","delay":0},{"id":"krannich_verfuegbarkeit","type":"SelectorText","parentSelectors":["produktseite"],"selector":"span.delivery--text","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelbezeichnung","type":"SelectorText","parentSelectors":["produktseite"],"selector":"[itemtype='http://schema.org/Product'] > div[itemprop='description']","multiple":false,"regex":"","delay":0},{"id":"krannich_technischeDetails","type":"SelectorText","parentSelectors":["produktseite"],"selector":".content--description div.tab--container-list","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelbild","type":"SelectorImage","parentSelectors":["produktseite"],"selector":"img[itemprop='image']","multiple":false,"delay":0},{"id":"krannich_download_1","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Datenblatt Deutsch']","multiple":false,"delay":0},{"id":"krannich_download_2","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Datenblatt Englisch']","multiple":false,"delay":0},{"id":"krannich_download_3","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Montageanleitung Englisch']","multiple":false,"delay":0},{"id":"krannich_download_4","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Garantiebedingungen Englisch']","multiple":false,"delay":0},{"id":"krannich_download_5","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Bedienungsanleitung Englisch']","multiple":false,"delay":0},{"id":"krannich_download_6","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Montageanleitung Deutsch']","multiple":false,"delay":0}]}

#2

Hi @wullner

It happens because the click element (load more button) is discovered only after a scroll is performed, so basically it should be like: scroll - click - scroll - click - scroll - click etc. till it discovers all of the 36 pages. Each page has 12 products in it.

If you notice each time after you perform the scroll or click the page link transforms. It seems that the very last page is 36. So you can use the predefined page interval pagination in the starting link itself.

After testing this sitemap in the Web Scraper Cloud all of the 432 produkten were disovered.

Page 36
Page 361586×1079 136 KB

product count
product count1806×239 19.7 KB

Sitemap:

{"_id":"krannich_solarspeicher","startUrl":["https://shop.krannich-solar.com/shop/solarspeicher/?p=[1-36]"],"selectors":[{"id":"produkten","type":"SelectorElement","parentSelectors":["_root"],"selector":"div.listing > div","multiple":true,"delay":0},{"id":"produktseite","type":"SelectorLink","parentSelectors":["produkten"],"selector":"a.product--title","multiple":false,"delay":0},{"id":"krannich_artikelname","type":"SelectorText","parentSelectors":["produktseite"],"selector":"[itemprop='offers'] h2","multiple":false,"regex":"","delay":0},{"id":"krannich_hersteller","type":"SelectorText","parentSelectors":["produktseite"],"selector":"[itemprop='offers'] h3:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelnummer","type":"SelectorText","parentSelectors":["produktseite"],"selector":"div.bs-col-xs-6","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelpreis","type":"SelectorText","parentSelectors":["produktseite"],"selector":"span.price--bold","multiple":false,"regex":"","delay":0},{"id":"krannich_verfuegbarkeit","type":"SelectorText","parentSelectors":["produktseite"],"selector":"span.delivery--text","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelbezeichnung","type":"SelectorText","parentSelectors":["produktseite"],"selector":"[itemtype='http://schema.org/Product'] > div[itemprop='description']","multiple":false,"regex":"","delay":0},{"id":"krannich_technischeDetails","type":"SelectorText","parentSelectors":["produktseite"],"selector":".content--description div.tab--container-list","multiple":false,"regex":"","delay":0},{"id":"krannich_artikelbild","type":"SelectorImage","parentSelectors":["produktseite"],"selector":"img[itemprop='image']","multiple":false,"delay":0},{"id":"krannich_download_1","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Datenblatt Deutsch']","multiple":false,"delay":0},{"id":"krannich_download_2","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Datenblatt Englisch']","multiple":false,"delay":0},{"id":"krannich_download_3","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Montageanleitung Englisch']","multiple":false,"delay":0},{"id":"krannich_download_4","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Garantiebedingungen Englisch']","multiple":false,"delay":0},{"id":"krannich_download_5","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Bedienungsanleitung Englisch']","multiple":false,"delay":0},{"id":"krannich_download_6","type":"SelectorLink","parentSelectors":["produktseite"],"selector":"a[title=' Montageanleitung Deutsch']","multiple":false,"delay":0}]}

Hope it helps!

1 Like
#3

Hi @ViestursWS
Thanks again for your really great help. Works perfect and it is much easier than my way. Even though I do not understand yet why the scrolling did not work as expected. But I can live with it :slight_smile:

Have a nice day and greetings
wullner

1 Like