Describe the problem.
Hello!
I am a new comer and trying to get data from lego, but don't understand how to do element scroll down after element click? This sitemap can just do element cilck.
Url: https://ideas.lego.com/search/global_search?type=product_ideas&query=&sort=top
Sitemap:
{"_id":"lego","startUrl":"https://ideas.lego.com/search/global_search?type=product_ideas&query=&sort=top","selectors":[{"parentSelectors":["content"],"type":"SelectorText","multiple":true,"id":"title","selector":"h3.card-title a","regex":"","delay":""},{"parentSelectors":["content"],"type":"SelectorText","multiple":false,"id":"supporters","selector":"div.project-block-left a.btn","regex":"","delay":""},{"parentSelectors":["content"],"type":"SelectorText","multiple":false,"id":"daysleft","selector":"div.days-left-value","regex":"","delay":""},{"parentSelectors":["_root"],"type":"SelectorElement","multiple":true,"id":"content","selector":"div.col-xs-12 div.card","delay":"2000"},{"parentSelectors":["content"],"type":"SelectorText","multiple":false,"id":"author","selector":"div.user-alias a","regex":"","delay":""},{"parentSelectors":["content"],"type":"SelectorText","multiple":false,"id":"time","selector":"time","regex":"","delay":""},{"parentSelectors":["content"],"type":"SelectorText","multiple":false,"id":"comments","selector":"span.stat-amount","regex":"","delay":""}]}
I can solve this problem by click the button first, and web scraper helps me to scroll down.
Yes, after you click "See More" the infinite scroll starts. This site organizes the items in UL blocks, similar to pages. Each "page" has 21 items. So I'm using the pages as the Scroller selector. I've made the Scroller stop at 5 pages (not infinite) so that would get you 105 items (5 x 21 = 105). If you want more items, you can increase the "pages" number in the selector: ul:nth-of-type(-n+5) li div.card
Or if you want infinite scroll, just remove the nth selector, like so: ul li div.card
I would not recommend infinite scroll 'cos you may run out of RAM and Chrome may crash. Also, you may need to increase the scroller's delay (currently 2000) if you're getting a large number of pages.
{"_id":"lego_scroll_test","startUrl":["https://ideas.lego.com/search/global_search?type=product_ideas&query=&sort=top"],"selectors":[{"id":"Click See More","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#search-more","multiple":false,"delay":2000,"clickElementSelector":"a.btn-alternate","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Scroller","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"ul:nth-of-type(-n+5) li div.card","multiple":true,"delay":2000},{"id":"Title","type":"SelectorText","parentSelectors":["Scroller"],"selector":".card-title a","multiple":false,"regex":"","delay":0},{"id":"Supporters","type":"SelectorText","parentSelectors":["Scroller"],"selector":"a.btn","multiple":false,"regex":"","delay":0},{"id":"Days left","type":"SelectorText","parentSelectors":["Scroller"],"selector":"div.days-left-value","multiple":false,"regex":"","delay":0},{"id":"Author","type":"SelectorText","parentSelectors":["Scroller"],"selector":".user-alias a","multiple":false,"regex":"","delay":0}]}
Thanks so so so much!!! 
You've solved another problem that I want to ask. I have wondering for a few days why web scraper stop working even if scroller delay is large.