Kickstarter - pagination problem

How to?
#1

Hey everybody,

In my PhD research I would like analyse the Kickstarter campaigns in the tech category. I have some problem with the pagination... The element scroll down doesn't work for me - don't know why. It scrapes only 12 projects but there are more than 40,000. Do you have maybe some idea? I read the previous forum topics as well, but haven't find the solution. I spent more than 3 hours to figure it out, without results. I got stuck, please help!

Url: https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=0&sort=newest&seed=2650363&page=4

Sitemap:
{"_id":"kickstarter_ihavenoidea","startUrl":["https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=0&sort=newest&seed=2650363"],"selectors":[{"id":"projects","type":"SelectorLink","parentSelectors":["scrolldown"],"selector":".js-react-proj-card a.mb3","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["projects"],"selector":".grid-col-10 h2","multiple":false,"regex":"","delay":0},{"id":"loadmore","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"h3.type-18","multiple":false,"delay":0,"clickElementSelector":"a.bttn","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"scrolldown","type":"SelectorElementScroll","parentSelectors":["_root","scrolldown"],"selector":"div.js-project-group","multiple":false,"delay":0}]}

Thank you for your help in advance!

#2

An interesting site. It has both Load More and lazy loading, so you'd need to deal with both. Also the projects are actually grouped in "pages" containing 12 projects in each.
For the scroller I am using the "pages" as selector, not the project cards. This example below will get you 180 projects. To increase the projects, you can change the number in:
div.grid-row:nth-of-type(-n+15)

You'll need to do some math to figure out the max number of pages (e.g. 15 x 12 = 180 projects in the example). I suggest you use some filtering criteria and don't try to scrape too many cards at once, definitely not 40,000+. You would probably hit the browser/RAM limits, plus WS itself has a limit of 10,000 lines of data max (20,000 max for the cloud scraper).

{"_id":"forum-kickstarter","startUrl":["https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=0&sort=newest&seed=2650363"],"selectors":[{"id":"Click Load More","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div[class^='load_more']","multiple":false,"delay":"2500","clickElementSelector":"div[class^='load_more'] a.bttn","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Separate scroller","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.grid-row:nth-of-type(-n+15)","multiple":true,"delay":"3500"},{"id":"Custom delay","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"b.count","multiple":false,"delay":"2000","clickElementSelector":"b.count","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Project cards","type":"SelectorElement","parentSelectors":["_root"],"selector":"div.js-react-async-proj-card","multiple":true,"delay":0},{"id":"Project link","type":"SelectorLink","parentSelectors":["Project cards"],"selector":"div > div > div[class*='hover-target'] > a[href*='kickstarter']","multiple":false,"delay":0},{"id":"Project name","type":"SelectorText","parentSelectors":["Project cards"],"selector":"div > a h3","multiple":false,"regex":"","delay":0},{"id":"Description","type":"SelectorText","parentSelectors":["Project cards"],"selector":"a h3 ~ p","multiple":false,"regex":"","delay":0},{"id":"Funded","type":"SelectorText","parentSelectors":["Project cards"],"selector":"div[data-test-id='percent-raised']","multiple":false,"regex":"","delay":0},{"id":"Category","type":"SelectorText","parentSelectors":["Project cards"],"selector":"div.flex a[href*='categories/']","multiple":false,"regex":"","delay":0},{"id":"Location","type":"SelectorText","parentSelectors":["Project cards"],"selector":"div.flex a[href*='places/']","multiple":false,"regex":"","delay":0}]}

1 Like