Problems with pagination

Describe the problem.

Hello,
I', trying to scrap some information from here: https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=24554868&sort=magic&seed=2541685.

I have all the data I need but I have problems with the pagination, don't work, anly get information from the first page.

I tried to use something like this https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=24554868&sort=magic&seed=2541685&page=1.... but there are problems...

Please, have you got any idea?.

Url: http://example.com

Sitemap:
{"_id":"kickstarterprueba2","startUrl":["https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=24554868&sort=magic&seed=2541685"],"selectors":[{"id":"menu","type":"SelectorLink","selector":"section.pb3 div.type-18 a.soft-black","parentSelectors":["_root"],"multiple":true,"delay":0},{"id":"enlace a tabla","type":"SelectorLink","selector":"div.col-md-8-24 a.bttn","parentSelectors":["menu"],"multiple":false,"delay":"10"},{"id":"tipo1","type":"SelectorText","selector":"li.pledge-selectable.pledge--no-reward div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo2","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(2) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo3","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(3) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo4","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(4) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo 5","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(5) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo6","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(6) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo7","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(7) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo8","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(8) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo9","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(9) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo10","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(10) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo11","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(11) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo12","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(12) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo13","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(13) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo14","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(14) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo15","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(15) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo16","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(16) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"tipo17","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(17) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Tipo18","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(18) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Tipo19","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(19) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Tipo20","type":"SelectorText","selector":"li.js-reward-available:nth-of-type(20) div.pledge__info","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Recaudado","type":"SelectorText","selector":"div.col-md-8-24 span.green-700","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Objetivo","type":"SelectorText","selector":"div.col-md-8-24 span.block span.money","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Patrocinadores","type":"SelectorText","selector":"div.col-md-8-24 div.js-backers_count","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"dias que faltan","type":"SelectorText","selector":"div.col-md-8-24 span.block.type-16","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Creador","type":"SelectorText","selector":"span.navy-700 a.medium","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Fecha fin","type":"SelectorText","selector":"div.col-md-8-24 p.mb3 time.js-adjust-time","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Galeria fotos","type":"SelectorLink","selector":"a.js-open-supporting-material","parentSelectors":["menu"],"multiple":false,"delay":0},{"id":"project description","type":"SelectorText","selector":"div.full-description","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"Riesgos y desafios","type":"SelectorText","selector":"div.mb3.js-risks p","parentSelectors":["menu"],"multiple":false,"regex":"","delay":0},{"id":"paginacion","type":"SelectorElementScroll","selector":"div.load_more","parentSelectors":["menu"],"multiple":true,"delay":0}]}

Hi,
this site has a strange behavior in the combination of pages numbers in the URL and the action of the buton "Load More".
I am not sure that my test is academic, but I achieved the scrapping of 21 projets by:

  • ajusting the delays (to short the scrapping )
  • modifying the name of the label of the selector concerning the button "Load More"
  • ajusting the range [1-2] in the metadata.

I suppose you need to test more to check the way of proceding of my sitemap:

Sitemaps
Sitemap test_kickstarter
Create new sitemap

{"_id":"test_kickstarter","startUrl":["https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=966591&sort=magic&seed=2541685&page=[1-2]"],"selectors":[{"id":"element","type":"SelectorElementClick","selector":"section.pb3 div.js-react-proj-card","parentSelectors":["_root"],"multiple":true,"delay":"4000","clickElementSelector":"div.load_more.mt3 a.bttn.bttn-green.bttn-medium","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"link","type":"SelectorLink","selector":"div.type-18 a.soft-black","parentSelectors":["element"],"multiple":true,"delay":"0"},{"id":"creator","type":"SelectorText","selector":"span.navy-700 a.medium","parentSelectors":["link"],"multiple":false,"regex":"","delay":"0"},{"id":"invest","type":"SelectorText","selector":"div.col-md-8-24 span.block span.money","parentSelectors":["link"],"multiple":false,"regex":"","delay":0},{"id":"project","type":"SelectorText","selector":"div.col-20-24 h2.type-24","parentSelectors":["link"],"multiple":false,"regex":"","delay":0}]}