Data is being duplicated and I can't figure out why

How to?
#1

I have used this same sitemap for months and it broke this week. The selectors highlight the correct data but when I scrape I'm only getting a few rows of data and they are duplicates of each other. The pagination was broken at first and I think I've fixed that but it's also possible that it is affecting the rest of the scrape.

Any ideas?

Url: https://www.kickstarter.com/discover/advanced?state=live&woe_id=0&pledged=3&sort=popularity&seed=2552956

Sitemap:
{"_id":"ks-100k-1m popular","startUrl":["https://www.kickstarter.com/discover/advanced?state=live&woe_id=0&pledged=3&sort=popularity&seed=2552956"],"selectors":[{"id":"load-more","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"section.pb3, div.load_more","multiple":true,"delay":"5000","clickElementSelector":"a.bttn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"project","type":"SelectorText","parentSelectors":["load-more"],"selector":"h3.type-18","multiple":false,"regex":"","delay":0},{"id":"user","type":"SelectorText","parentSelectors":["load-more"],"selector":"div.inline-block a.soft-black","multiple":false,"regex":"","delay":0},{"id":"percent-funded","type":"SelectorText","parentSelectors":["load-more"],"selector":"div.ksr-green-700 div.type-13:nth-of-type(2) span:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"days-left","type":"SelectorText","parentSelectors":["load-more"],"selector":"span.js-num","multiple":false,"regex":"","delay":0},{"id":"category","type":"SelectorText","parentSelectors":["load-more"],"selector":"a.dark-grey-500","multiple":false,"regex":"","delay":0},{"id":"url","type":"SelectorLink","parentSelectors":["load-more"],"selector":"a.block","multiple":false,"delay":0},{"id":"funding","type":"SelectorText","parentSelectors":["load-more"],"selector":"div.type-13:nth-of-type(1) span:nth-of-type(1)","multiple":false,"regex":"","delay":0}]}

#2

I've tried rebuilding from scratch instead of fixing what was already working and now I just get missing data in inconsistent locations. All of the data looks right when I use the data preview function of a selector but then the scrape doesn't work.

{"_id":"ks-refresh-test","startUrl":["https://www.kickstarter.com/discover/advanced?state=live&woe_id=0&pledged=3&sort=popularity&seed=2552956"],"selectors":[{"id":"load_more","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"section.pb3 div.grid-container, section.pb3","multiple":true,"delay":"5000","clickElementSelector":"div.load_more","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"project","type":"SelectorText","parentSelectors":["load_more"],"selector":"h3.type-18","multiple":true,"regex":"","delay":"2000"},{"id":"user","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.inline-block a.soft-black","multiple":true,"regex":"","delay":"2000"},{"id":"percent-funded","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.ksr-green-700 div.type-13:nth-of-type(2) span:nth-of-type(1)","multiple":true,"regex":"","delay":"2000"},{"id":"days-left","type":"SelectorText","parentSelectors":["load_more"],"selector":"span.js-num","multiple":true,"regex":"","delay":"2000"},{"id":"category","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.type-12:nth-of-type(4) a.dark-grey-500","multiple":true,"regex":"","delay":"2000"},{"id":"url","type":"SelectorLink","parentSelectors":["load_more"],"selector":"a.block","multiple":true,"delay":"2000"},{"id":"funding","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.type-13:nth-of-type(1) span:nth-of-type(1)","multiple":true,"regex":"","delay":"2000"}]}

Edit: I've played around with this some more and have found that my selectors scrape correctly when run solo but any selectors that scrape under one parent are having errors.

#3

A few fixes

1 - On you initial element click selector, the selector part was off, I fixed that
2 - On your child selectors:
A - No need t tick multiple as your parent element click selector is set to multiple
B - No need to put a delay as you have one on the parent
C - I changed your URL selector from Link to Element Attribute as I didn't see any child selector so I figured you just wanted the hyperlink URL

Looks like it works with no data loss

{"_id":"ks-refresh-test","startUrl":["https://www.kickstarter.com/discover/advanced?state=live&woe_id=0&pledged=3&sort=popularity&seed=2552956"],"selectors":[{"id":"load_more","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.black","multiple":true,"delay":"2000","clickElementSelector":"div.load_more","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"project","type":"SelectorText","parentSelectors":["load_more"],"selector":"h3.type-18","multiple":false,"regex":"","delay":""},{"id":"user","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.inline-block a.soft-black","multiple":false,"regex":"","delay":""},{"id":"percent-funded","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.ksr-green-700 div.type-13:nth-of-type(2) span:nth-of-type(1)","multiple":false,"regex":"","delay":""},{"id":"days-left","type":"SelectorText","parentSelectors":["load_more"],"selector":"span.js-num","multiple":false,"regex":"","delay":""},{"id":"category","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.type-12:nth-of-type(4) a.dark-grey-500","multiple":false,"regex":"","delay":""},{"id":"url","type":"SelectorElementAttribute","parentSelectors":["load_more"],"selector":"a.block","multiple":false,"extractAttribute":"href","delay":""},{"id":"funding","type":"SelectorText","parentSelectors":["load_more"],"selector":"div.type-13:nth-of-type(1) span:nth-of-type(1)","multiple":false,"regex":"","delay":""}]}
1 Like
#4

Thank you so much. I toiled on this for hours.

#5