--> Problem found, solution missing --> Bug? Same username, no data saved . <---
Hello there,
for my master thesis I want to analyze Kickstarter campaigns. The scraper is already programmed and when I am running small samples (1-2 Pages), everything gets downloaded completely. However, as soon as I want to download a larger sample, I am having a massive data loss and roughly 25-50% of the projects do not get downloaded at all. This is of course very problematic and my whole thesis is on risk by now. I hope someone can help me.
I played around with the delay time a lot, however, so far it did not really help much. Furthermore, I figured that the missing projects are random to a certain degree. Most promising was a delay for the Element Click selector of 3000 and a request interval and Page load delay of 3000.
To be honest, I understand what the request and page load delay is, but I am not quite sure how this plays together with the delay selected at the Element Click selector.
Should I massively increase one of the delays? And if yes - which one and which delay? However, I also tried a very high number and besides, of course, it takes incredible long, it happens that nothing at all gets scraped. Or what are other ideas / workarounds to avoid this massive data loss?
I'm navigating throw the sides using the page numbers [1-88]. I also tested an element click selector, but the results were even worth. I also tried to optimize the order by rearranging the code.
I want to scape page 1 – 88:
https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=23424829&sort=popularity&seed=2596372&page
Would really really appreciate some help. Thank you very much in advance.
My sitemap:
{"_id":"kickstarter_technology_germany_2019_06_21_version_2","startUrl":["https://www.kickstarter.com/discover/advanced?category_id=16&woe_id=23424829&sort=popularity&seed=2596372&page=[1-88"],"selectors":[{"id":"select_project","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.mb3","multiple":true,"delay":3000},
{"id":"finanzierungszeitraum","type":"SelectorText","parentSelectors":["select_project"],"selector":"p.f5","multiple":false,"regex":"","delay":0},
{"id":"success_financed","type":"SelectorText","parentSelectors":["select_project"],"selector":".mb0 span","multiple":false,"regex":"","delay":0},
{"id":"success_fiance_goal","type":"SelectorText","parentSelectors":["select_project"],"selector":".type-12 span.money","multiple":false,"regex":"","delay":0},
{"id":"anzahl_unterstuetzer_gesamt","type":"SelectorText","parentSelectors":["select_project"],"selector":".mb0 h3","multiple":false,"regex":"","delay":0},
{"id":"ueber_uns_tag_1","type":"SelectorText","parentSelectors":["select_project"],"selector":"a.grey-dark.flex","multiple":false,"regex":"","delay":0},
{"id":"ueber_uns_tag_2","type":"SelectorText","parentSelectors":["select_project"],"selector":"a.grey-dark:nth-of-type(2)","multiple":false,"regex":"","delay":0},
{"id":"ueber_uns_tag_3","type":"SelectorText","parentSelectors":["select_project"],"selector":"a.grey-dark:nth-of-type(3)","multiple":false,"regex":"","delay":0},
{"id":"ueber_uns_tag_4","type":"SelectorText","parentSelectors":["select_project"],"selector":"a.mr3:nth-of-type(1)","multiple":false,"regex":"","delay":0},
{"id":"number_FAQ","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-load-project-faqs span","multiple":false,"regex":"","delay":0},
{"id":"number_updates","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-load-project-updates span","multiple":false,"regex":"","delay":0},
{"id":"number_comments","type":"SelectorText","parentSelectors":["select_project"],"selector":"data","multiple":false,"regex":"","delay":0},
{"id":"failed_financed","type":"SelectorText","parentSelectors":["select_project"],"selector":".grid-col-4-md span.soft-black","multiple":false,"regex":"","delay":0},
{"id":"failed_finance_goal","type":"SelectorText","parentSelectors":["select_project"],"selector":".grid-col-4-md .inline-block-sm span","multiple":false,"regex":"","delay":0},
{"id":"failed_notification","type":"SelectorText","parentSelectors":["select_project"],"selector":".grid-col-4-md div.border","multiple":false,"regex":"","delay":0},
{"id":"failed_anzahl_unterstutzer_geasmt","type":"SelectorText","parentSelectors":["select_project"],"selector":".grid-col-4-md .block.type-16 span","multiple":false,"regex":"","delay":0},
{"id":"video_top","type":"SelectorText","parentSelectors":["select_project"],"selector":"button.m-auto","multiple":false,"regex":"","delay":0},
{"id":"video_middle_abspielen","type":"SelectorText","parentSelectors":["select_project"],"selector":"button.play_button_big","multiple":false,"regex":"","delay":0},
{"id":"Erstellt_von_successful ","type":"SelectorText","parentSelectors":["select_project"],"selector":"a.hero__link.remote_modal_dialog","multiple":false,"regex":"","delay":0},
{"id":"landing_page_image","type":"SelectorImage","parentSelectors":["select_project"],"selector":"img.js-feature-image","multiple":false,"delay":0},
{"id":"description_text_images","type":"SelectorGroup","parentSelectors":["select_project"],"selector":"img","delay":0,"extractAttribute":"src"},
{"id":"desciption_text_top","type":"SelectorText","parentSelectors":["select_project"],"selector":".full-description p:nth-of-type(1), div.full-description","multiple":false,"regex":"","delay":0},
{"id":"desciption_text_risiken und herausforderungen","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.js-risks","multiple":false,"regex":"","delay":0},
{"id":"failed_select_erstellt_von","type":"SelectorElementClick","parentSelectors":["select_project"],"selector":"div.pb3, div.py10-sm","multiple":false,"delay":3000,"clickElementSelector":"div.text-left.type-16","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},
{"id":"failed_erstellt_von ","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.text-left.type-16","multiple":false,"regex":"","delay":0},
{"id":"failed_facebook","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.flex:nth-of-type(3) span","multiple":false,"regex":"","delay":0},
{"id":"ueber_uns_tag_6","type":"SelectorText","parentSelectors":["select_project"],"selector":".hide a:nth-of-type(2) span","multiple":false,"regex":"","delay":0},
{"id":"failed_erstelle_UND_untersütze_projekte","type":"SelectorText","parentSelectors":["select_project"],"selector":"div:nth-of-type(4) > span","multiple":false,"regex":"","delay":0},
{"id":"failed_name_hiner_account","type":"SelectorText","parentSelectors":["select_project"],"selector":".flex.soft-black span","multiple":false,"regex":"","delay":0},
{"id":"failed_select_name_for_profile_picture","type":"SelectorLink","parentSelectors":["select_project"],"selector":".mb0 a","multiple":false,"delay":0},
{"id":"failed_profile_picture","type":"SelectorImage","parentSelectors":["failed_select_name_for_profile_picture"],"selector":"img.avatar-image","multiple":false,"delay":0},
{"id":"select_erstellt_von_success","type":"SelectorElementClick","parentSelectors":["select_project"],"selector":"div.pb3, div.py10-sm","multiple":false,"delay":3000,"clickElementSelector":"a.hero__link.remote_modal_dialog","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},
{"id":"erstelle_UND_untersütze_projekte","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.created-projects","multiple":false,"regex":"","delay":0},
{"id":"name_hiner_account","type":"SelectorText","parentSelectors":["select_project"],"selector":"span.identity_name","multiple":false,"regex":"","delay":0},
{"id":"mitarbeiter_1","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.flag:nth-of-type(1) .flag-body a","multiple":false,"regex":"","delay":0},
{"id":"rolle_mitarbeiter_1","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.flag:nth-of-type(1) div.type-sm--sans","multiple":false,"regex":"","delay":0},
{"id":"mitarbeiter_2","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.col:nth-of-type(2) .flag-body a","multiple":false,"regex":"","delay":0},
{"id":"rolle_mitarbeiter_2","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.flag:nth-of-type(2) div.type-sm--sans","multiple":false,"regex":"","delay":0},
{"id":"mitarbeiter_3","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.flag:nth-of-type(3) a","multiple":false,"regex":"","delay":0},
{"id":"rolle_mitarbeiter_3","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.flag:nth-of-type(3) div.type-sm--sans","multiple":false,"regex":"","delay":0},
{"id":"mitarbeiter_4","type":"SelectorText","parentSelectors":["select_project"],"selector":"div:nth-of-type(4) .flag-body a","multiple":false,"regex":"","delay":0},
{"id":"rolle_mitarbeiter_4","type":"SelectorText","parentSelectors":["select_project"],"selector":"div:nth-of-type(4) div.type-sm--sans","multiple":false,"regex":"","delay":0},
{"id":"facebook_account","type":"SelectorText","parentSelectors":["select_project"],"selector":"div.facebook.py2","multiple":false,"regex":"","delay":0},
{"id":"success_select_name_for_profile_picture","type":"SelectorLink","parentSelectors":["select_project"],"selector":".f2 a","multiple":false,"delay":0},
{"id":"success_profile_picture","type":"SelectorImage","parentSelectors":["success_select_name_for_profile_picture"],"selector":"img.avatar-image","multiple":false,"delay":0},
{"id":"select_community","type":"SelectorElementClick","parentSelectors":["select_project"],"selector":".js-project-nav-scroll div.row","multiple":false,"delay":3000,"clickElementSelector":"a.js-load-project-community","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},
{"id":"unterstuetzer_land_1","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(1) .left a","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_anzahl_1","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(1) div.tertiary-text","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_land_2","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(2) a","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_anzahl_2","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(2) .right div","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_land_3","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(3) a","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_anzahl_3","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(3) div.tertiary-text","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_land_4","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(4) a","multiple":false,"regex":"","delay":0},
{"id":"unterstuetzer_anzahl_4","type":"SelectorText","parentSelectors":["select_project"],"selector":".js-locations-countries div:nth-of-type(4) div.tertiary-text","multiple":false,"regex":"","delay":0},
{"id":"select_comments","type":"SelectorElementClick","parentSelectors":["select_project"],"selector":"div.project-nav__links","multiple":false,"delay":3000,"clickElementSelector":"a.js-load-project-comments","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},
{"id":"scroll_down_comments","type":"SelectorElementClick","parentSelectors":["select_project"],"selector":"ul.bg-grey-100","multiple":false,"delay":0,"clickElementSelector":"button.bttn-blue","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},
{"id":"comments","type":"SelectorText","parentSelectors":["select_project"],"selector":"ul.bg-grey-100","multiple":false,"regex":"","delay":0}
]}