Scrape a pop-up of a pop-up window in Kickstarter

[Paul]

Hello everyone,
for my Master Thesis I have to scrape kickstarter projects. So far it worked very well - love this tool. But now I am running into some problems, which are substantial for my thesis:
I want to know whether previously created projects (if there are any) of the user were successful or not. With two pop-ups I get an issue: In the project the web-scraper should click on the Usersname (pop-up opens), click on "previously created projects" (new windows open) and then navigate throw all of these projects to see whether they were successful or not. However, it seems like the second (new windows with all previously created projects) does not open and I don't understand why. I tried a lot of variations but it simply does not work, I really would appreciate help!
Thank you very much in advance!

Here is an example page I would like to scrape. :

UNICORN - GoPro Mount for Mountainbike Handlebars

Sitemap:
{"_id":"test_erstelle_projekte_3","startUrl":["https://www.kickstarter.com/projects/ninjamount/unicorn-gopro-mount-for-mountainbike-handlebars"],"selectors":[{"id":"click_erstellt_von_1","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"div.text-left.type-16","multiple":false,"delay":0},{"id":"click_erstellt_von_2","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"a.ksr-green-500:nth-of-type(1)","multiple":false,"delay":0},{"id":"select_all_previosly_created_projects","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.mb3","multiple":true,"delay":0},{"id":"days_untill_end","type":"SelectorText","parentSelectors":["select_all_previosly_created_projects"],"selector":".grid-col-4-md span.block.type-16","multiple":false,"regex":"","delay":0},{"id":"financed","type":"SelectorText","parentSelectors":["select_all_previosly_created_projects"],"selector":".mb0 span","multiple":false,"regex":"","delay":0},{"id":"finance_goal","type":"SelectorText","parentSelectors":["select_all_previosly_created_projects"],"selector":".type-12 span.money","multiple":false,"regex":"","delay":0},{"id":"text_failed_cancelled","type":"SelectorText","parentSelectors":["select_all_previosly_created_projects"],"selector":".grid-col-4-md .border div","multiple":false,"regex":"","delay":0}]}

[Paul]

I figured it out in the moment. The second pop up needs to be a link selector (not a popup). This already confused me in the past and I actually don't really understand why this is the case. Maybe someone can explain to me, when to use what.

Attached the updated sitemap (in case someone is interested):
{"_id":"test_erstelle_projekte_4","startUrl":["https://www.kickstarter.com/projects/675453850/kyteme-ultracompact-multifunction-sport-jacket-cyc/description"],"selectors":[{"id":"click_erstellt_von_1","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"div.text-left.type-16","multiple":false,"delay":0},{"id":"click_erstellt_von_2","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.ksr-green-500:nth-of-type(1)","multiple":false,"delay":0},{"id":"select_each_project_already_created","type":"SelectorLink","parentSelectors":["click_erstellt_von_2"],"selector":"a.mb3","multiple":true,"delay":0},{"id":"verbleibende_Tage_ongoing","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".grid-col-4-md span.block.type-16","multiple":false,"regex":"","delay":0},{"id":"finance_goal","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".type-12 span.money","multiple":false,"regex":"","delay":0},{"id":"finance_archieved","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".mb0 span","multiple":false,"regex":"","delay":0},{"id":"meldung_fehlgeschlagen","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".grid-col-4-md div.normal","multiple":false,"regex":"","delay":0}]}

[webber]

The first action requires an 'Element Click' as there is no links involved, just some JS that needs to be executed.

The second one is a link selector - that you got right.

The reason it is not a pop-up is because it opens in a new tab, not a window. A pop-up is needed if the scraper stays on the same page, but opens a new 'window' on the same page. Here is an example on how a pop-up link would open:

This sitemap should work:

{"_id":"test_erstelle_projekte_4","startUrl":["https://www.kickstarter.com/projects/675453850/kyteme-ultracompact-multifunction-sport-jacket-cyc/description"],"selectors":[{"id":"click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.py10-sm","multiple":false,"delay":"2000","clickElementSelector":"div.text-left.type-16","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"click_erstellt_von_2","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.ksr-green-500:nth-of-type(1)","multiple":false,"delay":0},{"id":"select_each_project_already_created","type":"SelectorLink","parentSelectors":["click_erstellt_von_2"],"selector":"a.mb3","multiple":true,"delay":0},{"id":"verbleibende_Tage_ongoing","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".grid-col-4-md span.block.type-16","multiple":false,"regex":"","delay":0},{"id":"finance_goal","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".type-12 span.money","multiple":false,"regex":"","delay":0},{"id":"finance_archieved","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".mb0 span","multiple":false,"regex":"","delay":0},{"id":"meldung_fehlgeschlagen","type":"SelectorText","parentSelectors":["select_each_project_already_created"],"selector":".grid-col-4-md div.normal","multiple":false,"regex":"","delay":0}]}

[Paul]

Ah wow, now I got this element click thing as well. The official tutorial is very confusing in regard to element click (it only talks about pagination) and popup link (it says popup as well as new windows). But anyways. Thanks a lot!!

One more question where you can maybe help me. When I am now using the right solution (yours) it lists each project as a new row. It is, however, very tricky to clean the data afterwards. Do you maybe know whether it is possible that I have one row and then a new column within the project, which lists previously created projects?

It should look sth like this:

It's of course now not about that it should says "successful" or not (I will do a workaround in the data cleaning), but just that I don't have each project in a new row but just an additional column instead.

Thank you very much for your help!

Best,
Paul

[czenci]

Dear Paul, I need the same thing:) I tried it in the summer and it worked perfectly (I only had problems with the security captcha), but now it doesn't work... It looks like it's running, but no data is scraped. Any idea why? Thank you for your help!