Loop over dropdown-menu when search results open in a new page?

How to?
#1

Dear Scraper Community,

I am having difficulties looping over a dropdown menu of a search page that opens the results in a new page. Specifically, I want my scraper to select the dropdown-option 'monday' from the weekday menu (german: 'Montag' option from 'Wochentag' menu) > then click 'Search' > scrape the results (seminar titles). Do the same for Tuesday, Wednesday etc.

URL: Suche nach Veranstaltungen - Leibniz Universität Hannover

Although my wrapper for the dropdown menu successfully selects all weekdays (as visible in the element preview) the Web Scraper ends after scraping the results for monday, without proceeding to tuesday and all other days.

Do you guys have an idea how to fix that?

Many thanks in advance!

Sitemap:
{"_id":"Uni_Hannover_Veranstaltungssuche","startUrl":["Suche nach Veranstaltungen - Leibniz Universität Hannover Wochentag","parentSelectors":["_root"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"select#k_wochentag\.wochentagid option:not(:contains('Alles auswählen'))","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard-when-click-element-exists","multiple":true,"selector":"html"},{"id":"Click Search","parentSelectors":["Select Wochentag"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"input[name='search_start'][tabindex]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":1000,"discardInitialElements":"do-not-discard","multiple":false,"selector":"html"},{"id":"elements wrapper","parentSelectors":["Select Wochentag"],"type":"SelectorLink","selector":"td a","multiple":true,"linkType":"linkFromHref"},{"id":"Veranstaltungsname","parentSelectors":["elements wrapper"],"type":"SelectorText","selector":"h1","multiple":false,"regex":""}]}

#2

I don't really understand, why do you need to click on each day + Search button instead of copying respective URL for each day....
Check it out:

{"_id":"Uni_Hannover_Veranstaltungssuche","startUrl":["https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=1&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display","https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=2&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display","https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=3&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display","https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=4&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display","https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=5&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display","https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=6&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display","https://qis.verwaltung.uni-hannover.de/qisserver/rds?state=wsearchv&search=1&subdir=veranstaltung&choice.k_wochentag.wochentagid=y&k_wochentag.wochentagid=7&veranstaltung.semester=20241&P_start=0&P_anzahl=10&P.sort=&_form=display"],"selectors":[{"id":"element","multiple":true,"parentSelectors":["next"],"selector":"tr:nth-of-type(n+2)","type":"SelectorElement"},{"clickActionType":"real","clickElementSelector":"input[name='search_start'][tabindex]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":1000,"discardInitialElements":"do-not-discard","id":"Click Search","multiple":false,"parentSelectors":["Select Wochentag"],"selector":"html","type":"SelectorElementClick"},{"id":"elements wrapper","linkType":"linkFromHref","multiple":true,"parentSelectors":["Select Wochentag"],"selector":"td a","type":"SelectorLink"},{"id":"Veranstaltungsname","multiple":false,"parentSelectors":["elements wrapper"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"vst-nr","multiple":false,"parentSelectors":["element"],"regex":"","selector":"td:nth-of-type(1)","type":"SelectorText"},{"id":"veranstaltung","linkType":"linkFromHref","multiple":false,"parentSelectors":["element"],"selector":"td:nth-of-type(2) a","type":"SelectorLink"},{"id":"vst-art","multiple":false,"parentSelectors":["element"],"regex":"","selector":"td:nth-of-type(3)","type":"SelectorText"},{"id":"next","paginationType":"auto","parentSelectors":["_root","next"],"selector":"a.linkAsButton","type":"SelectorPagination"}]}
#3

The URL seems to change along with the search options, so you can experiment to figure out the sequence and just use those as the Start URLs. Meaning, no need to click on dropdowns. E,g. there is only 1 character difference in URL for Monday and Tuesday:

image
image884×336 19.3 KB

#4

Thank you, that works! I didn't know you can have multiple start URLs.