Hi,
I have a sitemap that only traverses a webiste partially before finishing the scrap prematurely. What I want is to click on each subject area, following the link the course's webpage and extract some the 2018-19 and 2019-20 fees from the 'Fees and funding' tab. It works for the large majority of subjects but stops when it gets to 'Social Work'!?!
{"_id":"course_fees_june_2018","startUrl":["https://www.southwales.ac.uk/courses/"],"selectors":[{"id":"subject_dropdowns","type":"SelectorElementClick","selector":"div.course-listing div.container","parentSelectors":["_root"],"multiple":false,"delay":0,"clickElementSelector":"a.accordion-toggle.collapsed","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"course_link","type":"SelectorLink","selector":"div.panel-collapse.in a","parentSelectors":["subject_dropdowns"],"multiple":true,"delay":"20"},{"id":"fee_dropdown","type":"SelectorElementClick","selector":"section.about div.col-md-8","parentSelectors":["course_link"],"multiple":false,"delay":0,"clickElementSelector":"div.panel:nth-of-type(4) h4.panel-title a","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"year","type":"SelectorText","selector":"div.course_fees h3","parentSelectors":["fee_dropdown"],"multiple":false,"regex":"","delay":0},{"id":"cost","type":"SelectorText","selector":"ul.list-unstyled p, ul.list-unstyled li","parentSelectors":["fee_dropdown"],"multiple":false,"regex":"","delay":0},{"id":"year_2","type":"SelectorText","selector":"div.course_fees h3:nth-of-type(2)","parentSelectors":["fee_dropdown"],"multiple":false,"regex":"","delay":0},{"id":"cost_2","type":"SelectorText","selector":"ul.list-unstyled li","parentSelectors":["fee_dropdown"],"multiple":false,"regex":"","delay":0}]}
There are no error messages, the Webscraper.io appears to think it has finished navigating the website but I don't know why. Can anyone advise, please?