I need to scrape the data from this website, but it's proving tricky for me because the navigation drop-down menu I need, the year, is under a selector tag instead of a div. Does anybody have any suggestions for how I can grab the title, speaker, and date of every devotional in this database?
Url: Speeches
@HostileSpider Hello, in order to click through all of the available year options, you can use the 'Element click' selector: select[data-ng-model="year"] option:not(:contains("All Years"))
Here's a sitemap example:
{"_id":"web-byui-edu","startUrl":["https://web.byui.edu/DevotionalsAndSpeeches/"],"selectors":[{"clickElementSelector":"select[data-ng-model=\"year\"] option:not(:contains(\"All Years\"))","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1200,"discardInitialElements":"discard-when-click-element-exists","id":"years","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"id":"wrapper","multiple":true,"parentSelectors":["years"],"selector":"li.ng-scope","type":"SelectorElement"},{"id":"speaker","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"div.speaker","type":"SelectorText"},{"id":"title","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"div.speechTitle","type":"SelectorText"},{"id":"date","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"div.speechDate","type":"SelectorText"}]}
Please, note that when using the Pagination(with the click type), Element Click, or Element Scroll selectors, the extracted data only becomes available after the respective selector has finished (no new elements are being matched).
