Hello,
I am trying to scrape the headlines and time stamps for all articles listed by a news website's search results.
This is the link I would like to scrape: https://www.cbc.ca/search?q=racism§ion=news&sortOrder=date&media=all
The page has a load more that I have been able to get to work using a click element selector but whenever I run the scraper it abruptly ends after loading nearly 10% of the searches without scraping any data.
{"_id":"cbc5","startUrl":["https://www.cbc.ca/search?q=racism§ion=news&sortOrder=date&media=all"],"selectors":[{"id":"main1","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.contentListCards","multiple":true,"delay":"3000","clickElementSelector":"div > button[class^='sclt-loadmore']","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTML"},{"id":"sub","type":"SelectorElement","parentSelectors":["main1"],"selector":"div.card-content","multiple":true,"delay":0},{"id":"info","type":"SelectorGroup","parentSelectors":["sub"],"selector":"h3, time","delay":0,"extractAttribute":""}]}
Could somebody please help me get this thing working? I have been stuck on this for quite some time now.
Thank you!