Describe the problem.
Hey guys,
I have been working on scraping blogs in the recent weeks, but for some reason I have some troubles with scraping the blog of the enclosed link. I have integrted the elementclick selector, to identify all link blogs, but somehow the scraper does not scrape all the links, but stops after 130 links.
Does anyone have a hint for me?
Best
Tevin
Url: https://www.technologyreview.com/topic/business-impact/
Sitemap:
{"_id":"forum_mit-tech_business-impact","startUrl":["https://www.technologyreview.com/topic/business-impact/"],"selectors":[{"id":"linktoblogs","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.feed-tz__hgroup","multiple":true,"delay":"5000","clickElementSelector":"button.btn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"element_link","type":"SelectorLink","parentSelectors":["linktoblogs"],"selector":"a.feed-tz__title__link","multiple":false,"delay":0},{"id":"blog_title","type":"SelectorText","parentSelectors":["element_link"],"selector":"h1.article-topper__title","multiple":false,"regex":"","delay":0},{"id":"blog_date","type":"SelectorText","parentSelectors":["element_link"],"selector":"li.article-topper__meta-item:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"blog_subtitle","type":"SelectorText","parentSelectors":["element_link"],"selector":"h2.article-topper__subtitle","multiple":false,"regex":"","delay":0},{"id":"blog_editor","type":"SelectorText","parentSelectors":["element_link"],"selector":"li.article-topper__meta-item a","multiple":false,"regex":"","delay":0},{"id":"blog_text","type":"SelectorText","parentSelectors":["element_link"],"selector":"div.article-body__content > p","multiple":true,"regex":"","delay":0}]}