How can scrape article link from this page. This page has infinity scroll and article count is 47,645 articles. How can I scrape all the article link .
Url: Experimental Biology and Medicine | Articles
Sitemap:
{"_id":"EMB","startUrl":["https://www.ebm-journal.org/journals/experimental-biology-and-medicine/articles"],"selectors":[{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":"a.CardArticle__wrapper","type":"SelectorLink"}]}
Hi, just add a scroll to the sitemap:
{"_id":"EMB","startUrl":["https://www.ebm-journal.org/journals/experimental-biology-and-medicine/articles"],"selectors":[{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["wrapper"],"selector":"a.CardArticle__wrapper","type":"SelectorLink"},{"delay":2000,"elementLimit":100,"id":"wrapper","multiple":true,"parentSelectors":["_root"],"selector":"article","type":"SelectorElementScroll"}]}
Leave the 'Element limit' field blank for unlimited elements.
it works perfectly. Thanks man
@JanAp when I go for unlimited or limited data like 47000. Scraper is running 1/2 hour after finished no data showing.
Data will be returned only after all records are gathered.
Given the large data set, it might be possible that your computing resources are not sufficient to complete the job.
You try with a smaller number and see what happens, lets say 10,000
Okay thanx for the info. Let me check