Hello,
Linkedin profiles are accessible without being logged in. Of course, after several pages, the "auth wall" appears and it is not possible to scrape anymore. But with rotating IPs, I guess it is possible to keep scraping.
Also, perhaps it is possible to use a Linkedin API endpoint to ease this scraping.
Do you have any tips here to help scrape high volumes of profiles (hundreds of thousands par week)?
The simple answer is to just create a dummy account used only for scraping. Do not use your actual account. You'll also want to slow down your scraping speed; most sites will check for excessive traffic coming from 1 IP.
For LinkedIn, about 12 pages a minute should be OK, if I recall. I once scraped 50K+ profiles over two days with 2 machines.
Thanks for your answer!
When did you succeed to scrape 50K profiles over 2 days with 2 machines? Was it a long time ago? Because now, it seems that it is not possible anymore...
It must have been late 2021 I think. I have not scraped LinkedIn at that scale since then. I was using a dummy account though; if you try to scrape a lot without logging in, it would surely be flagged as scraper activity.