Scrape is missing about a third of the data

How to?
#1

I am trying to scrape just the names from the results list of a search of people on LinkedIn. The results show 383 results but Web Scraper only scrapes 190 names. Am I doing something wrong?

Url: https://www.linkedin.com/search/results/people/?facetCurrentCompany=["1766"%2C"1762"%2C"1765"%2C"533177"%2C"63819431"]&facetGeoRegion=["gb%3A4608"]&keywords=BBC%20journalist&origin=FACETED_SEARCH

Sitemap:
{"_id":"bbcsalfordpeeps","startUrl":["https://www.linkedin.com/search/results/people/?facetCurrentCompany=["1766"%2C"1762"%2C"1765"%2C"533177"%2C"63819431"]&facetGeoRegion=["gb%3A4608"]&keywords=BBC%20journalist&origin=FACETED_SEARCH&page=[1-39]"],"selectors":[{"id":"Name","type":"SelectorText","parentSelectors":["_root"],"selector":"span.name","multiple":true,"regex":"","delay":0}]}

#2

LinkedIn (and many other sites) use lazy loading, so you would need a scroller on each page to properly load all data, i.e. if you don't scroll down, the bottom data does not get loaded.

#3

Thank you, that is really helpful. I modified my sitemap as follows:

{"_id":"bbcsalfordpeeps","startUrl":["https://www.linkedin.com/search/results/people/?facetCurrentCompany=["1766"%2C"1762"%2C"1765"%2C"533177"%2C"63819431"]&facetGeoRegion=["gb%3A4608"]&keywords=BBC%20journalist&origin=FACETED_SEARCH&page=[1-39]"],"selectors":[{"id":"Scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"span.name","multiple":true,"delay":"500"},{"id":"Name","type":"SelectorText","parentSelectors":["_root"],"selector":"span.name","multiple":true,"regex":"","delay":0}]}

This scraped 345 names but still not all of the names in the search (383). So a lot closer, but still not 100%. Does anyone have any other tips/suggestions please?

#4

Your scrapers should normally reside under the scroller (child of the scroller). Try the sitemap below. I had to change the name selector for my test cos of different layout (due to either regional version or subsciption status). I used Page load delay 5000.

{"_id":"forum-bbcsalfordpeeps","startUrl":["https://www.linkedin.com/search/results/people/?facetCurrentCompany=%5B%221766%22%2C%221762%22%2C%221765%22%2C%22533177%22%2C%2263819431%22%5D&facetGeoRegion=%5B%22gb%3A4608%22%5D&keywords=BBC%20journalist&origin=FACETED_SEARCH","https://www.linkedin.com/search/results/people/?facetCurrentCompany=%5B%221766%22%2C%221762%22%2C%221765%22%2C%22533177%22%2C%2263819431%22%5D&facetGeoRegion=%5B%22gb%3A4608%22%5D&keywords=BBC%20journalist&origin=FACETED_SEARCH&page=[2-39]"],"selectors":[{"id":"Scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.search-result__wrapper","multiple":true,"delay":"600"},{"id":"Name","type":"SelectorText","parentSelectors":["Scroll"],"selector":"span > span.name.actor-name,span.actor-name","multiple":false,"regex":"","delay":0},{"id":"Position","type":"SelectorText","parentSelectors":["Scroll"],"selector":"p.subline-level-1 > span[dir]","multiple":false,"regex":"","delay":0}]}

#5

Thank you so much for your help, Lee. That's nailed it. Here's my version that works for me (using page load delay 5000 as you suggested):

{"_id":"bbcbodssalford","startUrl":["https://www.linkedin.com/search/results/people/?facetCurrentCompany=["1762"%2C"1765"%2C"533177"%2C"63819431"]&facetGeoRegion=["gb%3A4608"]&keywords=BBC%20journalist&origin=FACETED_SEARCH&page=[1-39]"],"selectors":[{"id":"Scroller","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.search-result__wrapper","multiple":true,"delay":"600"},{"id":"Name","type":"SelectorText","parentSelectors":["Scroller"],"selector":"span.name","multiple":true,"regex":"","delay":0}]}