Infinite scroll LinkedIn: sitemap not working if more than N contacts

How to?
#1

*** Describe the problem ***
The following sitemap works perfectly if there're less than 600 elements.
If there are more – that same sitemap finishes scrapping but doesn't return any data.

The page is structured with an infinite scroll. Initially 40 elements (contacts) are displayed and with every scroll 40 more are displayed.

At the beginning I thought that the issue could be caused by the number of contacts. I have more than 2,000 contacts so it takes a bit longer to load new contacts every time that a new scroll is done. I tried the scrapping with the following settings:
– Request interval (ms): 10000
– Page load delay (ms): 10000

And I also tried adding a delay in the
– Type: Element Scroll Down (Delay (ms): 10000

*** QUESTIONS ***
(1) If it's due to the page load; wouldn't it be reasonable to get data even if it stops before getting to the last contact? Currently the sitemap returns no data with long lists
(2) What could be causing the sitemap to work perfectly with short lists of elements and not getting any data with long lists?

Thanks!

*** Url ***
www. linkedin.com/mynetwork/invite-connect/connections/

*** Sitemap ***
{"_id":"linkedin-contacts","startUrl":["https://www.linkedin.com/mynetwork/invite-connect/connections/"],"selectors":[{"id":"element-contacto","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.mn-connection-card__details","multiple":true,"delay":"2000"},{"id":"url","type":"SelectorLink","parentSelectors":["element-contacto"],"selector":"a","multiple":false,"delay":0}]}

#2

Ya you can try increasing the scroller delay, and also a couple of related scroller tricks:

  • Separate the scroller from the data scrapers
  • Limit the scroller with nth-of-type (in the example below, it stops at 1,001)

{"_id":"forum-linkedin-contacts","startUrl":["https://www.linkedin.com/mynetwork/invite-connect/connections/"],"selectors":[{"id":"Separate scroller","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"ul > li.list-style-none:nth-of-type(-n+1001)","multiple":true,"delay":"3500"},{"id":"element-contacto","type":"SelectorElement","parentSelectors":["_root"],"selector":"div.mn-connection-card__details","multiple":true,"delay":""},{"id":"url","type":"SelectorLink","parentSelectors":["element-contacto"],"selector":"a","multiple":false,"delay":0}]}

#3

@leemeng the same issue happens with your sitemap.

I used your sitemap in an account with 300 contacts and worked fine.
I used it in an account with +1,000 contacts and it stops scrolling after a while – without returning any data.

  • The "ul > li.list-style-none:nth-of-type(-n+600)" works
  • The "ul > li.list-style-none:nth-of-type(n+600)" doesn't work

My "dirty" solution:
(1) I'm scrolling down using the "DATA PREVIEW" button.
It scrolls down several times. I repeat the process a few times until I get to the bottom.

image

(2) I copy-paste the table with the data from the DATA PREVIEW

The more contacts that are added, the longer that LinkedIn needs to display new contacts. Would it be possible that the scrapping process crashes because the loading time is longer than the delay that's been set?
If this happens – would it be reasonable to think that the crapping should return the data from the elements displayed until that point?

Could it be possible that LinkedIn is deliberately blocking the scraping?