I want to scrape data from a business social networking site like LinkedIn, called Xing. The idea is that I start at the root page, go to the group member page, go into the contacts list of every individual member, and scrape the data from their contacts list. So, I have to go one level deeper than what the tutorial vidoes explain. I tried doing it on my own by modifying the line of selectors to go deeper, while still keeping the same structure as in the videoes, but it only scrapes about 5 contacts from every group member. Can someone please explain to me what the issue is and how I can fix it? My selector graph looks as follows:
_root --> group members (link) --> contacts (link) --> contacts^2 (link) ---> contacts_final (element scroll down ((got the idea to do that instead of element from the forum, didn't help)) ) --> Name (text) // Company (text)
Then pagination comes after group memebers: pagination --> contacts (link) // pagination (link)
This is the URL of the root page, but you won't be able to access it because it requires that you log in: https://www.xing.com/communities/groups/vermoegensverwaltende-fonds-02f0-1006374
Sitemap: {"_id":"vvf","startUrl":["https://www.xing.com/communities/groups/vermoegensverwaltende-fonds-66cb-1006374/member_states?page=178&sorting=alphabetical"],"selectors":[{"id":"Gruppenmitglieder","type":"SelectorLink","selector":"ul.group-main-nav li:nth-of-type(4) a","parentSelectors":["_root"],"multiple":false,"delay":0},{"id":"kontakt","type":"SelectorLink","selector":"li.contact:nth-of-type(2) a.user-name","parentSelectors":["Gruppenmitglieder","pagination"],"multiple":true,"delay":0},{"id":"pagination","type":"SelectorLink","selector":"ul.pagination-pages li:nth-of-type(n+8) a, a.foundation-icon-shape-arrow-right","parentSelectors":["Gruppenmitglieder","pagination"],"multiple":true,"delay":0},{"id":"kontakte^2","type":"SelectorLink","selector":"a.src-TabBar-tab-c2e0f069:nth-of-type(3)","parentSelectors":["kontakt"],"multiple":false,"delay":0},{"id":"kontakt_final","type":"SelectorElementScroll","selector":"div#A.profile-contacts-index-contactsContainerLetterWrapper-142d9fc3 div.malt-grid-index-col-0979e3a1","parentSelectors":["kontakte^2"],"multiple":true,"delay":0},{"id":"Nam","type":"SelectorText","selector":"h3.malt-typography-index-headline3-26856c49","parentSelectors":["kontakt_final"],"multiple":false,"regex":"","delay":0},{"id":"Firma","type":"SelectorText","selector":"span.malt-user-info-UserInfo-company-3d4bf5f1","parentSelectors":["kontakt_final"],"multiple":false,"regex":"","delay":0}]}
Thanks for the help!