HELP! Scraper does find correct links but only checks the last one?

qwerty · June 9, 2021, 7:40pm

Hi all,

I am trying to scrape some info from the following site: Search Results

I managed to let the scraper expand the list and find the desired links (namely the clickable company names). Now I want to scrape some info from the page of every company link (for simplicity lets say I want to scrape every company name from the company link), but scraper returns the name of the last company for all company links! Anyone know how to solve?

Sitemap:
{"_id":"sapcompaniesdenmark","startUrl":["Search Results p","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"SelectCompanyLinkInsideWrapper","type":"SelectorLink","parentSelectors":["ExpandListAndSelectCompanyWrappers"],"selector":".search-result__head a","multiple":false,"delay":0},{"id":"SaveName","type":"SelectorText","parentSelectors":["SelectCompanyLinkInsideWrapper"],"selector":".partner-details section:nth-of-type(1) header","multiple":false,"regex":"","delay":0}]}

ViestursWS · June 10, 2021, 1:24pm

Hi @qwerty Your sitemap did not work when I tried to copy it but I hope that I managed to figure out what you were after!

{"_id":"partneredge-sap-com","startUrl":["https://partneredge.sap.com/content/partnerfinder/search.html#/search/results?country=scm_v_country061&itemsPerPage=10&sortBy=shortname&sortOrder=asc"],"selectors":[{"id":"company-wrapper","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"li.search-results__item","multiple":true,"delay":"1200","clickElementSelector":"button.btn__show-more","clickType":"clickMore","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueHTML"},{"id":"company-name","type":"SelectorText","parentSelectors":["company-wrapper"],"selector":".search-result__head a","multiple":false,"regex":"","delay":0}]}

Asad · June 10, 2021, 1:41pm

He doesn't need that. He wants to click the link and scrape the data which is on the company page like contact name, email, etc.

List> Show more + Click On Company Name> Get Data from the linked page.

I've tried but I didn't get it through.

ViestursWS · June 10, 2021, 2:21pm

@Asad Oh, i see. I tested it with 2 of the company starting URLs and it refuses to go to the next link.

ViestursWS · June 11, 2021, 11:58am

@qwerty It seems the issue lies in the link itself because of the "#" symbol the extension has difficuilties to proceed further.

After trying different combinations in the Cloud Scraper Environment, I managed to get the result you probably went for.

qwerty · June 11, 2021, 2:45pm

Thanks, both, for your effort! @ViestursWS, do the things you did in Cloud Scraper translate into the final sitemap? Or how can I replicate your findings? Because it seems as you indeed found what I am looking for!

Thanks for your help in advance!

leemeng · June 13, 2021, 10:17am

It is probably easier to scrape this site in two stages, where in stage 1, you get all the URLs of the company pages, and then in stage 2 you have a different sitemap which uses all those stage 1 URLs as Starturls.

The company links looks like this:

#/partner/details/0000608336

so you need to prefix the site URL to turn them all to proper URLs:

https://partneredge.sap.com/content/partnerfinder/search.html#/partner/details/0000608336

Straightforward search n replace.

qwerty · June 13, 2021, 6:45pm

Thanks @leemeng for your suggestion, makes sense, however the extension still doesn't work properly, probably due to the # then as @ViestursWS said?

The output with multiple start urls still gives the output of only a single link:

leemeng · June 14, 2021, 12:17am

Hmm interesting. The server seems to be reading from a database via URLS like:
https://partneredge.sap.com/bin/partnerfinder/partnerdetails?partnerId=0000608336
https://partneredge.sap.com/bin/partnerfinder/partnerdetails?partnerId=0000922309

The company numbers at the end of the URLs match the ones you've been getting. However the data is in JSON format so you probably can't parse it with WS alone. JSON parsing is a common task for languages like Python, Java, C# etc.