Https://de.trustpilot.com/categories/animals_and_pets

How to?
#1

I have used web scraper successfully on other pages but this one does not seem to work. As soon as the selector shop is clicked the scraper stops functioning (as far as I can tell). It does not stop but it seems to bring issues with the pop up link and no data is collected. The sitemap below is just a small example of how we would like to scrape. Help would be greatly appreciated!!

Url: https://de.trustpilot.com/categories/animals_and_pets

Sitemap:
{"_id":"trustest","startUrl":["https://de.trustpilot.com/categories/animals_and_pets"],"selectors":[{"id":"shop","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"a.category-business-card","multiple":true,"delay":0},{"id":"d_name","type":"SelectorText","parentSelectors":["shop"],"selector":"span.multi-size-header__big","multiple":false,"regex":"","delay":0},{"id":"d_mail","type":"SelectorText","parentSelectors":["shop"],"selector":".contact-point__details a","multiple":false,"regex":"","delay":0}]}

https://de.trustpilot.com/categories/animals_and_pets

#2

This requires just a regular Link selector and not a pop-up.

Link selector - follows a link or opens it in a new tab (Like in the example above)
Popup Link selector - opens a link in a new browser window

#3

Thanks a lot for your reply @webber . Thought I'd get a mail notification, that's why I only comment this late.
Changed it to link but it still something seems to be wrong. The scraping is set to 2000 ms interval and 2000 ms page load delay but the pages only load every 30 seconds (roughly) and no data is being scraped. Could you have another look with the the updated link selector?

#4

Is there a log file that gets created when scraping? Maybe this could give me some more insight of why it is not working.

#5

This seems to work:

{"_id":"trustest-2","startUrl":["https://de.trustpilot.com/categories/animals_and_pets"],"selectors":[{"id":"shop","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.category-business-card","multiple":true,"delay":0},{"id":"d_name","type":"SelectorText","parentSelectors":["shop"],"selector":"span.multi-size-header__big","multiple":false,"regex":"","delay":0},{"id":"d_mail","type":"SelectorText","parentSelectors":["shop"],"selector":".contact-point__details a","multiple":false,"regex":"","delay":0}]}

#6

Tried your sitemap but have the same issues. It scrawls the pages but data is empty.

Keep getting this:
image

#7

In order to show the issue better, here's a sitemap of only one page. Something seems to prevent the process of scraping/saving the data. The example sitemap loads one page and should extract to data points. When scraping it stays on the page for 30 seconds or so and once done it still says "No data scraped yet."

{"_id":"testpage","startUrl":["https://de.trustpilot.com/review/maledeinleben.com"],"selectors":[{"id":"name","type":"SelectorText","parentSelectors":["_root"],"selector":"span.multi-size-header__big","multiple":false,"regex":"","delay":0},{"id":"mail","type":"SelectorText","parentSelectors":["_root"],"selector":"div.contact-point:nth-of-type(1) a","multiple":false,"regex":"","delay":0}]}
#8

I have found a solution by simply changing the browser from Chrome to Firefox. For some reason something in Chrome prevents the site from being scraped. With Firefox it simply works.