Scraping from sitemap.xml links fails

How to?
#1

This runs for a few hours but then i get the aw snap error. No reason i can see for it failing but i'm very new to this whole scraping gig so any pointers would be very much appreciated, this is the sitemap:

{"_id":"PaperstoneSiteMap","startUrl":["https://paperstone.co.uk"],"selectors":[{"id":"sitemap","parentSelectors":["_root"],"sitemapXmlMinimumPriority":"0.0","sitemapXmlUrlRegex":"","sitemapXmlUrls":["https://www.paperstone.co.uk/sitemap.xml"],"type":"SelectorSitemapXmlLink"},{"id":"code","multiple":false,"parentSelectors":["sitemap"],"regex":"","selector":"span.js-price-match-guarantee-popup-initiator","type":"SelectorText"},{"id":"price","multiple":false,"parentSelectors":["sitemap"],"regex":"","selector":"span[itemprop='price']","type":"SelectorText"}]}

Url: https://www.paperstone.co.uk

#2

Hi, there was a typo in the start url, but after fixing that, the scraper seems to run normally. It is difficult to determine what could be the issue for a scrape done locally and not Webscraper cloud. Do you get any data returned at all?

#3

Hi - thanks for your reply, what was the typo? We do get data, sometimes around 1000 lines then other times more, but it always fails at some point. Would we get more info if we used Webscraper cloud?

#4

There was a 1 at the end of the link.

I would recommend creating a trial cloud account if you have not done so, and running the sitemap in the cloud. Webscraper support will then be able to check the feasibility of the scraping task and advise you on the associated costs.