Crashing while exporting huge amounts of data

How to?
#1

Hello,

So the extension worked too well and I got all the data I needed. The scraping job ran for almost 40 hours and collected lot of data, but while exporting to xlsx, the extension is crashing. I am using Edge and I am suspecting its crashing because the processing is taking too long and the host browser is viewing this is a problem and shutting it down. How can I export it without crashing? Also where is the raw data located in the disk so I can back it up in case I need to extract it manually? I am using extenstion on Edge (Version 125.0.2535.51 (Official build) (64-bit)) on Windows 11.

Url: I do not want to share the URL as it's sensitive.

Thank you,

#2

Hi,

Although there are no volume limits on scraping set from the extension side, for large scraping jobs, I would recommend exploring the feasibility of the Webscraper Cloud solution.

If the scrape has been conducted locally, there might be several factors that we can not affect, as an example: your location, browser version, network performance, OS version, and other factors.

Scraping jobs in Web Scraper Cloud are launched within a virtual environment (based on a server) and are not dependent on your machine in any aspect.