How much to wait for link to download csv?

malodar · January 10, 2018, 3:58pm

Hello! Yesterday I started scraping yelp with your tool, today the scraping has finished successfully (as per the pop-up window message). After I clicked Export to csv I'm waiting for more than 60 minutes for the link to download the file, but it doesn't appear. Same thing when I click Browse, I see only 'loading...'. Is it normal (maybe because of big size of data to save?)?. How can I get my scraping results? Storage type is 'local storage'.

Sitemap:
{{"_id":"yelpscrap","startUrl":["https://www.yelp.co.uk/locations"],"selectors":[{"id":"locs","type":"SelectorLink","selector":"li.state:nth-of-type(1) a","parentSelectors":["_root"],"multiple":true,"delay":0},{"id":"categor","type":"SelectorLink","selector":"div.y-container.u-bg-color-alt div.arrange_unit a","parentSelectors":["locs"],"multiple":true,"delay":0},{"id":"elements","type":"SelectorElement","selector":"li.regular-search-result","parentSelectors":["pag"],"multiple":true,"delay":0},{"id":"name","type":"SelectorText","selector":"a.biz-name span","parentSelectors":["elements"],"multiple":false,"regex":"","delay":0},{"id":"adr","type":"SelectorText","selector":"address","parentSelectors":["elements"],"multiple":false,"regex":"","delay":0},{"id":"pag","type":"SelectorLink","selector":"a.available-number","parentSelectors":["categor","pag"],"multiple":true,"delay":0}]}}

martins · January 11, 2018, 2:31pm

What OS, chrome, web scraper versions are you using?

It shouldn't take that long. How much data do you think you scraped? Is the data preview working?

dotmlj · January 22, 2018, 8:24am

Hi.

I have the same problem. Don't know exactly how much data, but I've scraped about 7.000 pages x around 10.000 byte = 70 MB.

Running:
macOS High Sierra (on a brand new MacBook Pro)
Chrome 63.0.3239.132
Web Scraper 0.3.5

Thanks
Morten

malodar · January 22, 2018, 8:47am

Hi! I decided to remake scraping and it was successful.. The csv file was about 20Mb weight. I ran scraping on Ubuntu, Chrome 63.0.3239.132, Web Scraper 0.3.5

martins · January 22, 2018, 12:02pm

@dotmlj Could you check whether "browse data" feature is working?

dotmlj · January 22, 2018, 12:43pm

@martins thank you for you fast reply!

It was not, it just got stuck at "loading".

I have moved to CouchDB for storage, so I have direct access to the data. Would expect that to solve the problem. Will run the scrape again tonight.

martins · January 22, 2018, 12:51pm

Can you post errors from background page after you have tried to export the data? We think that this might be caused by message passing limits in chrome.

To access error messages follow these steps:

Open chrome://extensions/ or go to manage extensions
Enable “developer mode” at the top right
Open Web Scrapers “background page”
A new popup window should appear.
Go to “Console” tab. You should see Web Scraper log messages and errors there.

dotmlj · January 22, 2018, 1:01pm

Having switched to CouchDB and back seems to have removed the results, so I don't see the error anymore.

I just signed up for yor cloud scraper and will give the scrape a try there

martins · January 22, 2018, 1:11pm

You can switch back to local storage. The data should be there.

martins · January 23, 2018, 10:41am

It seems that the problem was that chrome messaging system limited the amount of data that can be sent from database to devtools page within the extension. We added a pagination system so that the devtools panel can fetch records in batches of 100. This should fix the problem. We will release the fix in version 0.3.7. It should be released next week.

andres · April 14, 2019, 2:09pm

Hi,
I'm also having the same problem when trying to download csv files.
After more than an hour to get the download link, tried to see data using browse and the plug crashed
I'm running Chromium Version 73.0.3683.103 under Ubuntu 18.04.2 LTS and Webscraper 0.3.8

{"_id":"mlc_01","startUrl":["https://listado.mercadolibre.com.co/inmuebles/_PriceRange_0-1","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2-50001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_50002-100001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_100002-150001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_150002-200001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_200002-250001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_250002-300001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_300002-350001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_350002-400001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_400002-450001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_450002-500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_500002-600001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_600002-700001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_700002-800001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_800002-900001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_900002-1000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1000002-1100001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1100002-1200001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1200002-1300001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1300002-1400001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1400002-1500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1500002-1600001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1600002-1700001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1700002-1800001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1800002-1900001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_1900002-2000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2000002-2100001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2100002-2200001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2200002-2300001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2300002-2400001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2400002-2500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_2500002-3000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_3000002-3500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_3500002-4000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_4000002-4500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_4500002-5000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_5000002-5500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_5500002-6000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_6000002-6500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_6500002-7000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_7000002-7500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_7500002-8000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_8000002-8500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_8500002-9000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_9000002-9500001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_9500002-10000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_10000002-11000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_11000002-12000001","https://listado.mercadolibre.com.co/inmuebles/_PriceRange_12000002-13000001"],"selectors":[{"id":"pag","type":"SelectorLink","parentSelectors":["_root","pag"],"selector":"li.andes-pagination__button:nth-of-type(n+3) a.andes-pagination__link","multiple":true,"delay":0},{"id":"link","type":"SelectorLink","parentSelectors":["_root","pag"],"selector":"a.item__info-link","multiple":true,"delay":0}]}