Web scraper slows to a crawl

Bug Reports
#1

Web Scraper version: 0.3.7
Chrome version: Version 65.0.3325.181 (Official Build) (64-bit)
OS: Windows 10

All goes well at first, with new pages loading every couple of seconds. After a few hours of scrapping, the page advance slows to once every minute.

I am using CouchDB installed on my local machine to store data.
The computer has an i7 processor and my internet connection is 50mb down and 35mb up

http://variety411.com/c/ad-agencies-production-companies/

Sitemap:
{"_id":"variety411usadagenciesproductioncompanies","startUrl":["http://variety411.com/c/ad-agencies-production-companies/"],"selectors":[{"id":"contact","type":"SelectorLink","selector":"ul.company-info:nth-of-type(n+2) a","parentSelectors":["pagenumber","subcategory"],"multiple":true,"delay":0},{"id":"Element","type":"SelectorElement","selector":"div.map-address","parentSelectors":["contact"],"multiple":false,"delay":0},{"id":"company","type":"SelectorText","selector":"div.map-address h1","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"pagenumber","type":"SelectorLink","selector":"a.page-numbers","parentSelectors":["pagenumber","subcategory"],"multiple":true,"delay":0},{"id":"address","type":"SelectorText","selector":"tr:contains('Address:') span:nth-of-type(1)","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"city","type":"SelectorText","selector":"span:nth-of-type(2)","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"state","type":"SelectorText","selector":"span:nth-of-type(3)","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"zip","type":"SelectorText","selector":"span:nth-of-type(4)","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"phone","type":"SelectorText","selector":"table#contact-txt.business-address span.gc-cs-link","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"fax","type":"SelectorText","selector":"tr:contains('Fax:') span.gc-cs-link","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"industry","type":"SelectorText","selector":"div.map-address div:nth-of-type(1)","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorText","selector":"tr:contains('Website:') a","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"email","type":"SelectorText","selector":"a.contact-txt2","parentSelectors":["contact"],"multiple":false,"regex":"","delay":0},{"id":"subcategory","type":"SelectorLink","selector":"li.cat-item a","parentSelectors":["_root"],"multiple":true,"delay":0}]}

Error Message:

To access error messages follow these steps:

  1. Open chrome://extensions/ or go to manage extensions
  2. Enable “developer mode” at the top right
  3. Open Web Scrapers “background page”
  4. A new popup window should appear.
  5. Go to “Console” tab. You should see Web Scraper log messages and errors there.
#2

This shouldn't be related to CouchDB because the scraper is only writing results into the database which should work really fast.

This might be caused by some assets in the site loading slowly. You could try to monitor network traffic to see what is loading slowly. You can do this by following these steps:

  1. Start scraping
  2. Focus scraper window (bring to front)
  3. press CTRL+SHIFT+I
  4. Developer toolw window will apear
  5. Go to Network tab
  6. There you can see what is causing slow page loads.

You can also try to use an adblock which will reduce the number of assets loaded.

What does that mean?
#3