After dec 21 update all sitemaps disappeared

karolis666 · December 22, 2017, 10:15am

Ok so I just checked chrome web store and apparently scraper was updated on December 21, 2017. So the reason is clear. But now what? Will I be able to restore all my sitemaps?

martins · December 22, 2017, 10:34am

Before we released the upgrade we tested whether the sitemaps would migrate correctly to the new extension. There weren't any problems.

There isn't any auto delete or database cleanup functionality also.

Please do these steps so we can understand the problem:

open chrome://extensions/ or go to manage extensions
enable "developer mode" at the top right
open Web Scrapers "background page"
a new popup window should appear.
go to "Console" tab to check whether there are any errors there. If there are errors paste them here

If there aren't any errors you can check web scrapers storage to see whether the sitemaps are in chrome storage.

go to "Application" tab
Under Storage > IndexedDB there should be a database _pouch_scraper_sitemaps
under _pouch_scraper_sitemaps expand "document-store" there you should see your sitemaps
Do you see your sitemaps there?

karolis666 · December 22, 2017, 10:37am

Pasting errors.

background_script.js:24 initial configuration i
background_script.js:24 initializing Background Script message listener
background_script.js:24 Uncaught DOMException: Failed to execute 'put' on 'IDBObjectStore': Evaluating the object store's key path did not yield a value.
at IDBRequest.n.openCursor.onsuccess (chrome-extension://jnhgnonknehpejjnehehllkliplmbmhn/background_script.js:24:150938)
background_script.js:24 Failed to open indexedDB, are you in private browsing mode?
_ @ background_script.js:24
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 content script error Object {"status":500,"name":"indexed_db_went_bad","message":"unknown","reason":"Failed to open indexedDB, are you in private browsing mode?"}
background_script.js:24 extension usage ended
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 content script error Object {"status":500,"name":"indexed_db_went_bad","message":"unknown","reason":"Failed to open indexedDB, are you in private browsing mode?"}
background_script.js:24 Uncaught (in promise) n
background_script.js:24 extension usage ended
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 content script error Object {"status":500,"name":"indexed_db_went_bad","message":"unknown","reason":"Failed to open indexedDB, are you in private browsing mode?"}
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 content script error Object {"status":500,"name":"indexed_db_went_bad","message":"unknown","reason":"Failed to open indexedDB, are you in private browsing mode?"}
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n
background_script.js:24 Uncaught (in promise) n

karolis666 · December 22, 2017, 10:45am

I can see sitemap list when I click on 'document-store' and also each individual sitemap under '_pouch_scraper_sitemaps'. And to logs repeated question, no I'm not in private browsing mode.

martins · December 22, 2017, 3:15pm

We tried to replicate the issue without success. Currently it seems that this update has affected only your installation.

We released a 0.3.2 version which introduces a more detailed error messages for database calls and also wouldn't fail if the initial connection somehow fails. We hope that this update fixes the issue for you. That said we didn't make it a full release so you might not get the update immediately.

At the moment you can export the sitemaps from database and import them into another web scraper installation. Here are the steps:

Open chrome://extensions/ or go to manage extensions
Enable “developer mode” at the top right
Open Web Scrapers “background page”
Go to "Console" tab
Copy all of the code from here and paste it into the console
Hit "Enter" to execute the code
Additionally run this code:

var db = new PouchDB('scraper-sitemaps');
db.allDocs({include_docs: true}).then(function(response){

	for(var i in response.rows) {
		var sitemap = response.rows[i].doc;
		delete sitemap._rev;
		console.log(JSON.stringify(sitemap));
	}
});

You should see the sitemaps printed out in the console
Create another user profile in chrome and import sitemaps there.

If you still cannot access sitemaps after getting 0.3.2 version please post here.

karolis666 · December 22, 2017, 3:17pm

Thank you so much for all your trouble. So much work would go to waist if those sitemaps would be unrecoverable.

karolis666 · December 22, 2017, 4:55pm

Waited till I got 0.3.2 version. Still cant access sitemaps.

karolis666 · December 23, 2017, 9:26am

Hello

I tried to complete steps to retrieve sitemaps and after I did 7th step I got error messages in console:

karolis666 · December 27, 2017, 9:47am

Will sitemaps be deleted if I'll try to remove addon and add again?

karolis666 · December 27, 2017, 8:57pm

If possible could someone share any suggestions? At the moment I cant even export sitemaps with running codes as I get error messages

martins · December 28, 2017, 10:08am

Try running this script in the console. This script uses native database client instead of pouchdb wrapper.

var version = 5;
var openRequest = window.indexedDB.open("_pouch_scraper-sitemaps", version);
openRequest.onerror = function (event) {
	console.error("Database error: " + JSON.stringify(event));
};
openRequest.onupgradeneeded = function (event) {
	console.error("Database upgrade needed: " + JSON.stringify(event));
};
openRequest.onsuccess = function (event) {
	console.log("connected to database");
	let indexedDb = event.target.result;
	let transaction = indexedDb.transaction(["document-store", "by-sequence"], "readonly");
	let indexStore = transaction.objectStore("document-store");

	console.log("fetching sitemap index");
	let sitemapInfosRequest = indexStore.getAll();

	sitemapInfosRequest.onsuccess = function(event) {

		console.log("fetched sitemap index");
		let sitemapInfos = event.target.result;
		console.log(sitemapInfos);

		let sitemapRevs = [];
		for(let i in sitemapInfos) {
			let sitemapInfo = sitemapInfos[i];
			if(sitemapInfo.deletedOrLocal === "0") {
				sitemapRevs.push(sitemapInfo.winningRev)
			}
		}

		console.log("fetching sitemaps");
		let sitemapStore = transaction.objectStore("by-sequence");
		let sitemapsRequest = sitemapStore.getAll();
		sitemapsRequest.onsuccess = function(event) {

			console.log("fetchied sitemaps");
			let sitemaps = event.target.result;
			console.log(sitemaps);

			for(let i in sitemaps) {
				let sitemap = sitemaps[i];
				let sitemapRev = sitemap._rev;

				if(sitemapRevs.indexOf(sitemapRev) !== -1) {
					delete sitemap._rev;
					delete sitemap._doc_id_rev;
					let sitemapJSON = JSON.stringify(sitemap);
					console.log(sitemapJSON);
				}
			}

		};
		sitemapsRequest.onerror = function (event) {
			console.error("sitemaps error: " + JSON.stringify(event));
		};
	};
	sitemapInfosRequest.onerror = function (event) {
		console.error("sitemap infos error: " + JSON.stringify(event));
	};
};

karolis666 · December 28, 2017, 12:00pm

I appreciate for your spent time on this. Really. After executing latest code it only prints 'connected to database' and after waiting 10+mins nothing happens

martins · December 28, 2017, 12:14pm

I added some additional debug messages to the script. Try running it again.

karolis666 · December 28, 2017, 12:38pm

Ok. I got something now. Not sure how to export/copy sitemaps.

martins · December 28, 2017, 12:45pm

Those are only debug messages. It seems that the matcher script didn't work correctly on your side.

Please expand one record from sitemap index and one from sitemaps. Post a screenshot of those objects here. I'll update the script once again.

karolis666 · December 28, 2017, 12:50pm

From index:

karolis666 · December 28, 2017, 12:51pm

From sitemaps:

karolis666 · December 28, 2017, 12:51pm

Let me know if you want to expand somewhere specifically.

martins · December 28, 2017, 2:09pm

Try this one:

var version = 5;
var openRequest = window.indexedDB.open("_pouch_scraper-sitemaps", version);
openRequest.onerror = function (event) {
	console.error("Database error: " + JSON.stringify(event));
};
openRequest.onupgradeneeded = function (event) {
	console.error("Database upgrade needed: " + JSON.stringify(event));
};
openRequest.onsuccess = function (event) {
	console.log("connected to database");
	let indexedDb = event.target.result;
	let transaction = indexedDb.transaction(["document-store", "by-sequence"], "readonly");
	let indexStore = transaction.objectStore("document-store");

	console.log("fetching sitemap index");
	let sitemapInfosRequest = indexStore.getAll();

	console.log("fetching sitemaps");
	let sitemapStore = transaction.objectStore("by-sequence");
	let sitemapsRequest = sitemapStore.getAll();
	sitemapsRequest.onsuccess = function(event) {

		console.log("fetchied sitemaps");
		let sitemaps = event.target.result;
		console.log(sitemaps);

		let sitemapRevs = {};

		for(let i in sitemaps) {
			let sitemap = sitemaps[i];
			let sitemapId = sitemap._id;
			let sitemapRev = sitemap._rev;
			if(sitemapRev === undefined) {
				continue;
			}

			let revId  = parseInt(sitemapRev.match(/^(\d+)\-/)[1]);

			if(sitemapRevs[sitemapId] === undefined) {
				sitemapRevs[sitemapId] = revId;
			}
			else if(revId > sitemapRevs[sitemapId]) {
				sitemapRevs[sitemapId] = revId;
			}
		}

		for(let i in sitemaps) {
			let sitemap = sitemaps[i];
			let sitemapRev = sitemap._rev;
			if(sitemapRev === undefined || sitemap._deleted === true) {
				continue;
			}
			let sitemapId = sitemap._id;
			let revId  = parseInt(sitemapRev.match(/^(\d+)\-/)[1]);

			if(sitemapRevs[sitemapId] === revId) {
				delete sitemap._rev;
				delete sitemap._doc_id_rev;
				let sitemapJSON = JSON.stringify(sitemap);
				console.log(sitemapJSON);
			}
		}

	};
	sitemapsRequest.onerror = function (event) {
		console.error("sitemaps error: " + JSON.stringify(event));
	};
};

karolis666 · December 28, 2017, 2:43pm

Yay! I can see sitemaps printed! Will backup them as text files on pc and will try to import on fresh web scraper since doing it no yields no results (after I click import nothing happens).
Thank you so much