How to save my sitemaps (I want to make a backup)

chefas · April 1, 2018, 4:45pm

Hello,

I am trying to identify the files used by Web Scraper to store my SiteMaps.
I use Web Scraper on PC (Windows 10).
Could you give me the exact path and storage folder because I don't see anything related to my sitemaps here:
C: \ Users \ John \ AppData \ Local \ Google \ Chrome \ User Data \ Default \ Extensions

Thank's a lot for your answer.

mhasan · April 1, 2018, 7:23pm

You can save sitemap from Sitemap Menu -> "Export Sitemap". Copy JSON code and save it

chefas · April 2, 2018, 9:10am

Thanks
of course I know this trick.
It can be use if you have just a few sitemaps.
But if plenty more it's not convenient and you must pay attention to make regularly this kind export.
So I would prefer to copy all the database where are saved my sitemaps.
Where is located this database/folder ? This is my initial question.

Jeff · September 20, 2019, 1:50pm

Hello,

the same question, is there a way to save the sitemaps?
The export function is very awkward. If somebody knew the path where the data is stored, it would help a lot.

andy · February 12, 2020, 7:28pm

Same question - It seems a basic functionality but I'm surprised that I'm not able to find it.
How can I manage the sitemaps panel ? delete old/wrong sitemaps, (bulk)export, (bulk)import, etc.?
Do exist a data file somewhere containing the list of locally saved sitemaps?

leemeng · February 13, 2020, 1:15am

On my computer, scraped data appears to be stored in a series of *.ldb files located in
C:\Users\<myusername>\AppData\Local\Google\Chrome\User Data\Profile 1\IndexedDB\chrome-extension_xxxrandom-chars-fkipebgh_0.indexeddb.leveldb

I think for most users, the default Chrome profile folder would be C:\Users\<yourusername>\AppData\Local\Google\Chrome\User Data\Default\

I have no idea where the WS sitemaps are stored though. I'm guessing somewhere in the default Chrome profile folder.

hstanleycrow · June 19, 2020, 4:01pm

Hi, somebody have a workaround? My main computer has a windows failure and i need to format, but i need to copy the sitemaps, i can't access to windows, i will to try with GNU/Linux.
Thank you

leemeng · June 19, 2020, 11:22pm

Haven't tried it yet, but you could try backing up, then restoring the folders used by WS on the new PC/New Windows. These links might help:

hstanleycrow · June 20, 2020, 8:13pm

Hi, thank you for your answer. I tried but the sitemaps are not there but thank you.

ericlng · August 13, 2021, 3:17pm

@admins is there a place where all the sitemaps live to backup, download or share? I have a 3rd party worker who created many sitemaps for me and you can only export one sitemap per export....How can we export all sitemaps at once and also import more than one sitemap at a time?

lockdown2020 · December 31, 2021, 2:04pm

@leemeng @admins would love a bulk download feature, I have the same problem sas above

lemongrass · March 24, 2023, 4:15pm

I found following workaround in Chrome. There are probably much better ways to do this.

Open the webscraper tab in the developer tools.

1.copy paste all sitemap names from the sitemap index page to excel
2.build the following line for each sitemap line replacing XXXXXX with the corresponding ID. Adapt if necessary the extension ID.

Insert a line with <html><body> on top of your list and <\body><\html< tags after the last line. Copy the whole thing.
Go to the elements tab in the developer tools . Replace the full html with yours pasting the html built with excel.
Wait until all frames have been loaded....may take a while
Go to the console tab of the developer tools
Copy paste the following code and execute....and voila you should see a new tab opening with all your sitemaps.

// declare variable to store all text
var text = "";

// get all iframes
var iframes = document.querySelectorAll('iframe');

// loop through each iframe
iframes.forEach(function(iframe) {

  // get the iframe document
  var iframeDoc = iframe.contentDocument || iframe.contentWindow.document;

  // get all textareas
  var textareas = iframeDoc.querySelectorAll('textarea');

  // loop through each textarea
  textareas.forEach(function(textarea) {

    // concatenate all text
    text += textarea.value + '\n';
  });
});

// open new window with all text
var win = window.open();
win.document.write('<pre>' + text + '</pre>');

matteoraggi · September 24, 2023, 6:23pm

Chromebooks / Chrome OS cant go to path like C:\Users<yourusername>\AppData\Local\Google\Chrome\User Data\Default\
Is there some easy trick for Chrome OS to backup the WebScraper sitemaps quickly and easily?

KristapsWS · September 25, 2023, 7:46am

For sitemap backup you should use Web Scraper Cloud. You can create an account for free without subscribing and your sitemaps will be available there even after your free trial ends.

Sitemaps can be uploaded and downloaded from multiple browsers. This is the most convenient backup method for keeping sitemaps up to date on multiple PCs.

Use this as a precaution because if you are reading this when your sitemaps are already gone, it is too late.

More info: Sitemap sync | Web Scraper Documentation

lemongrass · November 16, 2023, 7:26pm

You mean one by one or is there a way to import them all at once if you have more than 100?

lemongrass · November 20, 2023, 5:49pm

Unfortunately the workaround I mentioned earlier doesn't seem to work anymore.

ViestursWS · November 22, 2023, 10:20am

@lemongrass Hi, currently the sitemaps via sitemap sync can be uploaded to cloud only 1 by 1.

Please, note that user has a limit of 50 sitemap sync actions per 15 minutes.

leemeng · November 23, 2023, 3:06am

Hi @KristapsWS @ViestursWS I'd like to develop a solution for this. Could I get more details on how and where WS stores all the sitemaps? From my early observations, I'm guessing it is in some form of SQL. If so, which flavor of SQL?