Anyone please help, Save data into CouchDB

I have success to configure and get data, able to eksport to CSV,
because my case need automatic and see documentasion able to connect with CouchDB.

wonder to get success. How to connect with CouchDB
anyone please help...

Url: http://bukalapak.com

Sitemap:
{"_id":"penjualan_bukalapak","startUrl":["https://www.bukalapak.com/payment/transactions?active_tab=sell&payment_transaction_searchable_search[filter_by]=sell"],"selectors":[{"id":"item","type":"SelectorElement","parentSelectors":["_root"],"selector":".c-table--transaction tr","multiple":true,"delay":0},{"id":"link","type":"SelectorLink","parentSelectors":["item"],"selector":".u-fg--black","multiple":false,"delay":0},{"id":"status","type":"SelectorText","parentSelectors":["item"],"selector":".u-txt--base","multiple":false,"regex":"","delay":0},{"id":"dikirim","type":"SelectorGroup","parentSelectors":["link"],"selector":".memo-right .address dl","delay":0,"extractAttribute":""},{"id":"pembeli","type":"SelectorLink","parentSelectors":["link"],"selector":".memo-left a","multiple":false,"delay":0},{"id":"biaya_kirim","type":"SelectorGroup","parentSelectors":["link"],"selector":".memo-left dl","delay":0,"extractAttribute":""},{"id":"kurir","type":"SelectorText","parentSelectors":["link"],"selector":".memo-left .notice","multiple":false,"regex":"","delay":0}]}

Step1: Download and Install CouchDB see http://docs.couchdb.org/en/2.2.0/install/index.html

Step2: Create a new DB (that will store your Sitemaps) the scraped data will be stored in a DB created by WebScraper. Remember the name of your new sitemap database.

Step3: Configure the WebScraper Extension to store your data into your CouchDB.
Therefore you've to go to you Chrome Extensions -> Details -> There is a Option: something like
"Extensionoptions".

Here you can edit three parameters:

  1. Storage type -> Couch DB
  2. Sitemap db -> http://localhost:5984/ws_sitemap (assume that you install CouchDB with the standard config otherwise you have to edit the port 5984 to the one you configured), behind the slash you have to put in your db name you configured in Step2.
  3. data db -> http://:@localhost:5984/
    Here you have to fill in the URL to your CouchDB. As long as I not give the username and password with the URL, the Scraper didn't start to scrape. Obviously because the scraper creates a new DB for each scraping-Sitemap it needs the right to create a new DB. Perhaps it is also possible to allow everyone to create new DBs.

Hopefully this will also work for you!

Best,
Philipp

  • [Change] CouchDB has been deprecated. Users that were using it will be able to continue to use it but new users won't be able to change data storage. We plan to replace the current data storage engine (PouchDB) with simpler one to reduce problems with sitemap and data storage.