Run script - Get data - Convert data - Upload data

Feature Requests
#1

A lots of my clients ask me when I create script for them "Can we run script and get data in our database?", they want to run script, data need to convert in right format (CSV...) and upload in their database (Site)... Is that possible?

#2

@Gagi Hi, are you familiar with the API feature the Web Scraper Cloud offers?

https://webscraper.io/documentation/web-scraper-cloud/api
https://webscraper.io/documentation/web-scraper-cloud/api/webhooks

#3

So with Cloud all be good? Are you test that before? Can you post some videos?

#4

@Gagi Hi.

If you are looking to automate the data delivery, this should be structured as follows:

  1. A notification endpoint is set up (in this case that would be the server you would like to receive the notification)
  2. Once a scraping job is finished, a notification is posted to your endpoint that a particular job has finished running. Example:

"scrapingjob_id": 1234
"status": "finished"
"sitemap_id": 12
"sitemap_name": "my-sitemap"
"custom_id": "custom-scraping-job-12"

  1. Once your server receives this, an API should be configured correctly on your end, that will execute a 'Download a scraping job' call (in a JSON or CSV format). This will retrieve the finished data and you will be able to handle this on your end accordingly.

Hope this helps.