API : use a fix job ID Or Sitemap ID to automate the download files

Feature Requests
#1

Actually, wa can't automate download datas with make, power automate or another automate solution.

Because the get url for download json or csv file is with the job ID, who change each time.

The logical flow for webscrapping i think is automating the download files for refreshing our datas.

So we can sheduling the scrap, but no automatically download the datas?

Or may be i must scrap webscraper.io to have the good job ID? :slight_smile:

thanks per advance.

#2

Hi, could you please describe what would be the ideal solution for your use-case?

An API call for the last scraping job for a particular sitemap?

Thanks

#3

Hi,.
Many thanks for your answer.
Yes, i think the Last job ID for a Sitemap ID will be great...
Regards.
Sylvain

#4

Hi,

We will consider implementing this feature in future releases, but, generally, it is recommended to utilize the API feature together with webhooks to achieve full automatization. A webhook will notify you and provide the job ID once it is finished.

#5

Many thanks for you answer.

Yes, and before the future release, the solution could be automate calling the list of jobs api, extract the latest one with timestamp value, and use it in automation.

Regards,

Sylvain