Is it possible to delay or slow down the scraper while the scrapper is running. I've tried adding delay through Element Click while the scraper is running, but it doesn't work.
My motive is to avoid getting blocked from server, and I can't quit now as 30-50% scraping is complete on AJAX website.
Is it possible through Dev Tools ?
You could just specify a longer Page load delay and/or Request interval, before you click Start scraping.
Using Element Click's delay also works and I sometimes use this method to create custom delays. The trick is to find a selector which consistently appears on all pages where you want the delay.
Hi Leemeng,
Thanks for your reply.
I use the above delay mechanisms along with Element Scroll for achieving desired delay while creating sitemap.
My question was related to fire fighting situation when you have already started scraping your pages. Where I had already set the Page load delay and/or Request interval while giving 'Scrape' command.
After the scraping started, I felt that scraper was going too fast and I need to increase Page load delay and/or Request interval. Due to which the server got alarmed and blocked me '403 forbidden You don't have permission to access this resource'
Wish that there is control mechanism once you start the script...
Thanks again for your reply leemeng.
Yup, I've used pause button from source tab of Dev tools frequently for stopping scrapping while resetting my proxy ip address.
Being a manual process, it is not convenient for automated scrapping of 1000s of pages.
For now, I think I've to take the necessary delay's initially itself at time of creating sitemap.