Request: Add 'Pause Scraping' option after N requests

I wish to scrape a site that uses a CDN gateway to ban IP addresses that make too many requests.

I can make up to 200 requests, and then I have to pause for 90 minutes before making another request. It will ONLY clear my odometer if there is a complete 90 minute pause. If I set the request delay to 30 seconds or 60 seconds or even 60 minutes, it will still ban me after 200 requests, because there wasn't a complete no-contact 90 minute deprivation period to reset my odometer back to zero.

1 Like