Hi there,
Been trying Web Scraper over the past week. Everything is working great but the delay/load time variables...
What happen is Web Crawler land on page 1 of any given start page, typically use the delay I gave him (say request interval 12000 and page load 5000) but then, navigating pages 2, 3 and 4 of the same "start page" ...will go as fast as it can. Sometimes even leaving the page before all pages have loaded.
Then, upon entering next "start page", will calm down again for page 1 only ...then go as fast as it can for pages 2 onward again.
Data is all scraped correctly but we're sure going to be shut down by site protections.
Anybody else having issues with Web Scraper seemingly ignore delay/wait timings?
My browser is Brave on Windows 11. No other extensions on that browser.
Thanks a lot,