Does not use delay settings

Bug Reports
#1

Hi there,

Been trying Web Scraper over the past week. Everything is working great but the delay/load time variables...

What happen is Web Crawler land on page 1 of any given start page, typically use the delay I gave him (say request interval 12000 and page load 5000) but then, navigating pages 2, 3 and 4 of the same "start page" ...will go as fast as it can. Sometimes even leaving the page before all pages have loaded.

Then, upon entering next "start page", will calm down again for page 1 only ...then go as fast as it can for pages 2 onward again.

Data is all scraped correctly but we're sure going to be shut down by site protections.

Anybody else having issues with Web Scraper seemingly ignore delay/wait timings?

My browser is Brave on Windows 11. No other extensions on that browser.

Thanks a lot,

#2

you can additionally use ElementClick selector with pause let say 5000msc.... We add extra time between pages scraping.

#3

Hi, could you please elaborate your suggestion a little.. I am also facing the same issue.. the first page load follows the Request Interval and the Page Load Delay set by me (e.g. 7000ms).. but as soon as the next page loads it starts off at 2000ms max.. in doing so much of data is not being scraped..

#4

Our experience here is exactly as you describe except, in our case, products are getting scraped. That's the main thing that stops us from moving all our operations to Webscraper

#5

Hi @don2010 and thanks for the suggestion. Been trying it this afternoon to no help... When doing this, are you still using pagination?

#6

Hard to diagnose without URL or sitemap.
I'm guessing you're using the new paginator, which does seem to have a short delay. You can fix it by either:

  1. Inserting a dummy element click on say head title and specifying a delay (you're just using the delay here; the click is not important)
  2. Use the older pagination methods ( a lot of examples in forum and documentation) where you can control the click delay.