I don't understand this arbitrary 2 second requirement to wait for the page to scrape. I get that there should be the option for various reasons, but I do not get why we don't at least have the option to use document.readyState === 'complete'.
Hi!
Ready state doesn't mean that the page has loaded. Sites build on JS frameworks can start loading data via network and rendering it after ready state. Web Scraper is combining multiple signals to detect when a page has loaded - the delay, document ready, no network activity, etc.
There are also sites that do JS/HTML redirects immediately after a page has loaded. For these situations a slight delay is actually the best way to not to miss the actual page load.