I am scraping multiple pages in a site where there is a delay to complete loading the fields I am scraping.
The sitemap involves iterating through a number of select fields to change the data - each change of select option loads a new page.
On running the scrape, I specify a longer delay for both the Request Interval and the Page Load Delay - setting both to 10000 to allow the data to complete loading so it can be scraped.
This works fine on loading the first page, but subsequent pages appear to only delay for the default 2000 (or actually less) which isn't long enough for the data to load, leading to many null values being recorded.