Hi there,
Describe the problem.
I want to scrape a website at a very low pace. I set Page load delay 30000 ms and Request interval to 30000 ms
I built the sitemap based on one URL - when I scrape, I see WS open a window with the right product page and I've got all the data I want. See below sitemap and screenshot.
Sitemap:
{"_id":"hoffmann","startUrl":["https://www.hoffmann-group.com/FR/fr/hof/p/554012-60"],"selectors":[{"id":"brand","type":"SelectorText","parentSelectors":["_root"],"selector":"div.manufacturer","multiple":false,"regex":"","delay":0},{"id":"prix","type":"SelectorText","parentSelectors":["_root"],"selector":"span.main-price","multiple":false,"regex":"[0-9]+\,[0-9]+","delay":0},{"id":"prix_pour","type":"SelectorText","parentSelectors":["_root"],"selector":"div.bold","multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","parentSelectors":["_root"],"selector":"h1","multiple":false,"regex":"","delay":0}]}
When I add URLs in the sitemap - up to 300 - I see WS open a window but a login is required.
How can I avoid reaching this page? I'm stuck on it for several days
Thank you fr your support
David