The scraper works perfectly but when I am scraping a range, it doesnt go from example.com/1 to example.com/2 and so on. But it skips some urls. and it just jumps all over the place. The URL goes to about 90000 if that is relevant.
Can you share the sitemap that you are using?
I can't share the URL. But I have figured it out. I have to extend the interval from 2000ms to 4000ms so that it could load properly. And the URL starts from the top to the bottom. if the URL that I am going to scrape is 1-100. It starts from 100 to 1.
Also can I continue a scrape if it was stopped for some reason? I don't want to run it again and start all over.
No, unfortunately, there is no such option. If you stop and re-start the scrape, it will start from the beginning.
Ah, i will just adjust the sitemap then. it's a hassle but still doable