My web crawler plugin encounters issues due to excessive crawling at once. Each crawl generates sub-pages within the webpage, leading to severe lagging and subsequent crawling failures. How should I address this problem? Is it possible to automatically close the sub-pages after each crawl or perform keyboard operations to manage them?
Hi, please share the sitemap where you are facing this problem.