I tried to select "link" to get the links of all sub-pages, but the webscraper needs to open each sub-page to get the link, which leads to very low efficiency, because my main webpage has 400 pages, if each main webpage is The links to the sub-pages all need to be opened one by one to get it, it is too slow, I need a faster method, please help.
@shawchen Hi there. Can you share the targeted website/your sitemap?
This is the link I attached. It is a search result of a shopping page. Please check it out.
The result is four hundred pages in total. I need to get the link addresses of all the products on each page, so that I can extract information from the addresses. I know that I can get it by selecting the crawl link, but this leads to the need for every link to crawl Open a product. This is too slow, I just need to get the link, I hope there is a faster way.
If you just use type: Link without any child selectors (no other data scrapers within it), it will just get the title and URL, and it will not click thru to the other pages. E.g.