I need a suggestion for an approach to a site design I want to scrape.
The site has categories and subcategories. Navigation page links are only shown for the first N pages. If you navigate to the Nth page, the navigation changes and changes the page link ranges until you (manually) navigate to the last set of pages.
Example:
One the first page of the subcategory, you see links:1,2,3,4,5,6,7,8,9, Next (Next is 2)
On the 9th page, you see links: 9,10,11,12,13,14,15, Next (Next is 10 because you are on page 9)
I know I can use page ranges: url/page=[1-15] but this requires manually changing the medata starturl and there are multiple subcategories which I would like to capture. Is there a technique to go beyond the first N pages of navigation links for an unknown number of pages?
Site is internal proprietary.
Thank you.
BTW, would be so helpful to have the sitemaps posted in the body of their corresponding example pages.