I'm trying to scrape 166 pages of links, but the pages links at the bottom of the page do not feature a NEXT PAGE link icon.
It would be nice if there was a simple feature that would allow us to say scrape all pages until the last one.
For instance, in the case of the site I am trying to scrape, I noticed that the pattern for the pages links is:
http://website.net/tags/searchname/page/2/
http://website.net/tags/searchname/page/3/
http://website.net/tags/searchname/page/4/
... all the way to:
http://website.net/tags/searchname/page/166/
What if WEBSCRAPER could identify the pattern by itself?
Maybe it would be a feature like: "IDENTIFY PAGES PATTERN", and user would click on all individual page links at the bottom of the page, and webscrapper would identify the pattern...
Martin