Hi there,
This is probably a newbie question but I'm having a problem with pagination.
I'm trying to scrape contents of 1000 links spread over 6 pages @ 200 links per page.
Pagination was multi by button 2 ~ 6.
The result was 200 links scraped (1 page only)
Pagination wouldn't work but when I looked at the results, I noticed result order was reversed.
I reset pagination to multi "next" element.
I ran another scrape and watched the run.
This time I observed that the last link on the page was scraped first.
Is this why pagination fails?
ie does it scrape from the bottom of the page to the top then find no further pages?
Any help would be greatly appreciated.
Hi there!
Unfortunately you didn't bring your sitemap, please follow this tutorial for proper pagination:
As for reversed order, WebScraper works bottom-up. In previous versions, there wasn't web-scraper-order stamp and all results were mixed, so those who wanted to have results ordered, used CouchDB (I still use it). I have all my sitemaps written reversed to have ending result top-down.