Sitemap structure for page scrape with linear pagination

How to?
#1

Hello again!

I have learned how to scrape the information I need from my pages:
Page Number (text)
Unit title (text)
Module title (text)
Learning Objective language, multiple (text)

But I cannot figure out how to make the scraper then go to the next page and repeat the process. What is the sitemap logic I need in order to repeat this process 100, 200, 500 times?

Thanks!
Hal

#2

Thanks -- I figured it out. The trick is to treat pagination as another item that needs to be collected, and make all the page content children of pagination. That way pagination is effectively "searching the next page for the items you need from every page, including searching for the next page once you're there."

#3

hey can you post your sitemap data so we can see how it's done?

#4

Hi chris_m, sorry for the delayed response. Hopefully this sitemap helps.

10
Screenshot 2018-04-02 11.27.10.png810×280 32.9 KB

Keep in mind, my scraping needs are perfectly linear, like crawling through a book page-by-page. Each page is "page number, title, unit, module, learning objectives, and what's on the next page."

Hal

#5

Awesome. Thanks for this. I ended up doing something really similar to this.