Can't get both link selector and pagination working together

How to?
#1

I'm trying to scrape data from individual product pages, which are linked to across numerous pages featuring a "next" button. I can get the link selector working on the first page, but when I add the pagination it just clicks through all the pages without scraping all the product pages within. I feel like there must be some small detail I'm missing.

Url: Search Results

Sitemap:
{"_id":"NetGalley","startUrl":["https://www.netgalley.com/catalog/category/24/recentlyAddedDRCs"],"selectors":[{"id":"booklinks","linkType":"linkFromHref","multiple":true,"parentSelectors":["pagination"],"selector":".coverBox5.pt-3 a","type":"SelectorLink","version":2},{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":"a[rel='next']","type":"SelectorPagination"},{"id":"title","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":"h1[itemprop='name']","type":"SelectorText","version":2},{"id":"author","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":"[itemprop='author'] span","type":"SelectorText","version":2},{"id":"pubdate","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":".pb-1.heading-6 strong:nth-of-type(1)","type":"SelectorText","version":2},{"id":"publisher1","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":"a span[itemprop='name']","type":"SelectorText","version":2},{"id":"publisher2","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":"[itemprop='publisher'] > span[itemprop='name']","type":"SelectorText","version":2},{"id":"categories","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":".categories p","type":"SelectorText","version":2},{"id":"description","multiple":false,"multipleType":"singleColumn","parentSelectors":["booklinks"],"regex":"","selector":"div.d-md-block","type":"SelectorText","version":2}]}

#2

Hi,

The sitemap seems to be constructed correctly. Note that the scraper will start to open the individual links only after clicking through all pagination pages first. Did you wait for it to finish pagination?

#3

On this site, the Url changes consistently with each page, so you can use the Specify multiple urls with ranges method. You do not need to create a paginator.

For your example, it would be:

https://www.netgalley.com/catalog/category/24/recentlyAddedDRCs?direction=desc&page=[1-52]

To find out the last page, click on the >> button.

#4

Nope, I didn't!! I was expecting it to the individual pages first and then paginate, so that was my mistake. And there are a lot of pages, so I didn't wait for it to cycle through. Thank you!!