Pagination not working with Link?!

How to?
#1

I am using a combination of pagination and SelectorLink. On it's own, SelectorLink works as needed, but as soon as I add the pagination function, it stops working.

Is it also possible to stop the pagination after 1000 times?

Here is my sitemap:
{"_id":"CityWireFixed","startUrl":["https://citywire.com/new-model-adviser/news/latest-news/list"],"selectors":[{"id":"click","linkType":"linkFromHref","multiple":true,"parentSelectors":["pagination"],"selector":".cw-mx-4 a.hover\:cw-text-primary-600","type":"SelectorLink"},{"id":"Category","multiple":false,"parentSelectors":["click"],"regex":"","selector":"a.cw-pr-3","type":"SelectorText"},{"id":"Date","multiple":false,"parentSelectors":["click"],"regex":"","selector":".cw-flex-wrap span.first\:cw-pl-0","type":"SelectorText"},{"id":"Title","multiple":false,"parentSelectors":["click"],"regex":"","selector":"h1.cw-text-primary-900","type":"SelectorText"},{"id":"Summary","multiple":false,"parentSelectors":["click"],"regex":"","selector":"h2.xl\:cw-text-xl","type":"SelectorText"},{"id":"Author","multiple":false,"parentSelectors":["click"],"regex":"","selector":".cw-font-bold a","type":"SelectorText"},{"id":"Blob","multiple":false,"parentSelectors":["click"],"regex":"","selector":"div.locked-content","type":"SelectorText"},{"id":"pagination","paginationType":"linkFromHref","parentSelectors":["_root","pagination"],"selector":"a.cw-mx-auto","type":"SelectorPagination"}]}

#2

Hi,

Here is a sitemap reference with a limited pagination. Check the pagination selector and let me know if it is unclear.

Note that the sitemap will first go over all pages and then open the links.

{"_id":"CityWireFixed","startUrl":["https://citywire.com/new-model-adviser/news/latest-news/list"],"selectors":[{"id":"click","linkType":"linkFromHref","multiple":true,"parentSelectors":["pagination"],"selector":"div#contentListContainer > div a:has(h2, h3)","type":"SelectorLink"},{"id":"Category","multiple":false,"parentSelectors":["click"],"regex":"","selector":"a.cw-pr-3","type":"SelectorText"},{"id":"Date","multiple":false,"parentSelectors":["click"],"regex":"","selector":"#ArticleContainer > div:nth-of-type(1) span","type":"SelectorText"},{"id":"Title","multiple":false,"parentSelectors":["click"],"regex":"","selector":"h1.cw-text-primary-900","type":"SelectorText"},{"id":"Summary","multiple":false,"parentSelectors":["click"],"regex":"","selector":"#ArticleContainer > div:nth-of-type(1) h2","type":"SelectorText"},{"id":"Author","multiple":false,"parentSelectors":["click"],"regex":"","selector":".cw-font-bold a","type":"SelectorText"},{"id":"Blob","multiple":false,"parentSelectors":["click"],"regex":"","selector":"div.locked-content","type":"SelectorText"},{"id":"pagination","paginationType":"linkFromHref","parentSelectors":["_root","pagination"],"selector":"body:not(:has(p:contains('Page 12'))) a:contains('Next')","type":"SelectorPagination"}]}
1 Like
#3

Thank you - that worked. Much appreciated. Why does the click element only happen after the pagination has completed?

#4

The scraper collects all links first and opens only unique links to avoid duplication.

1 Like