Trouble with Pagination and Sitemap Setup in Web Scraper

How to?
#1

I can’t get it to automatically move to the next page and repeat the product data collection steps — it basically only scrapes the first page and then stops. Could someone help me design a sample sitemap?

Also, the SelectorPagination "next_page" always has its parent selector set to _root. I’m not sure if this is a feature or a bug, but it doesn’t seem very logical.

Thanks a lot!

Url: Amazon.co.uk Best Sellers: The most popular items in Garden

Sitemap:
{"_id":"amz_best_seller","startUrl":["Amazon.co.uk Best Sellers: The most popular items in Garden a","type":"SelectorPagination"},{"id":"title","multiple":false,"multipleType":"singleColumn","parentSelectors":["wrapper"],"regex":"","selector":".a-link-normal span div","type":"SelectorText"},{"id":"price","multiple":false,"multipleType":"singleColumn","parentSelectors":["wrapper"],"regex":"","selector":".a-size-base span","type":"SelectorText"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["wrapper"],"selector":".p13n-sc-uncoverable-faceout > div > div > a","type":"SelectorLink"},{"id":"delivery","multiple":false,"multipleType":"singleColumn","parentSelectors":["link"],"regex":"","selector":"#deliveryBlockMessage #mir-layout-DELIVERY_BLOCK-slot-PRIMARY_DELIVERY_MESSAGE_LARGE span[data-csa-c-type]","type":"SelectorText"}]}

#2

Hi,

I was not able to open your sitemap, since the code is broken. Nonetheless, see below a reference sitemap for amazon:

{"_id":"amazon-category-pagination","startUrl":["https://www.amazon.com/s?k=coffee&rh=n%3A14092821","https://www.amazon.com/s?keywords=Dog+Food&i=pets&rh=n%3A2619533011%2Cp_123%3A329375&dc&c=ts&qid=1734087728&rnid=85457740011&ts_id=2975359011&ref=sr_nr_p_123_12&ds=v1%3AzyLFGmYnbB93bSqo33Yd51%2BPoGAw%2FikQ6G8q7eOcXGM"],"selectors":[{"id":"product-pagination","paginationType":"auto","parentSelectors":["_root","product-pagination"],"selector":".a-last a[href*='page='], a.pagnNext, .a-pagination a:contains('Next'), a.s-pagination-next[href*='page=']","type":"SelectorPagination"},{"id":"category","multiple":false,"multipleType":"singleColumn","parentSelectors":["_root"],"regex":"","selector":"div#departments li[class*='s-navigation-indent'] span.a-text-bold, .searchSelect [current='true']","type":"SelectorText"},{"elementLimit":0,"id":"product-elem","multiple":true,"parentSelectors":["product-pagination"],"scroll":false,"selector":".s-result-list div[data-component-type=\"s-search-result\"]:not(.AdHolder)","type":"SelectorElement"},{"id":"product-link","linkType":"linkFromHref","multiple":false,"parentSelectors":["product-elem"],"selector":"a:has(> h2)","type":"SelectorLink"},{"elementLimit":0,"id":"product-page","multiple":true,"parentSelectors":["product-link"],"scroll":false,"selector":"html:has(#productTitle):not(:empty)","type":"SelectorElement"},{"id":"product-title","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-elem"],"regex":"","selector":"h2","type":"SelectorText"},{"extractAttribute":"data-asin","id":"product-asin","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-elem"],"selector":"_parent_","type":"SelectorElementAttribute"},{"id":"product-brand","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-page"],"regex":"","selector":"#bylineInfo:contains('Brand: '), tr:contains('Brand') td:nth-of-type(2), th:contains('Brand') + td, th:contains('Manufacturer'):not(:contains('Discontinued')) + td","type":"SelectorText"},{"id":"product-price","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-elem"],"regex":"","selector":".a-price:not([data-a-strike]) .a-offscreen","type":"SelectorText"},{"id":"product-price-old","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-elem"],"regex":"","selector":".a-price[data-a-strike] .a-offscreen","type":"SelectorText"},{"id":"product-rating","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-page"],"regex":"","selector":"#averageCustomerReviews_feature_div span.a-color-base","type":"SelectorText"},{"extractAttribute":"aria-label","id":"product-review-count","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-elem"],"selector":"a[aria-label][href*='#customerReviews'], a[aria-label*=\"rating\"].a-link-normal","type":"SelectorElementAttribute"},{"id":"product-description","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-page"],"regex":"","selector":"#productDescription, #mas-product-description div","type":"SelectorText"},{"id":"product-features","multipleType":"singleColumnWithSeparator","parentSelectors":["product-page"],"regex":"","selector":"#feature-bullets li:not(#replacementPartsFitmentBullet)","type":"SelectorText"},{"id":"product-image","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-page"],"selector":"#imgTagWrapperId img","type":"SelectorImage"},{"extractAttribute":"data-old-hires","id":"product-image-hires-src","multiple":false,"multipleType":"singleColumn","parentSelectors":["product-page"],"selector":"#imgTagWrapperId img","type":"SelectorElementAttribute"}]}
1 Like