Scrape next pages for reviews? How to work with the next botton?

How to?
#1

Hi. I'm trying to scrape a page and the reviews inside. The pagination is 'next' until there are no more unique reviews.

I've tried using pagination or element click, but either it only scrapes the first page of reviews (so I got only 8 reviews) or it just keeps going on to the next pages but doesn't scrape any of the reviews.

What am I doing wrong??

Url: Gildan® Short Sleeve Adult T-Shirt | Michaels

Sitemap:
{"_id":"michaels-single-MASS","startUrl":["https://www.michaels.com/gildan-short-sleeve-adult-tshirt/M10093625.html"],"selectors":[{"delay":0,"id":"product_wrapper","multiple":false,"parentSelectors":["_root"],"selector":"div.pt_product-details","type":"SelectorElement"},{"delay":0,"id":"review_wrapper","multiple":true,"parentSelectors":["product_wrapper","review-pagination"],"selector":"li div.bv-content-core","type":"SelectorElement"},{"delay":0,"id":"review_title","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"h3","type":"SelectorText"},{"delay":0,"id":"review_date","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"span.bv-content-datetime-stamp","type":"SelectorText"},{"delay":0,"id":"review_rating","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"span[itemprop='reviewRating']","type":"SelectorText"},{"delay":0,"id":"review_body","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"p","type":"SelectorText"},{"delay":0,"id":"product_name","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":".product-header-left h1","type":"SelectorText"},{"delay":0,"id":"product_id","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":"span[itemprop='productID']","type":"SelectorText"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":".is-desktop div.product-sales-price","type":"SelectorText"},{"delay":0,"id":"average_rating","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":"span.bv-secondary-rating-summary-rating","type":"SelectorText"},{"id":"review-pagination","paginationType":"clickMore","parentSelectors":["product_wrapper","review-pagination"],"selector":"span.bv-content-btn-pages-next","type":"SelectorPagination"}]}

Appreciate any help!!

#2

I have tried the following, but it can only scrape 16 reviews

{"_id":"michaels-single-MASS","startUrl":["https://www.michaels.com/gildan-short-sleeve-adult-tshirt/M10093625.html"],"selectors":[{"delay":0,"id":"product_wrapper","multiple":false,"parentSelectors":["_root"],"selector":"div.pt_product-details","type":"SelectorElement"},{"delay":0,"id":"review_wrapper","multiple":true,"parentSelectors":["product_wrapper","review-pagination"],"selector":"li div.bv-content-core","type":"SelectorElement"},{"delay":0,"id":"review_title","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"h3","type":"SelectorText"},{"delay":0,"id":"review_date","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"span.bv-content-datetime-stamp","type":"SelectorText"},{"delay":0,"id":"review_rating","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"span[itemprop='reviewRating']","type":"SelectorText"},{"delay":0,"id":"review_body","multiple":false,"parentSelectors":["review_wrapper"],"regex":"","selector":"p","type":"SelectorText"},{"delay":0,"id":"product_name","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":".product-header-left h1","type":"SelectorText"},{"delay":0,"id":"product_id","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":"span[itemprop='productID']","type":"SelectorText"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":".is-desktop div.product-sales-price","type":"SelectorText"},{"delay":0,"id":"average_rating","multiple":false,"parentSelectors":["product_wrapper"],"regex":"","selector":"span.bv-secondary-rating-summary-rating","type":"SelectorText"},{"id":"review-pagination","paginationType":"clickMore","parentSelectors":["product_wrapper","review-pagination"],"selector":"li.bv-content-pagination-buttons-item-next","type":"SelectorPagination"}]}

#3

@loreal Hi, it does not appear like the 'pagination' selector would be configured properly.

Here's a working example:

{"_id":"michaels-single-MASS","startUrl":["https://www.michaels.com/gildan-short-sleeve-adult-tshirt/M10093625.html"],"selectors":[{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":"a:contains(\" Next Reviews \")","type":"SelectorPagination"},{"delay":0,"id":"wrapper","multiple":true,"parentSelectors":["pagination"],"selector":"li[itemprop='review']:nth-of-type(n+2)","type":"SelectorElement"},{"delay":0,"id":"title","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"div.bv-content-title-container","type":"SelectorText"}]}

#4

@ViestursWS Thanks very much for your help! I tried your code but it couldn't scrape any data for some reason..

#5

@loreal Please, note that the pagination selector, in this case, is based on the element click selector and the results will start to appear once the scraper has clicked through all of the pages.

Learn more: My scraping job is running, although no results are being returned - Web Scraper Knowledge Base

For testing purposes, you can limit it, for example till the page HTML contains - '129–158 of 28289 Reviews'.

Limited pagination example
Limited pagination example1662×1335 233 KB

Example:

{"_id":"michaels-single-MASS-limited-pagination","startUrl":["https://www.michaels.com/gildan-short-sleeve-adult-tshirt/M10093625.html"],"selectors":[{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":"html:has(span[role=\"alert\"]:not(:contains(\"129–158 of 28289 Reviews\"))) a:contains(\" Next Reviews \")","type":"SelectorPagination"},{"delay":0,"id":"wrapper","multiple":true,"parentSelectors":["pagination"],"selector":"li[itemprop='review']:nth-of-type(n+2)","type":"SelectorElement"},{"delay":0,"id":"title","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"div.bv-content-title-container","type":"SelectorText"}]}

#6

Brilliant, thanks a lot @ViestursWS. It's working now