I'm trying to scrape the reviews of this link (below), but instead of 'next' button it has 'see all reviews' and when I clicked on it, the URL changed to ..../reviews/. I tried using link selector, but all got 51 reviews so far. The problem is the ..../reviews/ link also has dynamic pages. And since the main product page is not the same link as the reviews, I'm not sure how to go with it. Also, would it be possible to get all the reviews without specifying the specific pages (i.e. it can detect the ending page).
I'd like to scrape all the reviews, ideally. Is there anything wrong in my sitemap?
Url: Gildan G500 Black Adult Heavy Cotton™ 5.3 oz. T-Shirt | JiffyShirts
Sitemap:
{"_id":"jiffyshirts","startUrl":["https://www.jiffyshirts.com/gildan-G500.html?ac=Black§ion=Our+Best+Selling+T-Shirts"],"selectors":[{"delay":0,"id":"page_selector","multiple":true,"parentSelectors":["_root","page_selector"],"selector":"a.reviews-list__all-reviews-link","type":"SelectorLink"},{"delay":0,"id":"review_elements","multiple":true,"parentSelectors":["_root","page_selector","pagination_review"],"selector":"div.review-has-enhanced-data","type":"SelectorElement"},{"delay":0,"id":"review_rating","multiple":false,"parentSelectors":["review_elements"],"regex":"","selector":"i","type":"SelectorText"},{"delay":0,"id":"review_name_and_date","multiple":false,"parentSelectors":["review_elements"],"regex":"","selector":"div.reviews-list-item__publish-info","type":"SelectorText"},{"delay":0,"id":"review_body","multiple":false,"parentSelectors":["review_elements"],"regex":"","selector":"p","type":"SelectorText"},{"delay":0,"id":"review_title","multiple":false,"parentSelectors":["review_elements"],"regex":"","selector":"div.reviews-list-item__title","type":"SelectorText"},{"delay":0,"id":"product_elements","multiple":false,"parentSelectors":["_root","page_selector"],"selector":"div.main-content-container","type":"SelectorElement"},{"delay":0,"id":"product_title","multiple":false,"parentSelectors":["product_elements"],"regex":"","selector":"h1.product-details-title","type":"SelectorText"},{"delay":0,"id":"average_rating","multiple":false,"parentSelectors":["product_elements"],"regex":"","selector":"div.review-rating__avg-rating","type":"SelectorText"},{"delay":0,"id":"recommend_percent","multiple":false,"parentSelectors":["product_elements"],"regex":"","selector":"div:nth-of-type(4) a.product-details-user-feedback__link","type":"SelectorText"},{"clickElementSelector":"a.reviews-list__all-reviews-link","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":15000,"discardInitialElements":"do-not-discard","id":"show_all","multiple":true,"parentSelectors":["page_selector"],"selector":"div.product-details__section-content:nth-of-type(1)","type":"SelectorElementClick"},{"id":"pagination_review","paginationType":"clickOnce","parentSelectors":["show_all","pagination_review"],"selector":"a.jiffy-pagination__number","type":"SelectorPagination"}]}
Would appreciate any help. Thank you very much