Scraper Tripadvisor's all reviews

How to?
#1

Describe the problem.

I'm trying to scraper all hotel's reviews from TripAdvisor, there are 1910 review spread across 382 pages.but every time the scraper just stops when it scrapers around 200 pages.I click the "refresh" button, but no scraperd data yet . Is there something that I can include to force it to go beyond this limit?

Thanks in advance!

Url: https://www.tripadvisor.com/Hotel_Review-g308272-d307540-Reviews-Pudong_Shangri_La_East_Shanghai-Shanghai.html

Sitemap:
{"_id":"tripadvisor_hotel","startUrl":["https://www.tripadvisor.com/Hotel_Review-g308272-d307540-Reviews-Pudong_Shangri_La_East_Shanghai-Shanghai.html"],"selectors":[{"id":"element","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.hotels-community-tab-common-Card__card--ihfZB:nth-of-type(n+3)","multiple":true,"delay":"2000","clickElementSelector":"span.pageNum, a.pageNum","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"read more","type":"SelectorElementClick","parentSelectors":["element"],"selector":"parent","multiple":false,"delay":"","clickElementSelector":"span.location-review-review-list-parts-ExpandableReview__cta--2mR2g","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"comment","type":"SelectorText","parentSelectors":["element"],"selector":"div.cPQsENeY","multiple":false,"regex":"","delay":0},{"id":"user and time","type":"SelectorText","parentSelectors":["element"],"selector":".social-member-event-MemberEventOnObjectBlock__event_type--3njyv span","multiple":false,"regex":"","delay":0},{"id":"living time","type":"SelectorText","parentSelectors":["element"],"selector":"span.location-review-review-list-parts-EventDate__event_date--1epHa","multiple":false,"regex":"","delay":0},{"id":"type","type":"SelectorText","parentSelectors":["element"],"selector":"span.location-review-review-list-parts-TripType__trip_type--3w17i","multiple":false,"regex":"","delay":0},{"id":"score","type":"SelectorElementAttribute","parentSelectors":["element"],"selector":"span.ui_bubble_rating","multiple":false,"extractAttribute":"class","delay":0},{"id":"location","type":"SelectorText","parentSelectors":["element"],"selector":"span.default","multiple":false,"regex":"","delay":0}]}

Stuck trying to scrap "Things to do" reviews from Trip advisor
#2

You're probably hitting WS's data limit. It seems to fail when there is too much data. One workaround is to limit your scraper, say to 100 pages each session. Tripadvisor's page urls do change, so you can use that to start and stop wherever you like.

You can use the Specify multiple urls with ranges method. Note: The Tripadvisor page url numbers change in the middle, not at the end.

#3

thank you for your reply. I tried this solution you said, but it still doesn't work. the scraper scrapes page 7 and page 1 2 3....,it's back to where it was before, seems like the "Specify multiple urls with ranges" method doesn't work. Did I do something wrong? looking forward your reply. thank you again.

this is my urls:

the urls like this "https://www.tripadvisor.com/Hotel_Review-g294212-d1234659-Reviews-or[0-30:5]-Fairmont_Beijing-Beijing.html#REVIEWS"

#4

Try it these two:
https://www.tripadvisor.com/Hotel_Review-g294212-d1234659-Reviews-or[5-30:5]-Fairmont_Beijing-Beijing.html
https://www.tripadvisor.com.my/Hotel_Review-g308272-d307540-Reviews-Pudong_Shangri_La_East_Shanghai-Shanghai.html

Sometimes you need to scrape the first page separately 'cos a range number like or0 would not be a valid url.