Tripadvisor scrape stopping at around 4,000 reviews

Hope someone can help.

The Tripadvisor scrape I am running for reviews works perfectly fine for attractions with less than 4,000 reviews. However, anything more than that and the scrape just stops when it gets to that limit. Is there something that I can include to force it to go beyond this limit?

Url: https://www.tripadvisor.co.uk/Attraction_Review-g186362-d190720-Reviews-Blenheim_Palace-Woodstock_Oxfordshire_England.html

Sitemap:
{"_id":"march21mortonarboretum2","https://www.tripadvisor.co.uk/Attraction_Review-g186362-d190720-Reviews-Blenheim_Palace-Woodstock_Oxfordshire_England.html"],"selectors":[{"id":"elementselector","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".review-container","multiple":true,"delay":"2000","clickElementSelector":"div.mobile-more a.nav.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"},{"id":"link_review","type":"SelectorLink","parentSelectors":["elementselector"],"selector":"a.title","multiple":false,"delay":0},{"id":"wholereviewinsidewholereview","type":"SelectorText","parentSelectors":["link_review"],"selector":"span.fullText","multiple":false,"regex":"","delay":0},{"id":"username","type":"SelectorText","parentSelectors":["elementselector"],"selector":"div.info_text div:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"location","type":"SelectorText","parentSelectors":["elementselector"],"selector":"strong","multiple":false,"regex":"","delay":0},{"id":"revieweddate","type":"SelectorText","parentSelectors":["elementselector"],"selector":"span.ratingDate","multiple":false,"regex":"","delay":0},{"id":"rating","type":"SelectorElementAttribute","parentSelectors":["elementselector"],"selector":"span.ui_bubble_rating","multiple":false,"extractAttribute":"class","delay":0},{"id":"contributionsnumber","type":"SelectorText","parentSelectors":["elementselector"],"selector":"span.badgetext:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"helpfulvotes","type":"SelectorText","parentSelectors":["elementselector"],"selector":"span.badgetext:nth-of-type(4)","multiple":false,"regex":"","delay":0}]}

Thanks a lot

There's probably nothing wrong with your scraper and you've hit a limitation in WS. There is a known issue where WS will fail with large amounts of data. It seems to vary with different computers; on mine WS will fail at around the 5,000-6,000 record range. The workaround is to add a limiter to your scraper and stop at 4,000 records (or whichever number it seems to be failing). There is usually a way to scrape sites in batches.

TripAdvisor scraping is actually pretty well covered in this forum. e.g. this thread will show you how to limit scraping and also to resume it later from where you had stopped (scrape in batches).