Limit TripAdvisor Next Selector

StuCPT · September 6, 2019, 9:02pm

I am trying to scrape TripAdvisor but only want to scrape the records for the first 10 pages - i.e. limit the Element click selector to 10 so that it opens the restaurant links on each of the first 10 pages and scrapes the 30 restaurant results. per page giving me a total of 300 results.

I have tried using the n:th child solution suggested by @iconocalst but can't get this to work.

Url: https://www.tripadvisor.co.za/Restaurants-g60878-Seattle_Washington.html

Sitemap:
{"_id":"tripadvisor","startUrl":["https://www.tripadvisor.co.za/Restaurants-g60878-Seattle_Washington.html"],"selectors":[{"id":"Next","type":"SelectorElementClick","parentSelectors":["_root","Next"],"selector":"div.restaurants-list-ListCell__cellContainer--2mpJS:nth-of-type(n+3) a.restaurants-list-ListCell__restaurantName--2aSdo","multiple":true,"delay":"2000","clickElementSelector":"a.nav.next:nth-child(-n+9)","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTML"},{"id":"Restaurant Link","type":"SelectorLink","parentSelectors":["_root","Next"],"selector":"div.restaurants-list-ListCell__cellContainer--2mpJS:nth-of-type(n+3) a.restaurants-list-ListCell__restaurantName--2aSdo","multiple":true,"delay":0},{"id":"Restaurant Name","type":"SelectorText","parentSelectors":["Restaurant Link"],"selector":"h1.ui_header","multiple":false,"regex":"","delay":0},{"id":"Claimed","type":"SelectorText","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-claimed-badge-ClaimedBadge__container--32Ufv","multiple":false,"regex":"","delay":0},{"id":"Overall Rating","type":"SelectorText","parentSelectors":["Restaurant Link"],"selector":"span.restaurants-detail-overview-cards-RatingsOverviewCard__overallRating--nohTl","multiple":false,"regex":"","delay":0},{"id":"Number of reviews","type":"SelectorText","parentSelectors":["Restaurant Link"],"selector":"a.restaurants-detail-overview-cards-RatingsOverviewCard__ratingCount--DFxkG","multiple":false,"regex":"","delay":0},{"id":"Ranking","type":"SelectorText","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-RatingsOverviewCard__ranking--17CmN","multiple":false,"regex":"","delay":0},{"id":"Awards","type":"SelectorText","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-RatingsOverviewCard__award--31yzt","multiple":false,"regex":"","delay":0},{"id":"Rating Food","type":"SelectorElement","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-RatingsOverviewCard__ratingQuestionRow--5nPGK:nth-of-type(1)","multiple":false,"delay":0},{"id":"Rating Service","type":"SelectorElement","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-RatingsOverviewCard__ratingQuestionRow--5nPGK:nth-of-type(2)","multiple":false,"delay":0},{"id":"Rating Value","type":"SelectorElement","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-RatingsOverviewCard__ratingQuestionRow--5nPGK:nth-of-type(3)","multiple":false,"delay":0},{"id":"Rating Atmosphere","type":"SelectorElement","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-RatingsOverviewCard__ratingQuestionRow--5nPGK:nth-of-type(4)","multiple":false,"delay":0},{"id":"Mentioned In","type":"SelectorHTML","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-MentionsOverviewCard__mentionsList--3bwkV","multiple":false,"regex":"","delay":0},{"id":"Details","type":"SelectorHTML","parentSelectors":["Restaurant Link"],"selector":".restaurants-details-card-DetailsCard__innerDiv--1Imq5 div.ui_columns","multiple":false,"regex":"","delay":0},{"id":"View all details","type":"SelectorElementClick","parentSelectors":["Restaurant Link"],"selector":"a.restaurants-detail-overview-cards-DetailsSectionOverviewCard__viewDetails--ule3z","multiple":false,"delay":"1000","clickElementSelector":"a.restaurants-detail-overview-cards-DetailsSectionOverviewCard__viewDetails--ule3z","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Details Pop Up","type":"SelectorHTML","parentSelectors":["Restaurant Link"],"selector":"div.restaurants-detail-overview-cards-DetailsSectionOverviewCard__detailsContent--1hucM","multiple":false,"regex":"","delay":0},{"id":"Rating - Food","type":"SelectorText","parentSelectors":["Rating Food"],"selector":"span.restaurants-detail-overview-cards-RatingsOverviewCard__ratingText--1P1Lq","multiple":false,"regex":"","delay":0},{"id":"Rating Food #","type":"SelectorElementAttribute","parentSelectors":["Rating Food"],"selector":"span.ui_bubble_rating","multiple":false,"extractAttribute":"class","delay":0},{"id":"Rating - Service","type":"SelectorText","parentSelectors":["Rating Service"],"selector":"span.restaurants-detail-overview-cards-RatingsOverviewCard__ratingText--1P1Lq","multiple":false,"regex":"","delay":0},{"id":"Rating - Service #","type":"SelectorElementAttribute","parentSelectors":["Rating Service"],"selector":"span.ui_bubble_rating","multiple":false,"extractAttribute":"class","delay":0},{"id":"Rating - Value","type":"SelectorText","parentSelectors":["Rating Value"],"selector":"span.restaurants-detail-overview-cards-RatingsOverviewCard__ratingText--1P1Lq","multiple":false,"regex":"","delay":0},{"id":"Rating - Value #","type":"SelectorElementAttribute","parentSelectors":["Rating Value"],"selector":"span.ui_bubble_rating","multiple":false,"extractAttribute":"class","delay":0},{"id":"Rating - Atmosphere","type":"SelectorText","parentSelectors":["Rating Atmosphere"],"selector":"span.restaurants-detail-overview-cards-RatingsOverviewCard__ratingText--1P1Lq","multiple":false,"regex":"","delay":0},{"id":"Rating -Atmosphere #","type":"SelectorElementAttribute","parentSelectors":["Rating Atmosphere"],"selector":"span.ui_bubble_rating","multiple":false,"extractAttribute":"class","delay":0}]}

leemeng · September 7, 2019, 1:49am

Hi, have you tried the ":not" CSS selector, as described by iconoclast in this post (scroll to bottom).

a.nav.next:not([data-page-number=11])

which basically means, "keep clicking on the Next button as long as the page number is not 11".

Anyway, here's a modified version of his scrape. This will just grab the restau name and type. Modify for your needs. I purposely made the name selector area larger so it would capture the listing number and the word SPONSORED. For test scrape, I used Page load delay (ms): 5000

{"_id":"tripadvisor_seattle","startUrl":["https://www.tripadvisor.co.za/Restaurants-g60878-Seattle_Washington.html"],"selectors":[{"id":"clicker","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.hotels_lf_redesign","multiple":true,"delay":"4000","clickElementSelector":"a.nav.next:not([data-page-number=11])","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"page_number","type":"SelectorText","parentSelectors":["clicker"],"selector":"span.pageNum.current","multiple":false,"regex":"","delay":0},{"id":"listing_selector","type":"SelectorElement","parentSelectors":["clicker"],"selector":".restaurants-list-ListCell__cellContainer--2mpJS","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["listing_selector"],"selector":"div.restaurants-list-ListCell__titleRow--3rRCX","multiple":false,"regex":"","delay":0},{"id":"type","type":"SelectorText","parentSelectors":["listing_selector"],"selector":".restaurants-list-ListCell__cuisinePriceMenu--r4-Re span:nth-of-type(1) span.restaurants-list-ListCell__infoRowElement--2E6E3","multiple":false,"regex":"","delay":0}]}

This method still works 'cos the tripadvisor Next element still contains the next page number.

StuCPT · September 7, 2019, 7:26am

@leemeng thank you very much. The pagination works perfectly now.

I have hit another issue in that the scraper is only scraping the results from the final page so I am only getting 30 results.

There seems to be some issue with my parent selectors for the restaurant link.

leemeng · September 7, 2019, 11:26am

That is quite a scraper. I restructured your sitemap and expanded the main selector. Now it looks like this:
2019-09-07_192038

I tested with 3 pages and it got 90 listings as expected. You can change the number of pages later:

StuCPT · September 7, 2019, 5:42pm

Amazing @leemeng thank you. You saved me a lot of time trying to figure this out.

Last question:

Is there a way to change the click selector " a.nav.next:not([data-page-number=8]) " to instaed do pages between - i.e. between 5 and 10?

leemeng · September 8, 2019, 1:32am

I don't know how to do that with this selector, but fortunately there is another way to navigate the tripadvisor pages which you can use to achieve this.

If you scroll down to the page navigation area and right-click -> Copy link address, you'll notice the URL is slightly different for each page. e.g. Page 5 has this bit in the middle:

/RestaurantSearch-g60878-oa120-Seattle_Washington.html#EATERY_LIST_CONTENTS

The URL for Page 6 has oa150 in the middle, and so on. It increases by 30 each page, presumably because there are 30 listings per page.

Now that you know the pattern, you just need to do a bit of math and use the range url with increment method to get the specific pages you need. You'd need to make a copy of your scraper without the paginator.

StuCPT · September 9, 2019, 9:33pm

Thank you @leemeng this is great