Tripadvisor pagination - no way to get through whole pagination

How to?
#1

Hi folks,

I am trying to scrape names, addresses and some other data of all listed restaurants in Munich.

Start URL is "https://www.tripadvisor.de/Restaurants-g187309-Munich_Upper_Bavaria_Bavaria.html#EATERY_LIST_CONTENTS"

I was able to open detail sites of about 30 restaurants for which I got name and address, but then the scraper just stopped.

So far I tried to use "work arounds" I found in the forum and modified them for my needs - but it didn't work properly.

Can someone please help me and set up the "frame" which I can modify?

Thank you so much in advance!

#2

Hey there!

In my attempt to get better at scraping, I tried to solve your problem! Here is what I got:

{"_id":"tripadvisor","startUrl":["https://www.tripadvisor.com/Restaurants-g60898-Atlanta_Georgia.html"],"selectors":[{"id":"element","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.listing:nth-of-type(n+3)","multiple":true,"delay":0,"clickElementSelector":"a.nav.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":"a.property_title","multiple":false,"regex":"","delay":0}]}

This is just the framework to get your scraper moving through all the pages, in this sitemap I just made a company title text selector just to check if I could get it to move through every page and get the info. It seems to work as intended, though I have not yet tried getting each popup's info extracted. Ill post with updates if I can get titles with addresses from the popup links!

PS: I did this on the American TripAdvisor and used Atlanta as an example city, don't try forget to change the starting URL if you want to use my sitemap!

#3

I came up with a similar solution, added a delay into the element click selector and used unique HTML instead of CSS (truthfully, not sure why but it seems to be working). You'll need to add in all the text selectors you want

{"_id":"forum-trip-advisor","startUrl":["https://www.tripadvisor.de/Restaurants-g187309-Munich_Upper_Bavaria_Bavaria.html#EATERY_LIST_CONTENTS"],"selectors":[{"id":"Page Changer ","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.listing:nth-of-type(n+3)","multiple":true,"delay":"2000","clickElementSelector":"a.nav:last","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTML"},{"id":"Link-in","type":"SelectorLink","parentSelectors":["Page Changer "],"selector":"a.property_title","multiple":false,"delay":0},{"id":"Name","type":"SelectorText","parentSelectors":["Link-in"],"selector":"h1.ui_header","multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["Link-in"],"selector":"div.blEntry.phone","multiple":false,"regex":"","delay":0},{"id":"Website","type":"SelectorElementAttribute","parentSelectors":["Link-in"],"selector":"div.is-hidden-mobile.website span.detail","multiple":false,"extractAttribute":"href","delay":0}]}
#4

thank you, it's exactly what I was looking for!

#5

thanks mate! use bfretfeig's solution now :wink:

1 Like
#6