Limit pages on Tripadvisor Forum

How to?
#1

Hi. I'm trying to scrape the questions/comments posted on the Philippines page of the tripadvisor forum but I want to limit the number of pages scraped to the first 50 pages. I tried using the ":not" CSS selector and by using the range url method as suggested in this page but I can't make it work.

Url: https://www.tripadvisor.com/ShowForum-g294245-i3256-Philippines.html

Here's my current sitemap:
{"_id":"tripadvisor-ph","startUrl":["https://www.tripadvisor.com/ShowForum-g294245-i3256-Philippines.html"],"selectors":[{"id":"questions","type":"SelectorLink","parentSelectors":["Main Page"],"selector":"tr:nth-of-type(n+7) b a","multiple":true,"delay":0},{"id":"question title","type":"SelectorText","parentSelectors":["questions"],"selector":".bx01 div.postTitle","multiple":false,"regex":"","delay":0},{"id":"question date","type":"SelectorText","parentSelectors":["questions"],"selector":".bx01 div.postDate","multiple":false,"regex":"","delay":0},{"id":"questioner name","type":"SelectorText","parentSelectors":["questions"],"selector":".bx01 div.username","multiple":false,"regex":"","delay":0},{"id":"question body","type":"SelectorText","parentSelectors":["questions"],"selector":".bx01 div.postBody","multiple":false,"regex":"","delay":0},{"id":"questioner location","type":"SelectorText","parentSelectors":["questions"],"selector":".profile div.location","multiple":false,"regex":"","delay":0},{"id":"Main Page","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.balance","multiple":true,"delay":"5000","clickElementSelector":"div.deckTools:nth-of-type(6) a:nth-of-type:not[(100)]","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"}]}

I would appreciate all you help.

Thanks in advance!

#2

Refer to:

#3

Hi there. Spent about 10 hours to read the forum and trying to scrap info from tripadvisor. Tried with paginator and with multiple URLs with ranges, tried to copy already suggested examples but didn't get through.
I'm just trying to scrape basic info about restaurants (like: website, email, phone etc.) and can't do more than 2-3 pages. If I will use multiple URLs with ranges and need to scrape all the results it will be a huge amount of links (about 140-150). Is there any other possibilities?

My last sitemap -
{"_id":"new-type","startUrl":["https://www.tripadvisor.com/Restaurants-g60713-San_Francisco_California.html"],"selectors":[{"id":"paginator3000","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.is-9","multiple":true,"delay":"2000","clickElementSelector":"a.nav","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"link-to","type":"SelectorLink","parentSelectors":["paginator3000"],"selector":"a._15_ydu6b","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["link-to"],"selector":"h1.ui_header","multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorPopupLink","parentSelectors":["link-to"],"selector":".restaurants-detail-overview-cards-LocationOverviewCard__contactItem--1flT6 ._2wKz--mA span.restaurants-detail-overview-cards-LocationOverviewCard__detailLinkText--co3ei","multiple":false,"delay":0},{"id":"email","type":"SelectorLink","parentSelectors":["link-to"],"selector":".restaurants-detail-overview-cards-LocationOverviewCard__detailLink--iyzJI span a","multiple":false,"delay":0},{"id":"address","type":"SelectorText","parentSelectors":["link-to"],"selector":".restaurants-detail-overview-cards-LocationOverviewCard__addressLink--1pLK4 ._2wKz--mA span.restaurants-detail-overview-cards-LocationOverviewCard__detailLinkText--co3ei","multiple":false,"regex":"","delay":0},{"id":"phone","type":"SelectorText","parentSelectors":["link-to"],"selector":"a span span.restaurants-detail-overview-cards-LocationOverviewCard__detailLinkText--co3ei","multiple":false,"regex":"","delay":0}]}

#4

Hi leemeng,

Thanks for the suggestion. I tried your suggestion for the first three pages but I only got results from the first page. If it's not much trouble, can you check what could be wrong with my sitemap?

Sitemap with range URLs:
{"_id":"tripadvisor-ph2","startUrl":["https://www.tripadvisor.com/ShowForum-g294245-i3256-[o20-o60:20]-Philippines.html"],"selectors":[{"id":"link","type":"SelectorLink","parentSelectors":["_root"],"selector":"tr:nth-of-type(n+7) b a","multiple":true,"delay":0},{"id":"body","type":"SelectorText","parentSelectors":["link"],"selector":".bx01 div.postBody","multiple":false,"regex":"","delay":0}]}

#5

Hi, there's a small error in your starturl's range. It should be

-o[20-60:20]

#6

Hi leemeng,

It's all working now. Thanks for the help!!