How can I limit pagination?

How to?
#1

I've created a good scrape getting all the data I want from Yelp, but I would also like to limit how much data is scraped. The pagination is just a little different within the url as the start=0 refers to the start of the reviews (i.e. review number 10 would be start=0).

I've tried to use a pagination technique to scrape only the first 100 reviews (start=[0-100]), but it doesn't work as hoped. Any suggestions?

Url: https://www.yelp.com/search?find_desc=Eyewear+%26+Opticians&find_loc=Salt+Lake+City,+UT&start=0&sortby=rating

Sitemap:
{"_id":"yelp","startUrl":["https://www.yelp.com/search?find_desc=Eyewear+%26+Opticians&find_loc=Salt+Lake+City,+UT&start=0&sortby=rating"],"selectors":[{"id":"business-name","type":"SelectorLink","selector":"span.indexed-biz-name a.biz-name","parentSelectors":["next"],"multiple":true,"delay":0},{"id":"name","type":"SelectorText","selector":"h1.biz-page-title","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorText","selector":"span.biz-website a","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0},{"id":"next","type":"SelectorElementClick","selector":"div.clearfix.scroll-map-container div.column.column-alpha","parentSelectors":["_root"],"multiple":true,"delay":"2000","clickElementSelector":"a.u-decoration-none.next span.icon","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"stars","type":"SelectorHTML","selector":"div.biz-rating.biz-rating-very-large.clearfix > div","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0},{"id":"review-count","type":"SelectorText","selector":"div.rating-info span.review-count","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0}]}

#2

Hi!

Array method doesn't work because pages increment by number of records shown (10 per page).

Solution is to use Element Click to click 'Next' button until all records are shown.

Your sitemap:
{"_id":"yelp","startUrl":["https://www.yelp.com/search?find_desc=Eyewear+%26+Opticians&find_loc=Salt+Lake+City,+UT&start=0&sortby=rating"],"selectors":[{"id":"business-name","type":"SelectorLink","selector":"a.biz-name","parentSelectors":["clicker"],"multiple":true,"delay":0},{"id":"name","type":"SelectorText","selector":"h1.biz-page-title","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorText","selector":"span.biz-website a","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0},{"id":"stars","type":"SelectorElementAttribute","selector":"div.rating-info div.i-stars","parentSelectors":["business-name"],"multiple":false,"extractAttribute":"title","delay":0},{"id":"review-count","type":"SelectorText","selector":"div.rating-info span.review-count","parentSelectors":["business-name"],"multiple":false,"regex":"","delay":0},{"id":"clicker","type":"SelectorElementClick","selector":"li.regular-search-result","parentSelectors":["_root"],"multiple":true,"delay":"2000","clickElementSelector":"a.u-decoration-none.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

#3

But isn't this just the same effect as what I've got? My sitemap will scrape all records, but I want to limit how many records are scraped. For instances, this sitemap has 113 records. How can I scrape the first 50 records? Do I just have to do a one-time click 5 times?

The numbers aren't a big deal here, but if I switch the city to LA, then I don't want the data for 1500 records.

#4

Oh, sorry, I think i misread your post.

You can limit your pagination using negative type range:
div.pagination-links div.arrange_unit:nth-of-type(-n+6) a.available-number
Please put it into your Click selector.

For further information please refer to this interesting page:
http://nthmaster.com/