Scrape multiple pages

#1

Hello please help me how to scrape 5 pages at once. My sitemap which i created will only allow me to scrape 2 pages every time i click scrape button. I hope you can help me. thank you in advance

Url: https://www.yellowpages.com/

Sitemap:
{"_id":"twopage","startUrl":["https://www.yellowpages.com/search?search_terms=divorce+attorney&geo_location_terms=Phoenix%2C+AZ"],"selectors":[{"id":"company name","type":"SelectorLink","parentSelectors":["_root","page"],"selector":".result a.business-name","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["company name"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"phone number","type":"SelectorText","parentSelectors":["company name"],"selector":"p.phone","multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","parentSelectors":["company name"],"selector":"h2.address","multiple":false,"regex":"","delay":0},{"id":"email address","type":"SelectorLink","parentSelectors":["company name"],"selector":"a.email-business","multiple":false,"delay":0},{"id":"page","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.next","multiple":true,"delay":0}]}

Manage pagination
#2

This is because you also have to loop the pagination button, as currently it is executed only once.

This should work:

{"_id":"twopage","startUrl":["https://www.yellowpages.com/search?search_terms=divorce+attorney&geo_location_terms=Phoenix%2C+AZ"],"selectors":[{"id":"company name","type":"SelectorLink","parentSelectors":["_root","page"],"selector":".result a.business-name","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["company name"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"phone number","type":"SelectorText","parentSelectors":["company name"],"selector":"p.phone","multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","parentSelectors":["company name"],"selector":"h2.address","multiple":false,"regex":"","delay":0},{"id":"email address","type":"SelectorLink","parentSelectors":["company name"],"selector":"a.email-business","multiple":false,"delay":0},{"id":"page","type":"SelectorLink","parentSelectors":["_root","page"],"selector":"a.next","multiple":true,"delay":0}]}

#3

okay thank you. Do i have to edit metadata every time i will scrape different page with same process?

#4

okay it worked but my problem is it scraped all the pages. is there an option that it would scraped only 10 pages at a time?

#5

This site's pages change their Url, so you don't really need a paginator. You could use the Specify multiple urls with ranges method:

https://www.yellowpages.com/search?search_terms=divorce%20attorney&geo_location_terms=Phoenix%2C%20AZ&page=[2-5]

https://www.yellowpages.com/search?search_terms=divorce+attorney&geo_location_terms=Phoenix%2C+AZ

#6

thank you so much. you guys are really a big help.