Hey guys, so I'm trying to scrape yellowpages using https://webscraper.io/ but unfortunately it will not click the next button to cycle through pages, has anybody else encountered this problem? If so, is there any solution?
Hi!
Please share your sitemap so we can help.
Sure thing.
{"_id":"autorepairyellowpages","startUrl":["https://www.yellowpages.com/search?search_terms=auto%20repair&geo_location_terms=[10001-99999]&page=[1-50]"],"selectors":[{"id":"Skeleton","type":"SelectorElementClick","selector":"div.search-results.organic div.v-card","parentSelectors":["_root"],"multiple":true,"delay":0,"clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"Name Link","type":"SelectorLink","selector":"div.info > h2.n a.business-name","parentSelectors":["Skeleton"],"multiple":false,"delay":0},{"id":"Name","type":"SelectorText","selector":"h1","parentSelectors":["Name Link"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"p.address span:nth-of-type(1)","parentSelectors":["Name Link"],"multiple":false,"regex":"","delay":0},{"id":"City","type":"SelectorText","selector":"p.address span:nth-of-type(2)","parentSelectors":["Name Link"],"multiple":false,"regex":"","delay":0},{"id":"State","type":"SelectorText","selector":"p.address span:nth-of-type(3)","parentSelectors":["Name Link"],"multiple":false,"regex":"","delay":0},{"id":"Zip","type":"SelectorText","selector":"p.address span:nth-of-type(4)","parentSelectors":["Name Link"],"multiple":false,"regex":"","delay":0},{"id":"Email","type":"SelectorLink","selector":"a.email-business","parentSelectors":["Name Link"],"multiple":false,"delay":0}]}
Do you realize that you have an array values in your sitemap?
https://www.yellowpages.com/search?search_terms=auto%20repair&geo_location_terms=[10001-99999]&page=[1-50]
At the moment WebScraper does not support having more than one array within sitemap URL.
What I can recommend you is to narrow the results, you can add a few URLs with page=[1-50] with different cities (unless you build a database).
P.S. you don't actually need any selectors for pagination if you're using pagination array