How to scrap google map?

How to?
#1

i am trying to scrape google maps data from a url
https://www.google.com/maps/search/mysearch/@mylat,mylong,15z/data=!3m1!4b1
on first page i want to grab the title only which is a pop up link, scrapper is scraping all the pop up link correctly , but after pop up link i want to scrape title, address and phone number
title(text)
/
root --title(pop up link)-- address(text)
\
phone number(text)

the problem is i have selected pop up link as multiple but it works for only first record and it works only in data preview mode, when i export the csv there is no data.
One more thing when i press < or > to go to see more results i am not able to select < or > as selector and the url remains the same as above(https://www.google.com/maps/search/mysearch/@mylat,mylong,15z/data=!3m1!4b1)
how to work it out to scrape all records from first results page and pages after that when i press > to see next 20 results(google maps only shows 20 results at a time)
any help would be much appreciated
thanx in advance.

#2

Hi there!

Can you post your sitemap please?

#3

here it is:
{"_id":"sirsa_garments","startUrl":["https://www.google.com/maps/search/garments+in+sirsa/@29.5344205,75.0313998,15z/data=!3m1!4b1"],"selectors":[{"id":"title","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"h3.section-result-title span","multiple":true,"delay":"2000"},{"id":"shop_name","type":"SelectorText","parentSelectors":["title"],"selector":"h1.section-hero-header-title","multiple":false,"regex":"","delay":"2000"},{"id":"address","type":"SelectorText","parentSelectors":["title"],"selector":"div.section-info:nth-of-type(5) span.widget-pane-link","multiple":false,"regex":"","delay":"2000"},{"id":"mobile_no","type":"SelectorText","parentSelectors":["title"],"selector":"div.section-info-action-button.section-info-speak-numeral span.widget-pane-link","multiple":false,"regex":"","delay":"2000"}]}

#4

I dont know about anyone else but when I click on your start url it takes me to a google maps search with only 16 results all fit on 1 page and no next buttons.

Is that the correct url?

#5

no it has multiple results and also has next button as shown in the picture below.

scrape
scrape.png900×489 208 KB

#6

I do see what @prakhar describes, with "Showing results 1-20". (But when I click in, there's no further results. Google must have a "borderline" of say 19-20 results in total here.)

Would it help this exploration if I point out that the search results are similar to Google Places, where you search regular Google, then click on the insert for "More Places", getting to this related set of results:

https://www.google.com/search?client=ubuntu&hs=2Wm&q=garments+in+sirsa&npsic=0&rflfq=1&rlha=0&rllag=29537010,75038104,1078&tbm=lcl&ved=2ahUKEwiEhvvxi7rdAhUQSo8KHQ6YDwYQjGp6BAgAEDw&tbs=lrf:!2m1!1e2!2m1!1e3!3sIAE,lf:1,lf_ui:10&rldoc=1#rlfi=hd:;si:;mv:!3m12!1m3!1d5315.439545008157!2d75.03919609999998!3d29.536536299999998!2m3!1f0!2f0!3f0!3m2!1i375!2i147!4f13.1;tbs:lrf:!2m1!1e2!2m1!1e3!3sIAE,lf:1,lf_ui:10

(I'm using Ubuntu as indicated in the URL)

Would this alternative Google UI be easier to access with WebScraper.io?

#7

@prakhar, unfortunately it won't be possible to scrape it with current iteration of WebScraper cause you will need to click "Go back to results" each time you enter particular Store. It can be done with using Tampermonkey extension though, to automatically click to go back after specific time.

UPD: it's possible now in Firefox

#8

Why I do feel like I'm going to have to download firefox now lol

#9

can you please explain how to do it in firefox

#10

@prakhar

First of all, you have to either install WebScraper for FireFox browser OR get the latest DEV version for Chrome here.

As I've mentioned earlier, you need to click 'Go Back To Results' button for WebScraper to continue scraping through listings. That can be done using a second Element Click selector that will click it once listing is opened.

I've re-worked your sitemap, you've also made a mistake when added PopUp Link selector, as when you open a listing, it's not a PopUp, but a java script action that will work only when clicked.

Your sitemap:

{"_id":"sirsa_garments","startUrl":["https://www.google.com/maps/search/garments+in+sirsa/@29.5344205,75.0313998,15z/data=!3m1!4b1"],"selectors":[{"id":"title","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.widget-pane-content","multiple":true,"delay":"1000","clickElementSelector":"h3.section-result-title span","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"shop_name","type":"SelectorText","parentSelectors":["title"],"selector":"h1.section-hero-header-title","multiple":false,"regex":"","delay":"2000"},{"id":"address","type":"SelectorText","parentSelectors":["title"],"selector":"div.section-info:nth-of-type(5) span.widget-pane-link","multiple":false,"regex":"","delay":"2000"},{"id":"mobile_no","type":"SelectorText","parentSelectors":["title"],"selector":"div.section-info-action-button.section-info-speak-numeral span.widget-pane-link","multiple":false,"regex":"","delay":"2000"},{"id":"GoBack","type":"SelectorElementClick","parentSelectors":["title"],"selector":"button.section-back-to-list-button span","multiple":false,"delay":"1000","clickElementSelector":"button.section-back-to-list-button span","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}
2 Likes
#11

Is there any limit on scraping?
because it is scraping right on 1st set of 1-20 results shown
from next 21-40 results it only shows the first result.
and there is no selector for next button as shown in picture i send here

#12

can anybody help here?

#15

Refer to

#18