Impossible to scrap Mail in TripAdvisor

MqzakpoOk · October 12, 2018, 9:31am

Hi there,
I encounter a problem in the following sitemap,
To experience the problem, please go on the following link. Inspect the page, then import the sitemap. Go to exploreur/Fiche/ and you will be able to experiment what i mean.

The email of the following URL is displayed. Once you click on it, a popup start and display the right Email of the contact.
But i did not find a way to scrap that Email. Usually in a similar case, i use the "link" option, but this Email link seem to not be recognized as a link by webscraper.
Only Linkpopup is giving me data but the href si "null"

Any thoughts about how to recognize that email ? Is TripAdvisor starting to protect their Emails ? Is there a potential new functionnality to devellop to be able to get around that ?

Thank you very much for your advices,

Url: https://www.tripadvisor.fr/Attraction_Review-g187147-d1600380-Reviews-La_Cuisine_Paris_Cooking_Classes-Paris_Ile_de_France.html

nctbrtc · October 12, 2018, 1:42pm

where is thee email in the page? Can you share a screenshot ?

bretfeig · October 12, 2018, 10:25pm

Not sure about this one. Click on e-mail runs, what i think is a java script, that calls an open-window function. Not sure how to scrape. Anton ? @iconoclast

MqzakpoOk · October 15, 2018, 7:18am

Hi there,
Thank you Bretfreig and nctbrtc,

Sorry the link i shared was the root of the Sitemap, here is the right page where your can see the Email link below on the right content box.

I tried to share a screen but it didnt want to upload :
Here it work now :

Yes it does call an open-window function but i use to be able to scrap that kind of link on other website but here i'm struggling,

If anybody has a solution i would be very happy

Thank you

iconoclast · October 20, 2018, 7:34pm

Hello there!

Have you tried clicking an E-mail button? Does it bring anything? It doesn't work for me at all.

MqzakpoOk · October 22, 2018, 8:10am

Hello Iconoclast,

Thanks for your answer,
Yes i tried the element click
I also tried using a link as a child of element click but always bring null

Maybe there is something hiding the real Email ?

I don't know ...

MqzakpoOk · October 22, 2018, 9:25am

hi there @iconoclast ,

Here is some news. I discussed it with my developper.
Here is what we found out :

The adress is effectively not on the code of the page
TripAdvisor use a JS to call an open window to redirect to your default mail manager
The Email of the contact appear in the URL of your mail manager (see screenshot)

Now the question is : With webscraper, is it possible to get URL (http) from website that use JavaScript to redirect to a different link ? If yes, how ? And if not, is there any update coming up soon for it ?

For exemple, here i try to use a click selector on email, then it open, in the same window, my gmail with the adresse. So i try to use as a child a text selector, but it does not recognize the parent selector (the Email redirection link in JS)

New functionnality to implement ?

Thank you for your help !
Have a nice one

iconoclast · October 22, 2018, 10:00am

It can be done using Tampermonkey script. I need some time to make one.

MqzakpoOk · October 22, 2018, 12:13pm

I don't know much about scripting, but i found that, i don't know if it could be of any help (apparently script to block a pop-up) : https://greasyfork.org/fr/scripts/37654-popup-blocker-script

rysioslaw · December 19, 2018, 3:44am

It's very interesting.
We should make an application to store parameters to the .txt file and setup this application as default mail manager.
Or just use .bat file and compile using bat2exe.

MqzakpoOk · January 31, 2019, 8:54am

Any news on webscraper to be able to get these email displayed on the Urls ?

Thanks in advance for your help

leemeng · September 18, 2019, 1:26pm

On my PC, I have a separate instance of Chrome used only for scraping. So it does not have any mail program set up, meaning if I click on any email or mailto:, it doesn't do anything.

For the tripadvisor example above, I clicked on the Email button and nothing happened on screen. But checking the inspector, I found that the server will return the email as a href in the HTML:

Once the email appears, it is only a matter of scraping it with Element Attribute selector:

2019-09-18_212152

Settings I used:
Type: Element Attribute
Selector: div.attractions-contact-card-ContactCard__contactCard--zMh3H > div:nth-child(2) > div:nth-child(3) > div > div > a
Attribute: href

Of course, you would need to figure out the site navigation and button clicking first.

MqzakpoOk · September 19, 2019, 9:32am

Thank you so much @leemeng for this update,

Very interresting ... You gave me some hope
But ... It does not work for me for some reasons

Where do you get this link " div.attractions-contact-card-ContactCard__contactCard--zMh3H > div:nth-child(2) >" ? Me, when i inspect the element of the Email, here is what i get :

Here is a sample of the data i can get :
Email is returning me a "null" result with the selector as you said

{"_id":"tripadvisor_test","startUrl":["https://www.tripadvisor.fr/Attractions-g11038854-Activities-c41-Bourgogne_Franche_Comte.html","https://www.tripadvisor.fr/Attractions-g2349678-Activities-c56-t272-Haute_Garonne_Occitanie.html","https://www.tripadvisor.fr/Attractions-g187153-Activities-c56-Montpellier_Herault_Occitanie.html"],"selectors":[{"id":"Exploreur","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.listing_details","multiple":true,"delay":"5000","clickElementSelector":"a.next:last","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"fiche","type":"SelectorLink","parentSelectors":["Exploreur"],"selector":".listing_info > a","multiple":true,"delay":0},{"id":"Nom","type":"SelectorText","parentSelectors":["fiche"],"selector":"h1.ui_header","multiple":false,"regex":"","delay":0},{"id":"Popularité","type":"SelectorText","parentSelectors":["fiche"],"selector":"span.header_popularity","multiple":false,"regex":"","delay":0},{"id":"Adresse1","type":"SelectorText","parentSelectors":["fiche"],"selector":"span.detail span.street-address","multiple":false,"regex":"","delay":0},{"id":"Adresse2","type":"SelectorText","parentSelectors":["fiche"],"selector":"span.detail span.locality","multiple":false,"regex":"","delay":0},{"id":"Adresse3","type":"SelectorText","parentSelectors":["fiche"],"selector":"span.detail span.country-name","multiple":false,"regex":"","delay":0},{"id":"Types","type":"SelectorText","parentSelectors":["fiche"],"selector":"span.is-hidden-mobile","multiple":false,"regex":"","delay":0},{"id":"Urls","type":"SelectorPopupLink","parentSelectors":["fiche"],"selector":"div.contactType.website span.taLnk","multiple":false,"delay":0},{"id":"Emails","type":"SelectorElementAttribute","parentSelectors":["fiche"],"selector":"div.attractions-contact-card-ContactCard__contactCard--zMh3H > div:nth-child(2) > div:nth-child(3) > div > div > a","multiple":false,"extractAttribute":"href","delay":"0"}]}

leemeng · September 19, 2019, 10:10am

Hi you would need to click on the Email button (or use Element Click) to trigger the server. If it works, you'll see the see the email element pop up in real time in the inspector.

But as I mentioned, this only seems to work if you have a "clean" copy of Chrome which does not have email client set up (i.e. Chrome must not intercept mailto: links).

MqzakpoOk · September 19, 2019, 10:26am

Thank you for the reply,

Do you have any sitemap to display ?
So you are saying i need to have an element attribute as a child of an element click with a lean chrome interface ?

I'm gonna try Thank you

But i'm still curious of how you get this inspection like tris ...

In a link like this one : https://www.tripadvisor.fr/Attraction_Review-g187147-d3704252-Reviews-Le_Foodist-Paris_Ile_de_France.html
i get this :

On a link like this one : https://www.tripadvisor.fr/Attraction_Review-g187109-d1792256-Reviews-Burgundy_Wine_School-Beaune_Cote_d_Or_Bourgogne_Franche_Comte.html

So there is two tipes of presentation and so two different element attributes to display i suppose.

And also, i'm curious how do you see the element afterclick ?
Thank you for your help you save me hips of time

leemeng · September 19, 2019, 1:31pm

I can see the email when tripadvisor shows the Burgundy Wine layout, but not for the other type of layout. I don't know how to extract that one.

Video_2019-09-19_210730

I'm not really sure how to replicate this. It kinda works by accident for me because I don't set any default mailto: handler, i.e. I don't want email links to launch anything. If I recall, when you launch Chrome for the first time, it will eventually ask you if you want Chrome to handle mailto: links, or something like that. I just click No. There may also be a similar setting in Windows.

MqzakpoOk · September 19, 2019, 1:53pm

Thank you very much @leemeng,

I think there is much more data displayed as the first encoded state than the "open" state like Burgondy where the email seem to pop into the HTML,

I suppose it exist a workaround that @NetworkReject tried to explain to me intothis subject :

By trying to pull the "data-encoded-url" of the attribute and then decode it with a tierce software,
So i suppose we have all the element not far but i still don't manage to get it right.

I'll try to dig it deeper by setting my chrome out of the mailto: and i'll try to keep you updated with further avancement ...

So frustrating though that i was able few month ago to get it right and now i'm incapable of getting any data.

Thank you for your help

leemeng · September 19, 2019, 11:58pm

If you want to explore more, try the link below. It is currently beyond my skills and I don't know how to integrate it with WS:

https://kaijento.github.io/2017/03/17/web-scraping-tripadvisor.com/

leemeng · September 20, 2019, 2:59am

LATEST: I have found a solution for the "Le Foodist" type of layout. Turns out the cleartext for both the website and email is stored right in the source code. But it not located anywhere near contact info elements.

So I just used HTML element to grab body, and I picked out the email and website with Regex. A bit crude (we're basically grabbing the source code for entire page ) but it seems to work here. You will need to do some post-cleanup but it should be straightforward. This sitemap works on my PC:

{"_id":"tripadvisor_get_email","startUrl":["https://www.tripadvisor.fr/Attraction_Review-g187147-d3704252-Reviews-Le_Foodist-Paris_Ile_de_France.html"],"selectors":[{"id":"name","type":"SelectorText","parentSelectors":["_root"],"selector":"h1.ui_header","multiple":false,"regex":"","delay":0},{"id":"get_email","type":"SelectorHTML","parentSelectors":["_root"],"selector":"body","multiple":false,"regex":"email\":\"[\\w_@\\.\\-]{3,99}\\b","delay":0},{"id":"get_website","type":"SelectorHTML","parentSelectors":["_root"],"selector":"body","multiple":false,"regex":"website\":\"[^\"]{5,125}\\b","delay":0}]}

MqzakpoOk · September 20, 2019, 1:20pm

@leemeng, thank you so much !!!

I had a very rough day and a tough week to be honest,
And your post just cleared all my mind and soul before i can take my holliday,

I can be now i peace for the all year xD

Thank you so much ! It seems to work.

I'll see when i come back from holliday

Have a good one !