URL selector for multiple external sites

How to?
#1

I'm compiling a list of contacts for substance abuse facilities. Psychology Today has a great repository with many contacts. I was able to scrape their comprehensive list of about 39k contacts. This returned name, location, and phone number. The issue is I need email and URL for each contact. The site itself offers email contact through a form that obscures the actual email address from view. It does the same for the counselors websites. I'm able to create a link selector to follow each website link, but pulling the actual URL is proving difficult once I hit their sites, since they are all structured differently. Anyone have a workaround for this?

https://www.psychologytoday.com/us/therapists/addiction/

Once at this link click on a state, then a provider, then the "website" link to see what I mean.

Sitemap:
{"_id":"pt_add_w_site","startUrl":["https://www.psychologytoday.com/us/therapists/addiction"],"selectors":[{"id":"State Pagination","type":"SelectorLink","parentSelectors":["_root"],"selector":".us_region_list a","multiple":true,"delay":0},{"id":"ind_state_pagination","type":"SelectorLink","parentSelectors":["State Pagination","ind_state_pagination"],"selector":"a.pager-page","multiple":true,"delay":0},{"id":"Ind Link","type":"SelectorLink","parentSelectors":["State Pagination","ind_state_pagination"],"selector":"a[itemprop='url']","multiple":true,"delay":0},{"id":"Name","type":"SelectorText","parentSelectors":["Ind Link"],"selector":".hidden-sm-down h1","multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["Ind Link"],"selector":"[itemprop='telephone'] a","multiple":false,"regex":"","delay":0},{"id":"city","type":"SelectorText","parentSelectors":["Ind Link"],"selector":".hidden-sm-down span[itemprop='addressLocality']","multiple":false,"regex":"","delay":0},{"id":"state","type":"SelectorText","parentSelectors":["Ind Link"],"selector":".hidden-sm-down span[itemprop='addressRegion']","multiple":false,"regex":"","delay":0},{"id":"web_link","type":"SelectorLink","parentSelectors":["Ind Link"],"selector":"a.btn-md.btn-default","multiple":false,"delay":0}]}

#2

This site uses server-side redirects and email forms, so WS probably can't get those data. The data is not embedded within the code.

There's a fairly crude method you can try which involves using the <body> tag and grabbing all the text from the page, but you would need post-processing to get useful data. e.g.

{"_id":"forum-psychologytoday-test","startUrl":["https://www.psychologytoday.com/us/therapists/addiction/north-carolina/313043?sid=5e769c59d485d&ref=1&tr=ResultsProfileBtn"],"selectors":[{"id":"Name","type":"SelectorText","parentSelectors":["_root"],"selector":".hidden-sm-down h1","multiple":false,"regex":"","delay":0},{"id":"Website","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.btn-md.btn-default","multiple":false,"delay":0},{"id":"grab-text","type":"SelectorText","parentSelectors":["Website"],"selector":"body","multiple":false,"regex":"","delay":0}]}

#3

Thanks for the response! I'm not sure the body tag would have the information I need. I looked at the whole page source and I don't see any email addresses in it. I've been trying to scrape emails from here also: https://www.naadac.org/sap-directory?locsearch=22314&loccountry=US&locdistance=any&sortdir=distance-asc
Once you click on the name, a contact window pops up in the map on the right. The email is contained in the popup. Here is what I've been trying to do:
{"_id":"naadac_directory","startUrl":["https://www.naadac.org/sap-directory?locsearch=22314&loccountry=US&locdistance=any&sortdir=name-asc"],"selectors":[{"id":"Contact Wrapper","type":"SelectorElement","parentSelectors":["_root"],"selector":"li.places-app-location-item","multiple":true,"delay":0},{"id":"Name","type":"SelectorText","parentSelectors":["Contact Wrapper"],"selector":"span.places-app-location-name","multiple":false,"regex":"","delay":0},{"id":"Street","type":"SelectorText","parentSelectors":["Contact Wrapper"],"selector":"span.places-app-location-street","multiple":false,"regex":"","delay":0},{"id":"City_State_Zip","type":"SelectorText","parentSelectors":["Contact Wrapper"],"selector":"span.places-app-location-citystatezip","multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["Contact Wrapper"],"selector":".places-app-location-phone a","multiple":false,"regex":"","delay":0},{"id":"link info","type":"SelectorLink","parentSelectors":["Contact Wrapper"],"selector":"span.places-app-location-name","multiple":true,"delay":0},{"id":"email","type":"SelectorText","parentSelectors":["link info"],"selector":".gm-style-iw-d p.places-app-location-description","multiple":true,"regex":"","delay":0}]}

Is the link or popup link selector the correct one for this application?