Describe the problem:
Hi, I'm trying to scrape email addresses of listings on a craigslist sections. The problem I encounter is that once you click on a listing, you need to click a box to get the email, I am having problem selecting the email in the web scraper. Please help
Url: https://atlanta.craigslist.org/d/software-qa-dba-etc/search/sof
You weren't entirely wrong. When you define a click selector, there are two options to define. 1:Selector and 2(what it select) : Click Selector(what it clicks)
You can the second right but you had the first wrong. You needed to define the Selector element to be the box after you clicked on it.
Then you create a child element to scrape the e-mail.
See if this works for you
{"_id":"craig","startUrl":["https://atlanta.craigslist.org/d/software-qa-dba-etc/search/sof"],"selectors":[{"id":"Name","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.result-title","multiple":true,"delay":0},{"id":"email","type":"SelectorElementClick","parentSelectors":["Name"],"selector":"aside.reply-flap","multiple":false,"delay":"2000","clickElementSelector":"button.reply_button","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"Email","type":"SelectorText","parentSelectors":["email"],"selector":"a.mailapp","multiple":false,"regex":"","delay":0}]}
Wanted to piggyback on this conversation -- i think i figured this out... Now trying to make it so it moves on to the next listing and repeats itself ... help?:
{"_id":"chattanooga-fsbo","startUrl":["https://chattanooga.craigslist.org/search/reo?search_distance=30&postal=37402&availabilityMode=0&housing_type=2&housing_type=4&housing_type=6&housing_type=9&housing_type=12&sale_date=all+dates"],"selectors":[{"id":"listing-link","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.result-row:nth-of-type(1) a.result-title","multiple":true,"delay":0},{"id":"replybutton","type":"SelectorElementClick","parentSelectors":["listing-link"],"selector":"aside.reply-flap","multiple":false,"delay":"3000","clickElementSelector":"button.reply_button","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"phonenumber","type":"SelectorText","parentSelectors":["replybutton"],"selector":"p.reply-tel-number","multiple":false,"regex":"","delay":0},{"id":"email","type":"SelectorText","parentSelectors":["replybutton"],"selector":"a.mailapp","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["listing-link"],"selector":"span.price","multiple":false,"regex":"","delay":0},{"id":"title","type":"SelectorText","parentSelectors":["listing-link"],"selector":"span.postingtitletext","multiple":false,"regex":"","delay":0},{"id":"street","type":"SelectorText","parentSelectors":["listing-link"],"selector":"div.mapaddress","multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","parentSelectors":["listing-link"],"selector":"section.userbody section","multiple":false,"regex":"","delay":0},{"id":"beds-baths","type":"SelectorText","parentSelectors":["listing-link"],"selector":"span.shared-line-bubble:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"type","type":"SelectorText","parentSelectors":["listing-link"],"selector":"p.attrgroup:nth-of-type(2) span","multiple":false,"regex":"","delay":0},{"id":"next","type":"SelectorElementClick","parentSelectors":["listing-link"],"selector":"a.next","multiple":false,"delay":0,"clickElementSelector":"a.next","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}
This seems to work, but where I'm stuck is having it either navigate to main listing page where the listing-links are, OR click "next" within each listing link so it navigates to the next listing .. make sense?
Hi all,
New here. I was able to get this to work with the following. Perhaps someone can benefit from this? One issue though, it stops after 4 items. Not sure why that is.
{"_id":"cl-losangeles","startUrl":["https://losangeles.craigslist.org/search/ggg?bundleDuplicates=1&excats=108-2-2-2&is_paid=yes"],"selectors":[{"id":"listing-link","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.result-row:nth-of-type(1) a.result-title","multiple":true,"delay":0},{"id":"replybutton","type":"SelectorElementClick","parentSelectors":["listing-link"],"selector":"aside.reply-flap","multiple":false,"delay":"2000","clickElementSelector":"button.reply-button","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"emailaddress","type":"SelectorLink","parentSelectors":["replybutton"],"selector":"a.mailapp","multiple":false,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["listing-link"],"selector":"span.postingtitletext","multiple":false,"regex":"","delay":0},{"id":"compensation","type":"SelectorText","parentSelectors":["listing-link"],"selector":".attrgroup span b","multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","parentSelectors":["listing-link"],"selector":"section.userbody section","multiple":false,"regex":"","delay":0},{"id":"location-lat","type":"SelectorElementAttribute","parentSelectors":["listing-link"],"selector":"div.viewposting","multiple":false,"extractAttribute":"data-latitude","delay":0},{"id":"location-lon","type":"SelectorElementAttribute","parentSelectors":["listing-link"],"selector":"div.viewposting","multiple":false,"extractAttribute":"data-longitude","delay":0},{"id":"next","type":"SelectorElementClick","parentSelectors":["listing-link"],"selector":"a.next","multiple":false,"delay":0,"clickElementSelector":"a.next","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"}]}
I am having the same problem here.
When I get to the page that has the Reply button on it. I create a Element_Click. 2 selectors show up. Is the first one the Reply button and the second is the Email Text located at the bottom of the little pop up box appears. Or iis Selector 2 the Reply button and Selector is the email?
Is the Parent the link that brought you to the second page(in my case "business")? That is the link that came from the element from page one to send an email.
This'll work for individual pages. On my side, there seems to be a long pause after clicking on Reply, so I set Click delay = 5000. modify as needed:
{"_id":"forum-craigslist","startUrl":["https://losangeles.craigslist.org/wst/evg/d/los-angeles-pa-beverly-hills-asap/7172117176.html"],"selectors":[{"id":"Click reply","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.reply-button-row > div > div.reply-info.js-only > aside","multiple":false,"delay":"5000","clickElementSelector":"div.actions-combo > button.reply-button","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueText"},{"id":"email","type":"SelectorText","parentSelectors":["Click reply"],"selector":"p.reply-email-address","multiple":false,"regex":"","delay":0}]}