Im trying to scrape craigslist posts and take advantage of a neat feature on CL that maintains the search parameters when you click through the location dropdown. I've had partial success and I cant figure out what I'm doing wrong. What I find is that it scrapes the NJ site 100% and navigates to some of the dropdown links, but not all, and the links it does follow it does not scrape completely. I've been playing with this for a while, but I'm stuck. This is the code that has brought me closest to my goal:
{"_id":"jerseytacoma","startUrl":["https://newjersey.craigslist.org/search/cta?auto_make_model=tacoma"],"selectors":[{"id":"product links","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.result-title","multiple":true,"delay":"200"},{"id":"Year","type":"SelectorText","parentSelectors":["product links"],"selector":"span#titletextonly","multiple":false,"regex":"\d{4}","delay":0},{"id":"odometer","type":"SelectorText","parentSelectors":["product links"],"selector":"section.userbody","multiple":false,"regex":"(?<=odometer.)(\d+)","delay":0},{"id":"trans","type":"SelectorText","parentSelectors":["product links"],"selector":"section.userbody","multiple":false,"regex":"(?<=transmission.)(\w+)","delay":0},{"id":"Cost","type":"SelectorText","parentSelectors":["product links"],"selector":"span.price","multiple":false,"regex":"","delay":0},{"id":"TRD Sport","type":"SelectorText","parentSelectors":["product links"],"selector":"section.userbody","multiple":false,"regex":"TRD","delay":0},{"id":"DropdownLink","type":"SelectorElement","parentSelectors":["_root"],"selector":"li.crumb.area select.js-only","multiple":false,"delay":0},{"id":"returnLink","type":"SelectorLink","parentSelectors":["DropdownLink","returnLink"],"selector":"parent","multiple":true,"delay":0}]}
Edit:
I inconsiderately forgot to ask you kind people out there to help me with suggestions or help with choosing the right selectors. I've tried the element click and all kinds of combinations of the element and link selectors. Anywhoo, thanks for any help yall.
John