Element Click Selector Only Scrapes Some Data

How to?
#1

I have an element click selector that pulls the first nine pieces of text from a drop down list, but not the remaining 10+.

On the Copart Vehicle finder site, I have my element click select on the "Year" drop down, then I select all years in the drop down. I have a child text selector after my element selector.

In the data preview, I see data for all the years, but in the excel output I only get data through 2009. Can someone suggest a fix for having the scrape pull all the years? Thank you!

Url: https://www.copart.com/search/automobiles/?displayStr=Automobiles&from=%2FvehicleFinder

Sitemap: {"_id":"copartyears","startUrl":["https://www.copart.com/vehicleFinder/"],"selectors":[{"id":"autos","type":"SelectorLink","selector":"div.panel:nth-of-type(3) ul.list-group:nth-of-type(2) a","parentSelectors":["_root"],"multiple":false,"delay":0},{"id":"year","type":"SelectorElementClick","selector":"div#collapseinside4.panel-collapse label","parentSelectors":["autos"],"multiple":true,"delay":"2000","clickElementSelector":"li.list-group-item:nth-of-type(4) h4.panel-title a:nth-of-type(1)","clickType":"clickOnce","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"},{"id":"yeartext","type":"SelectorText","selector":"parent","parentSelectors":["year"],"multiple":true,"regex":"","delay":0}]}

#2

Change your click selector to this - div#collapseinside4.panel-collapse label input[id*="lot_year"]:not([checked="checked"]), and selector to this - div.dataTables_wrapper. Change "click element uniqueness" to Unique HTML+Text and your element click selector should work fine.

#3

Hey KristapsWS,

I don't think I was clear enough, but i'm SO close to finishing the project. Could you help me just get over the finish line?

When I use "data preview" for the Sitemap below it shows data for all the labels from 2018 to 1926. But when I actually scrape the data, it stops showing the labels after 2009.

I would love the scrape to pull labels beyond 2009, but it just won't pull. Any ideas to get me over the hump?

{"_id":"copartyears","startUrl":["https://www.copart.com/vehicleFinder/"],"selectors":[{"id":"autos","type":"SelectorLink","selector":"div.panel:nth-of-type(3) ul.list-group:nth-of-type(2) a","parentSelectors":["_root"],"multiple":false,"delay":0},{"id":"year","type":"SelectorElementClick","selector":"div#collapseinside4.panel-collapse label","parentSelectors":["autos"],"multiple":true,"delay":"2000","clickElementSelector":"li.list-group-item:nth-of-type(4) h4.panel-title a:nth-of-type(1)","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"yeartext","type":"SelectorText","selector":"parent","parentSelectors":["year"],"multiple":true,"regex":"","delay":0}]}

Data Preview shows labels for years past 2008

image
image.png1907×336 4.49 KB

But the excel scrape stops at 2009
image

#4

You need to interact with those elements to make them render. You don't have to click on "Year" to access them. Configure your element click selector like this:

Selector: div#collapseinside4.panel-collapse div.checkbox

Click selector: div#collapseinside4.panel-collapse label input[id*="lot_year"]:not([checked="checked"])

Click element uniqueness: Unique HTML+Text

Multiple: Checked

Delay: 2000ms

Change your "yeartext" selector to label.

#5

Thanks Kristaps! That worked. I really appreciate the help.

Andrew