Element Click Drop Down List

Hi Guys...

I am trying to use element click to click into a drop down box and cycle through the variants inside the box.
Once it clicks one of the variants i want it to scrape some HTML/text from three different elements which change when clicking the variant inside the drop down box

I can somewhat get this working but not correctly enough. It skips data and doesn't scrape everything and is just really problematic.
I am currently using another webscraper which works fine but its slow and buggy and i really would like to use webscraper.io This issue is the only thing holding me back from using it fully.

Here is my current sitemap:
The same issue arises across different URL product listings - but with different results.
This URL works somewhat and clicks through the list but misses/skips data(pricing): https://www.ebay.com.au/itm/283340166980
This URL just scrapes the default variant already selected in the drop down list and doesn't click through the list: https://www.ebay.com.au/itm/232744877949

{"_id":"variations","startUrl":["https://www.ebay.com.au/itm/283340166980"],"selectors":[{"id":"getvariations","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"span[id="prcIsum"],option[selected="selected"],span[id="sel-msku-variation"]","multiple":true,"delay":0,"clickElementSelector":"select.msku-sel, #vi_main_img_fs li:nth-of-type(n+14), div#test12","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"names","type":"SelectorText","parentSelectors":["getvariations"],"selector":"parent","multiple":true,"regex":"","delay":0}]}

Please, Any help would be much appreciated
I have tried a multitude of things and still cannot get it working perfectly.
I have also seen alot of other people are having real issues with the Element Click Selector and so i am hoping it's not just me being dumb. I have really tried everything i can. I have tried different element uniqueness, delays, click type, different elements to select etc etc.. The above sitemap is the best i can get it. - It atleast somewhat works.

Regards,
Richard.

1 Like

I was able to figure out he pagination but not how to grab the name

perhaps @webber can help?

{"_id":"a-forum-ebay","startUrl":["https://www.ebay.com.au/itm/283340166980"],"selectors":[{"id":"Element Click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.lsp-c","multiple":true,"delay":0,"clickElementSelector":"select.msku-sel, #vi_main_img_fs li:nth-of-type(n+13)","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"Name","type":"SelectorText","parentSelectors":["Element Click"],"selector":".vi-msku-cntr+","multiple":false,"regex":"","delay":0},{"id":"Condition","type":"SelectorText","parentSelectors":["Element Click"],"selector":"div[itemprop='itemCondition']","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["Element Click"],"selector":"span[itemprop='price']","multiple":false,"regex":"","delay":0}]}

Try this one:

{"_id":"ebay-variation","startUrl":["https://www.ebay.com.au/itm/283340166980"],"selectors":[{"id":"element-click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#CenterPanelDF","multiple":true,"delay":"1000","clickElementSelector":"select.msku-sel option:not(:contains('-Select-'))","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"name","type":"SelectorText","parentSelectors":["element-click"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["element-click"],"selector":"span[itemprop='price']","multiple":false,"regex":"","delay":0},{"id":"condition","type":"SelectorText","parentSelectors":["element-click"],"selector":"div[itemprop='itemCondition']","multiple":false,"regex":"","delay":0},{"id":"variation","type":"SelectorText","parentSelectors":["element-click"],"selector":"select.msku-sel option[selected]","multiple":false,"regex":"","delay":0},{"id":"variation-sku","type":"SelectorText","parentSelectors":["element-click"],"selector":"#sel-msku-variation","multiple":false,"regex":"","delay":0}]}

That worked much better then mine. Where did you come up with

  • "select.msku-sel option:not(:contains('-Select-'))
  • select.msku-sel option[selected]

I couldn't quite figure those out,

Both of those are elements that can be found in the drop down menu.

The first one is just to exclude an 'empty' click so there is no duplicate records produced with a value 'Select'

As for the second one - as you iterate through the options, a 'selected' element is added to the option that is active at that time, so that you can extract the correct option that aligns with the rest of the data.

HI Webber,

Thankyou very much for your help. It works quite well.
The only trouble i am having is the scraping is very slow.
Chrome just seems to sit there and wait.
On the bottom of the page it says "establishing secure connection" sometimes takes a good 30 seconds to a minute before it actually scrapes the site and clicks the drop down list and then moves on... only to wait establishing secure connection again.

The first few pages are quick but then comes to a halt and just waits. But this does not happen if i browse the same site myself.

Any thoughts?

Wait... Sorry guys. I figured it out.

I checked the requests on each page loading and found that on some pages, external links where loading and taking a long time to load before the scraper could do its thing.

I just used the Dev tools in Chrome to watch the network requests and then watched for the long ones and blocked those domains. Very very speedy now!

This is working excellently but i've run into trouble with double options Similar to this URL

The selector for the second option is msku-sel-2. i can pick up the options, however this sitemap cycles through all items within the first option and then the second option. So it reads it wrong and i can't quite get it to work correctly. i always get so close.

I tried element click for the first option and then it should cycle through all the second options but again, i cant get it to work.

Would you know how to complete this with 2 options?

Regards.

There are definitely 'cleaner' ways to do this, but this will work as well:

{"_id":"ebay-variation-two","startUrl":["https://www.ebay.com.au/itm/283340166980"],"selectors":[{"id":"element-click-1","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#CenterPanelDF","multiple":true,"delay":"1000","clickElementSelector":".nonActPanel div > div:has(select):nth(0) select.msku-sel option:not(:contains('-Select-'))","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"name","type":"SelectorText","parentSelectors":["click-2"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["click-2"],"selector":"span[itemprop='price']","multiple":false,"regex":"","delay":0},{"id":"condition","type":"SelectorText","parentSelectors":["click-2"],"selector":"div[itemprop='itemCondition']","multiple":false,"regex":"","delay":0},{"id":"variation-sku","type":"SelectorText","parentSelectors":["click-2"],"selector":"#sel-msku-variation","multiple":false,"regex":"","delay":0},{"id":"click-2","type":"SelectorElementClick","parentSelectors":["element-click-1"],"selector":"_parent_","multiple":true,"delay":"1000","clickElementSelector":".nonActPanel div > div:has(select):nth(1) select.msku-sel option:not(:contains('-Select-'))","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"variation-oprion-1","type":"SelectorText","parentSelectors":["click-2"],"selector":".nonActPanel div > div:has(select):nth(0) select.msku-sel option[selected]","multiple":false,"regex":"","delay":0},{"id":"variation-option-2","type":"SelectorText","parentSelectors":["click-2"],"selector":".nonActPanel div > div:has(select):nth(1) select.msku-sel option[selected]","multiple":false,"regex":"","delay":0}]}

Thankyou very much for your input webber. You're a champion and i appreciate your help

There was a small issue with your sitemap where it would not scrap all the data..
It would keep the second option as the last item in the list before it would move the first option to the next item.

For example, lets say the first option is 1M which has Blue Red Grey and the second option 2M would have say only Blue and Red but not Grey

What this does is it would cycle through the first list, get 1M, then cycles the second list and stops at the last item, "Grey" and then cycles the first option again but because it was still set to Grey, some options in the first list gets missed.

Because the second option was still set to Grey, it would go and change the first option but it would skip over 2M because 2M never had Grey.
From here it just gets worse. If the next available item 3M only had Red as the last option. It stays on Red and cycles through. And if no other option in the first list (1M,2M,3M,4M,5M etc...) had Red, Then it stops right there and finishes the scrape.

What i did was add an extra click where it would select the "-Select-" option and reset all variations in the second option before it cycled the first option again.

I dont know if it is the best way of doing it, I dont even know if this is efficient to do it this way or whether it wastes time clicking the "-Select-" option between all options but it works.

{"_id":"webber2","startUrl":["https://www.ebay.com.au/itm/283340166980"],"selectors":[{"id":"element-click-1","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#CenterPanelDF","multiple":true,"delay":"0","clickElementSelector":".nonActPanel div > div:has(select):nth(0) select.msku-sel option:not(:contains('-Select-'))","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"price","type":"SelectorText","parentSelectors":["click-2"],"selector":"span[itemprop='price']","multiple":false,"regex":"","delay":0},{"id":"variation-sku","type":"SelectorText","parentSelectors":["click-2"],"selector":"#sel-msku-variation","multiple":false,"regex":"","delay":0},{"id":"click-2","type":"SelectorElementClick","parentSelectors":["element-click-1"],"selector":"parent","multiple":true,"delay":"0","clickElementSelector":".nonActPanel div > div:has(select):nth(1) select.msku-sel option:not(:contains('-Select-'))","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"variation-oprion-1","type":"SelectorText","parentSelectors":["click-2"],"selector":".nonActPanel div > div:has(select):nth(0) select.msku-sel option[selected]","multiple":false,"regex":"","delay":0},{"id":"variation-option-2","type":"SelectorText","parentSelectors":["click-2"],"selector":".nonActPanel div > div:has(select):nth(1) select.msku-sel option[selected]","multiple":false,"regex":"","delay":0},{"id":"Click Select","type":"SelectorElementClick","parentSelectors":["click-2"],"selector":".nonActPanel div > div:has(select):nth(1) select.msku-sel option:contains('-Select-')","multiple":false,"delay":0,"clickElementSelector":".nonActPanel div > div:has(select):nth(1) select.msku-sel option:contains('-Select-')","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"}]}