Multiple Options for one Product that changes some information

#1

Hi

Trying to scape this site https://www.specialized.com/za/en/s-works-tarmac-sl7-frameset/p/199470?color=321701-199470

I can get all the products titles and descriptions as they are the same, but each product has a few colour options and size options. You need to select both these to get the product code(this is important information for me ). Both the colour options and the size option are selectable buttons that also change a text box showing what was selected.

This is complicated and I cant figure out how to get all iterations of the products.

Any help will be appreciated

#2

use Element Click on both: color (as a parent) and size (as child) selector... And scrape changed data... easy

#3

Thanks for the Reply. I am new to this so just bare with me:

{"_id":"Specialized_Bikes","startUrl":["https://www.specialized.com/za/en/c/bikes"],"selectors":[{"id":"page","paginationType":"auto","parentSelectors":["_root","page"],"selector":"button.sc-5893c326-3","type":"SelectorPagination"},{"id":"Link","linkType":"linkFromHref","multiple":true,"parentSelectors":["page"],"selector":"a.sc-1592a746-7","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"h1.sc-78b3ddfd-20","type":"SelectorText"},{"id":"Body","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".sc-e637be74-0 > div","type":"SelectorHTML"},{"clickActionType":"real","clickElementSelector":"button.sc-36e2ca2d-0","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"Size","multiple":true,"parentSelectors":["Link"],"selector":"div.sc-34e81a17-12","type":"SelectorElementClick"},{"clickActionType":"real","clickElementSelector":"button.sc-98b59d5f-0","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"Colour","multiple":true,"parentSelectors":["Size"],"selector":"_parent_","type":"SelectorElementClick"},{"id":"Colour_Text","multiple":false,"parentSelectors":["Colour"],"regex":"","selector":"p.sc-78b3ddfd-10","type":"SelectorText"},{"id":"SKU","multiple":false,"parentSelectors":["Colour"],"regex":"","selector":".sc-78b3ddfd-4 span","type":"SelectorText"},{"id":"Size_text","multiple":false,"parentSelectors":["Colour"],"regex":"","selector":"div.sc-75df178d-6","type":"SelectorText"}]}

This is where I got to, Made Size the parent and colour the child, but it only iterates through the size and not the colours

#4

Don't have too much free time now... will check later...

#5

Hi,

I would recommend setting up the same order as on the website (color first).

Also, when doing pagination, be sure to target only the 'next' button.

Please check the selectors, I have made some adjustments for a smooth performance:

{"_id":"Specialized_Bikes2","startUrl":["https://www.specialized.com/za/en/c/bikes"],"selectors":[{"id":"page","paginationType":"auto","parentSelectors":["_root","page"],"selector":"[aria-label=\"Next Page\"]","type":"SelectorPagination"},{"id":"Link","linkType":"linkFromHref","multiple":true,"parentSelectors":["page"],"selector":"a.sc-1592a746-7","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"h1.sc-78b3ddfd-20","type":"SelectorText"},{"id":"Body","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".sc-e637be74-0 > div","type":"SelectorHTML"},{"clickActionType":"real","clickElementSelector":"button.sc-36e2ca2d-0","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"discard-when-click-element-exists","id":"Size","multiple":true,"parentSelectors":["Colour"],"selector":"_parent_","type":"SelectorElementClick"},{"clickActionType":"real","clickElementSelector":"button.sc-98b59d5f-0","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"discard-when-click-element-exists","id":"Colour","multiple":true,"parentSelectors":["Link"],"selector":"_parent_","type":"SelectorElementClick"},{"id":"Colour_Text","multiple":false,"parentSelectors":["Size"],"regex":"","selector":"p.sc-78b3ddfd-10","type":"SelectorText"},{"id":"SKU","multiple":false,"parentSelectors":["Size"],"regex":"","selector":".sc-78b3ddfd-4 span","type":"SelectorText"},{"id":"Size_text","multiple":false,"parentSelectors":["Size"],"regex":"","selector":"div.sc-75df178d-6","type":"SelectorText"}]}
#6

thanks, I got it working somehow, but I will you yours as its probably more efficient.

#7

here is more universal sitemap with correct selectors:

{"_id":"Specialized_Bikes2","startUrl":["https://www.specialized.com/za/en/c/bikes"],"selectors":[{"id":"Link","linkType":"linkFromHref","multiple":true,"parentSelectors":["page"],"selector":"article a:has(h3)","type":"SelectorLink"},{"id":"page","paginationType":"auto","parentSelectors":["_root","page"],"selector":"[aria-label=\"Next Page\"]","type":"SelectorPagination"},{"id":"Title","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"tech","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"div[data-component=\"technical-specifications-accordion\"]","type":"SelectorHTML"},{"clickActionType":"real","clickElementSelector":"div[data-component=\"size-selection\"] button","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":100,"discardInitialElements":"discard-when-click-element-exists","id":"Size","multiple":true,"parentSelectors":["Colour"],"selector":"_parent_","type":"SelectorElementClick"},{"clickActionType":"real","clickElementSelector":"div[data-component=\"color-selection\"] button","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":100,"discardInitialElements":"discard-when-click-element-exists","id":"Colour","multiple":true,"parentSelectors":["Link"],"selector":"_parent_","type":"SelectorElementClick"},{"id":"Colour_Text","multiple":false,"parentSelectors":["Size"],"regex":"","selector":"div[data-component=\"color-selection\"] p:last-of-type","type":"SelectorText"},{"id":"SKU","multiple":false,"parentSelectors":["Size"],"regex":"(?<=:\\s)[\\d-]+","selector":"span:contains(Part No.:)","type":"SelectorText"},{"id":"Size_text","multiple":false,"parentSelectors":["Size"],"regex":"(?<=>\\s)[\\w\\d]+","selector":"span:contains(\"Size:\")","type":"SelectorHTML"}]}

But I would better use page numbers to walk through the bikes collection instead of using pagination.... You can exactly choose a range of pages you want to scrape.... got it?

#8

yes thanks very much