How to handle javascript link dropdown menus

You can see the there is a dropdown menu with links to product variations. But this linking mechanism does not use html's a tag or href attribute. I suspect Javascript is involved. How do I tell web scraper to follow these links?

https://www.nisbets.com.au/vogue-premium-stainless-steel-table-with-splashback-600dmm/p_da337

This will be harder 'cos clicking on the dropdown options cause the Url to change (probably server redirect). I don't think WS can follow those lnks.

You can actually grab the variation links without any clicking, but you'd need to prepend the main url to them, and scrape them separately:

{"_id":"nisbets-test","startUrl":["https://www.nisbets.com.au/vogue-premium-stainless-steel-table-with-splashback-600dmm/p_da337"],"selectors":[{"id":"Product name","type":"SelectorText","parentSelectors":["_root"],"selector":".breadcrumb .active span[property='name']","multiple":false,"regex":"","delay":0},{"id":"Variation wrappers","type":"SelectorElement","parentSelectors":["_root"],"selector":"div.variant-selector > select > option:not(\":contains('Please select to see options')\")","multiple":true,"delay":0},{"id":"Variation name","type":"SelectorText","parentSelectors":["Variation wrappers"],"selector":"_parent_","multiple":false,"regex":"","delay":0},{"id":"Variation link","type":"SelectorElementAttribute","parentSelectors":["Variation wrappers"],"selector":"_parent_","multiple":false,"extractAttribute":"value","delay":0}]}

1 Like