Drop down issue

How to?
#1

Cant figure out what im doing wrong, trying to go through drop down menus and then scape titles. First dropdown option selected, then it stops. what im doing wrong?

Url: https://www.plantasur.com/en/products/air-purification-systems?items_per_page=100

Sitemap:
{"_id":"testdropdown","startUrl":["https://www.plantasur.com/en/products/air-purification-systems?items_per_page=100"],"selectors":[{"id":"dropdown","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#content-area","multiple":true,"delay":"4000","clickElementSelector":"div > div[class^='form-item'] > select#edit-jump--4 > option:nth-child(n+2)","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"title","type":"SelectorText","parentSelectors":["dropdown"],"selector":"h2 a","multiple":false,"regex":"","delay":0}]}

#2

any thoughts on this would be much appreciated

#3

This site changes URLs on the server side so WS will have trouble navigating it.

It is still possible to scrape this site in two stages, where in stage 1, you get all the subfamily (subcategory) URLs from the main page, then in stage 2 you have a different sitemap which uses all those stage 1 URLs as Starturls:

{"_id":"forum-plantasur-get-sub","startUrl":["https://www.plantasur.com/en/products/air-purification-systems?items_per_page=100&field_familia_categoria_tid=13626"],"selectors":[{"id":"Sub-families","type":"SelectorElement","parentSelectors":["_root"],"selector":"form > div:first-of-type div.container-inline:contains('Subfamily') select > option","multiple":true },{"id":"Name","type":"SelectorText","parentSelectors":["Sub-families"],"selector":"_parent_","multiple":false,"regex":"" },{"id":"URL","type":"SelectorElementAttribute","parentSelectors":["Sub-families"],"selector":"_parent_","multiple":false,"extractAttribute":"value" }]}

1 Like