Multi URLs - same site / dummy URL

How to?
#1

The site has a dropdown menu which Web Scraper can click on the choice, but I can't execute the selection.

So instead I tried to scrape by adding the multi URL's. Because the URL's are similar Web Scraper doesn't run the different URL's.

To make this work I've inserted a dummy website and after the first URL scrapes, it closes and then runs the dummy (fails), then opens the second URL and scrapes.

Is there a method to get the multi URL's to execute without the dummy URL? Otherwise it won't work.

Better yet - nice if Web Scraper can select the item in the dropdown and run the code.

Thanks

{"_id":"testurls","startUrl":["https://www.URLx.com/area/f5d2c671/","http://www.test","https://www.URLx.com/area/22b94eaa/"],"selectors":[{"id":"cash","type":"SelectorText","parentSelectors":["_root"],"selector":"div.font__gmsanc-regular div div div div.row td.right:nth-of-type(4)","multiple":false,"regex":"","delay":"5000"},{"id":"name","type":"SelectorText","parentSelectors":["_root"],"selector":"div.col-md-8 select.bc-selector option:selected","multiple":false,"regex":"","delay":0}]}

#2

After trying this problem with the test-sites it works fine. I guess the problem is with the website I'm trying to scrape. It's a pay site and I can't share the link.

For some reason running multi URL's won't update on this site unless I run the dummy and then reload the next URL.

Will post a separate question about the drop down menu.

#3

I forgot to post the Sitemap that works:

{"_id":"testmulti","startUrl":["https://www.webscraper.io/test-sites/e-commerce/allinone","https://www.webscraper.io/test-sites/e-commerce/allinone/computers"],"selectors":[{"id":"select1","type":"SelectorText","parentSelectors":["_root"],"selector":"div.col-sm-4:nth-of-type(1) a.title","multiple":false,"regex":"","delay":0},{"id":"select2","type":"SelectorText","parentSelectors":["_root"],"selector":"li:nth-of-type(3) a.category-link","multiple":false,"regex":"","delay":0}]}

#4

Hi there!

You can use Browser Inspect Tool to narrow down the links/menu items you want to click. That's a little advanced to what you can do with WebScraper.
It all depends on a particular dropdown and your target - either get list of all items (that can be done without using anything but a Text selector), or switching through items, and so on.

image
image.png909×1019 71.2 KB

Try to run this one and analyze it:

{"_id":"w3_dropdown","startUrl":["https://www.w3.org/TR/2017/NOTE-wai-aria-practices-1.1-20171214/examples/listbox/listbox-collapsible.html"],"selectors":[{"id":"click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".focused","multiple":true,"delay":"","clickElementSelector":"button, [role=\"option\"]","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"item","type":"SelectorText","parentSelectors":["click"],"selector":"_parent_","multiple":false,"regex":"","delay":0}]}
#5

Thanks - very helpful. I'm able to select each item in the dropdown. I see it changing, but the page doesn't get updated. Your example works fine. It's from my paid site and I'm not sure what triggers the update when the dropdown item is selected.

I could share the code if that helps.