Cannot select drop down menu items from JavaScript / ajax website

How to?
#1

I want to scrape the following data for all countries in the drop down list: name, address, type, website, info, certification 1, certification 2.

My sitemap doesn't seem to work. Can someone help me correct it?

Url: https://www.brewersassociation.org/directories/breweries/

Sitemap:
{"_id":"brewery","startUrl":["https://www.brewersassociation.org/directories/breweries/"],"selectors":[{"id":"item","type":"SelectorElement","selector":"div.brewery:nth-of-type(n+2)","parentSelectors":["country"],"multiple":true,"delay":0},{"id":"name","type":"SelectorText","selector":"li.name","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","selector":"ul.vcard li:nth-of-type(3)","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"type","type":"SelectorText","selector":"li.brewery_type:nth-of-type(5) a","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorText","selector":"li.url:nth-of-type(6) a","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"info","type":"SelectorText","selector":"li.ownership:nth-of-type(7) em","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"Certification 1","type":"SelectorText","selector":"img.ba","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"Certification 2","type":"SelectorText","selector":"a img.craft-seal","parentSelectors":["item"],"multiple":false,"regex":"","delay":0},{"id":"Select a country","type":"SelectorLink","selector":"li#country.dropdown a.dropdown-toggle","parentSelectors":["_root"],"multiple":false,"delay":"2000"},{"id":"country","type":"SelectorLink","selector":"li.dropdown.open li a","parentSelectors":["Select a country"],"multiple":true,"delay":"2000"}]}

Scrape data that a Dropdown Box displays
#2

Hi Joel,

i'd rather set an drop-down list to 'Show All' and then scrape the results using Show preview button on a element selector, it's less time-consuming and easier to do.

I've made a sitemap for you, it's a little tricky, as click selector has two selectors inside it, separated by a comma. First opens the drop-down list, and second opens city, after that, all results that show up being fully scraped.

{"_id":"brewersassociation_dropdown","startUrl":["https://www.brewersassociation.org/directories/breweries/"],"selectors":[{"id":"click_menu","type":"SelectorElementClick","selector":"article.page","parentSelectors":["_root"],"multiple":true,"delay":"5000","clickElementSelector":"li#state.dropdown a.dropdown-toggle, li.dropdown.open li:nth-of-type(5) a","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"info_combined","type":"SelectorElement","selector":"div.brewery:nth-of-type(n+2)","parentSelectors":["click_menu"],"multiple":true,"delay":"5000"},{"id":"ttl","type":"SelectorText","selector":"li.name","parentSelectors":["info_combined"],"multiple":false,"regex":"","delay":0},{"id":"str","type":"SelectorText","selector":"li.address","parentSelectors":["info_combined"],"multiple":false,"regex":"","delay":0},{"id":"phn","type":"SelectorText","selector":"li.telephone","parentSelectors":["info_combined"],"multiple":false,"regex":"","delay":0}]}

You can modify it to press 'Show All' (selector is called li#state.dropdown a.dropdown-toggle, li.dropdown.open li:nth-of-type(#), where # is a number of item in a drop-down list. Show All is number 1.
So correct selector for it would be li#state.dropdown a.dropdown-toggle, li.dropdown.open li:nth-of-type(1).
Since total amount of records is more than 8K, i would recommend you to increase delay up to minute for click selector, loading all results took me about 10 seconds though.

1 Like
#3

Thanks a lot iconoclast, your sitemap works perfectly.

#4