New to Web Scraper: Struggling With Drop Downs

Hi everyone,

I am new to web scraping and have recently picked it up to help speed up some processes for a business I am working for.

For this task, I need to load the web page, select the "City" radio button, open the "City list" drop down, select a city, click submit, then retrieve the total number of businesses. I thought this would be straightforward, however, I do not know how to iterate through the drop down list while ensuring I do not get duplicate data.

Apologies if this is a stupid question and I appreciate any help.

Url: http://www.hoosierdata.in.gov/buslookup/BusLookup.aspx

Sitemap:
{"_id":"businessind","startUrl":["http://www.hoosierdata.in.gov/buslookup/BusLookup.aspx"],"selectors":[{"id":"citybutton","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"input[value='3']","multiple":false,"delay":0,"clickElementSelector":"input[value='3']","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"citylist","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"select#geo_area","multiple":true,"delay":0,"clickElementSelector":"select#geo_area","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"submit","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"input#ContentPlaceHolder1_submit","multiple":false,"delay":0,"clickElementSelector":"input#ContentPlaceHolder1_submit","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"}]}

This site uses older tech (aspx) so WS probably can't navigate it. However, the URL does change for each city so you can still scrape it if you build a separate a list of city URLs. e.g. the URLs for Alamo, Atlanta, and Columbia City are:

http://www.hoosierdata.in.gov/buslookup/page2.aspx?scope=3&geo_area=Alamo&name_text=&company_size=Z&datacode=%

http://www.hoosierdata.in.gov/buslookup/page2.aspx?scope=3&geo_area=Atlanta&name_text=&company_size=Z&datacode=%

http://www.hoosierdata.in.gov/buslookup/page2.aspx?scope=3&geo_area=Columbia%20City&name_text=&company_size=Z&datacode=%

The URL pattern is the same, only the city name changes.

1 Like