How to select multiple dropdown options to search before scraping results

How to?
#1

Hi!

The search results I'm trying to scrape first require me to select from 3 separate dropdown lists before clicking 'Search'.

I want to scrape the results for a specific set of dropdown options, being:

Services provided via = Face-to-face
Area of Practice = Disability
Funding Scheme = NDIS

Note: There is a free text field to search for "OT's Name" but I want to leave this blank.

When the search results appear the URL remains generic "otaus.com.au/find-an-ot"... so I can't simply setup the scraper to start from the desired results page. Can someone please explain how I setup the scraper to select the desired options and search, before then starting the scrape?

Url: Occupational Therapy Australia - Find an Occupational Therapist

#2

@jamesio Hello, in order to select each of the targeted options a separate 'Element click' selector will have to be used.

Please, note that the results will start to appear only once the scraper has finished clicking through the pagination pages.

Learn more: My scraping job is running, although no results are being returned - Web Scraper Knowledge Base

Here's a test sitemap(limited to paginate through till the 5th page):

{"_id":"otaus-com-au","startUrl":["https://otaus.com.au/find-an-ot"],"selectors":[{"clickElementSelector":"body:not(:has(div.results__item)) select option:contains(\"Face-to-face\")","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"click-1","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"clickElementSelector":"body:not(:has(div.results__item)) select option:contains(\"Disability\"):not(:contains(\"Int\"))","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"click-2","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"clickElementSelector":"body:not(:has(div.results__item)) select option:contains(\"NDIS\"):not(:contains(\"Practice\"))","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"click-3","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"clickElementSelector":"body:not(:has(div.results__item)) button.button-blue","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":7000,"discardInitialElements":"do-not-discard","id":"search","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"clickElementSelector":".pagination li.active + li a:not(:contains(\"5\"))","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":8000,"discardInitialElements":"do-not-discard","id":"listing-click-more","multiple":true,"parentSelectors":["_root"],"selector":"div.results__item","type":"SelectorElementClick"},{"id":"mobile","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"a[target]","type":"SelectorText"},{"id":"email","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"span:contains(\"Email:\") + a","type":"SelectorText"},{"id":"web","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"span:contains(\"Web\") + a","type":"SelectorText"},{"id":"desc","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"div.content__col:nth-of-type(2) p","type":"SelectorText"},{"id":"address","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"p:nth-of-type(2)","type":"SelectorText"},{"id":"name","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"strong.name","type":"SelectorText"},{"id":"title","multiple":false,"parentSelectors":["listing-click-more"],"regex":"","selector":"div.title__tag","type":"SelectorText"}]}

If you are looking to click through more pages, just change the number within '.pagination li.active + li a:not(:contains("5"))' - listing-click-more selector.

Please, note that due to how slowly the next page results are rendered it can take up to 1-2 hours to finish the extraction process.

Learn more: Top 5 CSS Selectors You Need to Know.

#3

Thanks so much for this! Super helpful!