Click, Load, Scrape

How to?
#1

The website I'm trying to scrape is public but requires a button click to load results. The results usually take between 10-15 seconds to load, and then a results table appears. I'm having trouble understanding how to click, load (wait), and then scrape. I was able to build a scraper once the results are loaded, but the click and load part escapes me. Any suggestions?

Url: https://prod-landrecords.browncountywi.gov/GCSWebPortal/Search.aspx

In my sitemap, I'm only including a simple text reader to read the total records after the search is done.

Sitemap:
{"_id":"brown-county-gcs","startUrl":["https://prod-landrecords.browncountywi.gov/GCSWebPortal/Search.aspx"],"selectors":[{"clickElementSelector":"input.submitBtn","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":15000,"discardInitialElements":"do-not-discard","id":"property-search-button","multiple":false,"parentSelectors":["_root"],"selector":"input.submitBtn","type":"SelectorElementClick"},{"delay":0,"id":"results-count","multiple":false,"parentSelectors":["_root","property-search-button"],"regex":"","selector":"span#resultCount","type":"SelectorText"}]}

#2

@pflugs30 Hello. You can try to add an additional selector below the 'search' click selector so the scraper would spend extra time in the page in the case the table has not loaded in time.

Example:

{"_id":"brown-county-gcs","startUrl":["https://prod-landrecords.browncountywi.gov/GCSWebPortal/Search.aspx"],"selectors":[{"clickElementSelector":"input[value=\"I Accept\"]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"property-search-button","multiple":true,"parentSelectors":["_root"],"selector":"html","type":"SelectorElementClick"},{"clickElementSelector":"input[value=\"Search For Properties\"]","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"search-click","multiple":true,"parentSelectors":["_root"],"selector":"html","type":"SelectorElementClick"},{"delay":10000,"id":"scroll-delay","multiple":true,"parentSelectors":["_root"],"selector":"html","type":"SelectorElementScroll"},{"delay":0,"id":"results-count","multiple":false,"parentSelectors":["_root"],"regex":"","selector":"span#resultCount","type":"SelectorText"}]}

1 Like