Scrape a page with js search required to get the search results

[hypnoman]

I have a membership site that has a directory of members. To get the members to show up, a search is mandatory. The search is done with javascript so there is no url change to link to.

The issue is the mandatory search. I can manually search but webscraper loads a new page that has the search results reset requiring another manual search.

Is there any way to pause webscraper so I can manually search and then start webscraper back up again?

URL is pointless because it is behind a password anyways.

Sitemap isn't part of the question and can be built easily once it can access the search.

Thank you in advance for any help !!!!

Cheers,

[iconoclast]

Hi!

Please refer to this post, maybe it can be helpful:

Scraping data from password protected sites

It all depends on a particular website. You can use autofill data (login+password being automatically filled), and a click selector to enter a website, then you can build rest of your selectors. Most of the websites that require a login and password will actually create a cookie so you won't need to login each time you enter it.

Meanwhile, a new version of WebScraper will possibly have this feature (handling search form) in near future.

[hypnoman]

what I did was set the url to the search page of the website and set the first item to a load more click element with a delay of 10 seconds. The 10 seconds gave enough time for the page to load and for me to manually type in a search term so the page would load with the search results before the webscraper started again.

Very much a hack workaround and it isn't perfect but it worked for me.

Thank you for the reply!!