I have been stuck on this for day... so unbelievably frustrated as I know its probably a simple fix but I can't find the right setting. This scraper is genius btw. pure brilliant genius.
Here is the situation:
I have a members page unfortunately behind a password so its not publicly available.
I can get the page to load more results. Easy to do with an event click.
I can get the results from the pop up now with another element click. (thanks to your help with another question.in the forum.)
Now, I am stuck. The load more will work and the element click will work but they won't work together.
When I put them together I can mixed results from various settings on the load more event click. One result was that I get null values. Another result was that if there were 20 people in the initial search, I would get 20 lines in the scraped data but they were ALL the first result over and over like it wouldn't click the second pop up link.
Here are a few of the sitemaps I am working on. This one is the popup link scrape working WITHOUT the load more on the front.
Here is the sitemap I was trying with the load more element click on the front:
{"_id":"startover","startUrl":["https://www.mpiweb.org/membership/member-directory"],"selectors":[{"id":"loadmoredata","type":"SelectorElementClick","selector":"div.col-md-4 div.row","parentSelectors":["_root"],"multiple":true,"delay":"10000","clickElementSelector":"div.row.margin-lg-bottom a.btn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"fullparentelement","type":"SelectorElementClick","selector":".member-card","parentSelectors":["loadmoredata"],"multiple":true,"delay":0,"clickElementSelector":"a.ng-binding","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"anothername","type":"SelectorText","selector":"h3.modal-title","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotherphone","type":"SelectorText","selector":"span.member-contact","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotheremail","type":"SelectorText","selector":"div.col-xs-7.ng-binding","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0}]}
Here is a screenshot of what the search results look like:
When you click on the name this is the pop up that comes up with the information:
Here is a screenshot of the structure of the search results:
Here is a screenshot of the pop up structure:
If there is anything else I need to provide, I can. It should be super simple .... and its probably something I am missing but I just don't understand why it won't load more and grab the pop up info as I am doing it ...
Thanks for your help, this would be a huge help!!!
Adam