Load more and pop up not working togther

I have been stuck on this for day... so unbelievably frustrated as I know its probably a simple fix but I can't find the right setting. This scraper is genius btw. pure brilliant genius.

Here is the situation:

I have a members page unfortunately behind a password so its not publicly available.

I can get the page to load more results. Easy to do with an event click.

I can get the results from the pop up now with another element click. (thanks to your help with another question.in the forum.)

Now, I am stuck. The load more will work and the element click will work but they won't work together.

When I put them together I can mixed results from various settings on the load more event click. One result was that I get null values. Another result was that if there were 20 people in the initial search, I would get 20 lines in the scraped data but they were ALL the first result over and over like it wouldn't click the second pop up link.

Here are a few of the sitemaps I am working on. This one is the popup link scrape working WITHOUT the load more on the front.

{"_id":"startover2","startUrl":["https://www.mpiweb.org/membership/member-directory"],"selectors":[{"id":"fullparentelement","type":"SelectorElementClick","selector":".modal-content","parentSelectors":["_root"],"multiple":true,"delay":"8000","clickElementSelector":"a.ng-binding","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"anothername","type":"SelectorText","selector":"h3.modal-title","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotherphone","type":"SelectorText","selector":"span.member-contact","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotheremail","type":"SelectorText","selector":"div.col-xs-7.ng-binding","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0}]}

Here is the sitemap I was trying with the load more element click on the front:

{"_id":"startover","startUrl":["https://www.mpiweb.org/membership/member-directory"],"selectors":[{"id":"loadmoredata","type":"SelectorElementClick","selector":"div.col-md-4 div.row","parentSelectors":["_root"],"multiple":true,"delay":"10000","clickElementSelector":"div.row.margin-lg-bottom a.btn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"fullparentelement","type":"SelectorElementClick","selector":".member-card","parentSelectors":["loadmoredata"],"multiple":true,"delay":0,"clickElementSelector":"a.ng-binding","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"anothername","type":"SelectorText","selector":"h3.modal-title","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotherphone","type":"SelectorText","selector":"span.member-contact","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotheremail","type":"SelectorText","selector":"div.col-xs-7.ng-binding","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0}]}

Here is a screenshot of what the search results look like:

When you click on the name this is the pop up that comes up with the information:

Here is a screenshot of the structure of the search results:

Here is a screenshot of the pop up structure:

If there is anything else I need to provide, I can. It should be super simple .... and its probably something I am missing but I just don't understand why it won't load more and grab the pop up info as I am doing it ...

Thanks for your help, this would be a huge help!!!

Adam

Hi Adam!

You're getting same rows because you selected wrong wrapper for element click, try selecting what you're trying to scrape from, then hit 'P' to select a parent object, or edit selector inside WebScraper.

You should also note that a selector for pop-up window must contain whole pop-up for better result, as text selectors inside it rely on it as on parent wrapper. I think you already know that.

1 Like

Thank you !

I made some adjustments and changed the parent wrapper for the click selector for the load more. I think that's what you were talking about and unbelievably it worked. I have posted my working sitemap below for others in case it helps.

Again, many many thanks for the help and webscraper!!!

Adam

{"_id":"oooo","startUrl":["https://www.mpiweb.org/membership/member-directory"],"selectors":[{"id":"loadalldata","type":"SelectorElementClick","selector":"div.col-sm-10","parentSelectors":["_root"],"multiple":true,"delay":"8000","clickElementSelector":"div.row.margin-lg-bottom a.btn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"fullparentelement","type":"SelectorElementClick","selector":".modal-content","parentSelectors":["_root"],"multiple":true,"delay":"8000","clickElementSelector":"a.ng-binding","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"anothername","type":"SelectorText","selector":"h3.modal-title","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotherphone","type":"SelectorText","selector":"span.member-contact","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"anotheremail","type":"SelectorText","selector":"div.col-xs-7.ng-binding","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"company","type":"SelectorText","selector":"div.member-role","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0},{"id":"city","type":"SelectorText","selector":"div.member-location span.ng-binding","parentSelectors":["fullparentelement"],"multiple":false,"regex":"","delay":0}]}