Scraping Data from a Popup and infinity scroll webpage, Please help :(

Hi everyone,

I've been trying to scrape a page (The Wing), but the webpage has a popup page that slides when you click on a members profile. I've been able to get the scraper to open the popup page for several members and scrape the data successfully, BUT there's an infinity page that you need to continue scrolling through to reveal more members. How would I go about doing this?

I've attached some photos to show what I'm doing. Any help would be BEYOND helpful!!

This process lets me scrape the data from the popup window.


I tried adding a scrolling selector to the root page before the element click selector but that didn't really work for me.
How do I combine both of these tools together so that the scraper will scroll and scrape the popup window simultaneously?

Thank you!

1 Like

Sitemap in the studio...

Here's the sitemap!

{"_id":"thewing-","startUrl":["https://witches.the-wing.com/community"],"selectors":[{"id":"select","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.eXrlVk","multiple":true,"delay":"2000","clickElementSelector":"div.Member__User-sc-1crs6o1-1","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Name","type":"SelectorText","parentSelectors":["select"],"selector":"span.jlLLOK","multiple":false,"regex":"","delay":0},{"id":"Job Function","type":"SelectorText","parentSelectors":["select"],"selector":"span.izECkQ","multiple":false,"regex":"","delay":0},{"id":"Industry","type":"SelectorText","parentSelectors":["select"],"selector":"div.iDTxna:nth-of-type(3) div.epdcYW","multiple":false,"regex":"","delay":0},{"id":"Email","type":"SelectorText","parentSelectors":["select"],"selector":".AdditionalInfo__Text-cw1bi3-3 a","multiple":false,"regex":"","delay":0},{"id":"Instagram","type":"SelectorLink","parentSelectors":["select"],"selector":"a.Social__SocialLink-sc-140echj-0","multiple":false,"delay":0}]}