Help scraping auto loading page

How to?
#1

Describe the problem.

Hey there,

So everything is working great, but I can't figure out the autoloader. I tried to research some solutions on this platform, and I've found several people who had similar problems that were resolved. But when I attempt their solutions, no dice, doesn't work. So I'm hoping I can have my personal issue resolved. Please advise, thanks.

How can I design the scraper to scrape autoloader/infinite loader?

I'm trying to scrape this page

https://www.meetup.com/find/events/?allMeetups=true&radius=Infinity&userFreeform=Washington%2C+District+of+Columbia%2C+USA&mcId=c20001&change=yes&eventFilter=all

Sitemap:
{"_id":"meetup","startUrl":["https://www.meetup.com/find/events/?allMeetups=true&radius=100&userFreeform=washington&mcId=c20001&change=yes&eventFilter=all"],"selectors":[{"id":"event_link","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.resetLink.big","multiple":true,"delay":0},{"id":"event_host","type":"SelectorText","parentSelectors":["event_link"],"selector":"span span.text--secondary","multiple":false,"regex":"","delay":0},{"id":"invite_only","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.infoToggle-label span","multiple":false,"regex":"","delay":0},{"id":"event_attendees","type":"SelectorText","parentSelectors":["event_link"],"selector":"p.eventStatusIndicator-status.text--secondary span","multiple":false,"regex":"","delay":0},{"id":"event_description","type":"SelectorText","parentSelectors":["event_link"],"selector":"div.flex-item > section.section:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"event_date","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.eventTimeDisplay-startDate > span:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"event_start_time","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.eventTimeDisplay-startDate-time span","multiple":false,"regex":"","delay":0},{"id":"event_end_time","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.eventTimeDisplay-endDate-partialTime span","multiple":false,"regex":"","delay":0},{"id":"event_repeat_information","type":"SelectorText","parentSelectors":["event_link"],"selector":"p.eventTimeDisplay-recurrence","multiple":false,"regex":"","delay":0},{"id":"event_location","type":"SelectorText","parentSelectors":["event_link"],"selector":"address p.wrap--singleLine--truncate","multiple":false,"regex":"","delay":0},{"id":"event_location_address","type":"SelectorText","parentSelectors":["event_link"],"selector":"p.venueDisplay-venue-address","multiple":false,"regex":"","delay":0},{"id":"event_location_map","type":"SelectorLink","parentSelectors":["event_link"],"selector":"a.venueMap-mapLink","multiple":false,"delay":0},{"id":"event_comments","type":"SelectorText","parentSelectors":["event_link"],"selector":"section.section:nth-of-type(3)","multiple":false,"regex":"","delay":0}]}

#2

This is an interesting one because the site uses two methods. You have an initial click "show more" followed by an infinite scroll.

I think I have it working but you've set your range so high that it keep loading more and more events. I think you might get tot he point where it's loaded so many events that it crashes your browser. If that happens, you'll need to narrow your search query a bit.

Here you go..

{"_id":"meetup-untech-video-fix","startUrl":["https://www.meetup.com/find/events/?allMeetups=true&radius=100&userFreeform=washington&mcId=c20001&change=yes&eventFilter=all"],"selectors":[{"id":"event_link","type":"SelectorLink","parentSelectors":["Infinite Load"],"selector":"a.resetLink.big","multiple":false,"delay":0},{"id":"event_host","type":"SelectorText","parentSelectors":["event_link"],"selector":"span span.text--secondary","multiple":false,"regex":"","delay":0},{"id":"invite_only","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.infoToggle-label span","multiple":false,"regex":"","delay":0},{"id":"event_attendees","type":"SelectorText","parentSelectors":["event_link"],"selector":"p.eventStatusIndicator-status.text--secondary span","multiple":false,"regex":"","delay":0},{"id":"event_description","type":"SelectorText","parentSelectors":["event_link"],"selector":"div.flex-item > section.section:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"event_date","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.eventTimeDisplay-startDate > span:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"event_start_time","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.eventTimeDisplay-startDate-time span","multiple":false,"regex":"","delay":0},{"id":"event_end_time","type":"SelectorText","parentSelectors":["event_link"],"selector":"span.eventTimeDisplay-endDate-partialTime span","multiple":false,"regex":"","delay":0},{"id":"event_repeat_information","type":"SelectorText","parentSelectors":["event_link"],"selector":"p.eventTimeDisplay-recurrence","multiple":false,"regex":"","delay":0},{"id":"event_location","type":"SelectorText","parentSelectors":["event_link"],"selector":"address p.wrap--singleLine--truncate","multiple":false,"regex":"","delay":0},{"id":"event_location_address","type":"SelectorText","parentSelectors":["event_link"],"selector":"p.venueDisplay-venue-address","multiple":false,"regex":"","delay":0},{"id":"event_location_map","type":"SelectorLink","parentSelectors":["event_link"],"selector":"a.venueMap-mapLink","multiple":false,"delay":0},{"id":"event_comments","type":"SelectorText","parentSelectors":["event_link"],"selector":"section.section:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"Infinite Load","type":"SelectorElementScroll","parentSelectors":["Show more"],"selector":"li.row","multiple":true,"delay":"3000"},{"id":"Show more","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.lt","multiple":false,"delay":"4000","clickElementSelector":"span.button:contains(\"Show\")","clickType":"clickOnce","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"}]}

Update It's now up to July 31st. and going and going..
*Update #2 - It's finally finished the scroll and has moved on to scraping each individual profile. I'm going to let it run overnight and post the data for you in the morning, if it doesn't overflow and crash.

1 Like