Hi Everyone -
I know this has been brought up in the past but none of those solutions work. I am trying to scrape the followers here (https://public.tableau.com/app/profile/adam.e.mccann)
Scrolling down will 'load more elements' and at some point the scrolling ends with a click more button and then more scrolling to load elements and back to the beginning it goes.
I have all the selectors work but can't quite sort out how to keep it going.
Heya Bret, if it is just a one-off scrape, you can just set a long enough Page load delay, say 90000 (1.5 minutes) then manually scroll and click until you have enough items. You can use the "zoom out first so page is really tiny" trick which forces website to load more items for every scroll.
Alternatively, you can have independent scrollers and clickers for this site, i.e. your data scrapers do not need to be child of your scrollers.
{"_id":"tableu-test-manua-scrolll","startUrl":["https://public.tableau.com/app/profile/adam.e.mccann"],"selectors":[{"id":"Item wrappers","type":"SelectorElement","parentSelectors":["_root"],"selector":"div[class^='WorkbookList_workbookGridItem']","multiple":true},{"id":"Title n Link","type":"SelectorLink","parentSelectors":["Item wrappers"],"selector":"a[class^='VizCard_title']","multiple":false}]}
Thanks, I'll give the second option a try. I was planning on having this map crawl through followers of followers and that would require the long delay trick multiple time throughout this scrape.