Unable to get pagination using element click to work

I'm trying to get the pagination to work on the below site...

The pagination works for the first two pages. I set the pagination as the parent to the root page, so it scrapes all elements on the page, moves to the second, and scrapes all elements on the second page. However, when I set the pagination to a parent to itself to create the loop and hopefully scrape all pages, nothing happens and no data is scraped. Also, the pagination style on the site seems kind of unique so that might be causing the issue. Any ideas on how to solve this?

Sitemap:
{"_id":"food-drink-eventbrite","startUrl":["Chicago, IL Food & Drink Events | Eventbrite div.search-event-card","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["events"],"selector":"div.card-text--truncated__three","multiple":false,"regex":"","delay":0},{"id":"date/time","type":"SelectorText","parentSelectors":["events"],"selector":"div.eds-media-card-content__sub-content > div.eds-text-bs--fixed","multiple":false,"regex":"","delay":0},{"id":"venue","type":"SelectorText","parentSelectors":["events"],"selector":"div.card-text--truncated__one","multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root","pagination"],"selector":"a.eds-btn","multiple":false,"delay":"2000","clickElementSelector":"a.eds-btn","clickType":"clickOnce","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"}]}

Hi there!

You can save your time and use pagination array, your URL would look like:
https://www.eventbrite.com/d/il--chicago/food-and-drink--events/?page=[1-41]

Try it out, don't forget to remove your pagination selector beforehand.

And if you want it the 'right' way, you've selected not only 'next' button for your Element Click selector, but also 'previous' button as well. You don't need to set an Element Click selector as a child of itself, all you need to do is set it as 'Click More' click type, so it will continue pressing selected button until it disappears or becomes disabled.

A selector for an Element Click (the upper one inside) acts as an Element selector (you've had it in your sitemap), it's not necessary to add another element selector. What is important, is selecting proper wrapper to a selector.

Here's the sitemap with the clicker:

{"_id":"food-drink-eventbrite2","startUrl":["https://www.eventbrite.com/d/il--chicago/food-and-drink--events/?page=1&place_id=85940195"],"selectors":[{"id":"name","type":"SelectorText","selector":"div.card-text--truncated__three","parentSelectors":["pagination"],"multiple":false,"regex":"","delay":0},{"id":"date/time","type":"SelectorText","selector":"div.eds-media-card-content__sub-content > div.eds-text-bs--fixed","parentSelectors":["pagination"],"multiple":false,"regex":"","delay":0},{"id":"venue","type":"SelectorText","selector":"div.card-text--truncated__one","parentSelectors":["pagination"],"multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorElementClick","selector":"div.eds-g-cell div.search-event-card","parentSelectors":["_root"],"multiple":true,"delay":"2000","clickElementSelector":"div.eds-l-pad-left-4 a.eds-btn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

Please refer to this post in future:

Hey! Thanks for the info. I actually imported your site map and ran it, but it only made it to the second page still. I'm still playing around with it, but still having some issues. I'll also try your pagination array in the URL

I forgot to mention that you might need to increase page load delay before the actual scrape, as I have default value of 3000 ms for both Page load delay and request interval.

yea i think might have done it. Thanks a lot for your help!

Hey @iconoclast I found another issue maybe you can help? I scraping facebook and using the element scroll. When it gets to the end of the list, it seems to freeze. The window never closes and the data doesnt populate. When do the data preview prior to scraping, everything looks good. Any thoughts?

{"_id":"facebook-october-events-food/drink","startUrl":["https://www.facebook.com/events/discovery/?suggestion_token={"city"%3A"108659242498155"%2C"event_categories"%3A["370585540007142"%2C"412284995786529"]%2C"time"%3A"{\"start\"%3A\"2018-10-01\"%2C\"end\"%3A\"2018-10-09\"}"}&acontext={"ref"%3A51%2C"source"%3A2%2C"source_dashboard_filter"%3A"discovery"%2C"action_history"%3A"[{\"surface\"%3A\"discover_filter_list\"%2C\"mechanism\"%3A\"surface\"%2C\"extra_data\"%3A{\"dashboard_filter\"%3A\"discovery\"}}]"%2C"has_source"%3Atrue}"],"selectors":[{"id":"events","type":"SelectorElement","parentSelectors":["_root","scroll-pagination"],"selector":"div._5lqg","multiple":true,"delay":"3000"},{"id":"name","type":"SelectorText","parentSelectors":["events"],"selector":"a._7ty","multiple":false,"regex":"","delay":0},{"id":"date/venue","type":"SelectorText","parentSelectors":["events"],"selector":"div._42ef div > span","multiple":false,"regex":"","delay":0},{"id":"moreinformation","type":"SelectorLink","parentSelectors":["events"],"selector":"a._7ty","multiple":false,"delay":0},{"id":"scroll-pagination","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div._5lqg","multiple":true,"delay":"2000"},{"id":"category","type":"SelectorText","parentSelectors":["events"],"selector":"div._7u2","multiple":false,"regex":"","delay":0}]}

@iconoclast nevermind you can disregard. I figured it out

1 Like

Hi,

Trying to get pagination working on this page(https://www.ncbi.nlm.nih.gov/pmc/?term=Surface+Plasmon+Resonance+Biacore) without any success. Could someone please help.

Thank you very much in advance

{"_id":"creoptix_spr_biacore","startUrl":["https://www.ncbi.nlm.nih.gov/pmc/?term=surface+plasmon+resonance+biacore"],"selectors":[{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.rprt","multiple":true,"delay":"500","clickElementSelector":"a.active[sid='3']","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"LInk","type":"SelectorLink","parentSelectors":["pagination"],"selector":".title a","multiple":true,"delay":0},{"id":"Names","type":"SelectorText","parentSelectors":["LInk"],"selector":"div.fm-affl","multiple":true,"regex":"","delay":0}]}

hey ! iconoclast. I am also using pagination array. On each page I have 25 elements to grab. However when I look at the extract data it seems it goes randomly to page and select only one element and it come back to all the page (also randomly) until i have my 25 results per page.

I still have all the data I need. However I find the process weird and not efficient. Can you please explain ? Did I do something wrong ?

{"_id":"test400","startUrl":["https://link/students?page=[351-400]&per_page=25&sort_direction=desc&sort_column=default"],"selectors":[{"id":"student","type":"SelectorElement","parentSelectors":["_root"],"selector":".col-md-9 div div:nth-of-type(1) .style__card___1rhof div.style__media-body___1_M6P","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["student"],"selector":"a","multiple":false,"regex":"","delay":0},{"id":"uni","type":"SelectorText","parentSelectors":["student"],"selector":"div.style__text___2ilXR","multiple":false,"regex":"","delay":0},{"id":"degree","type":"SelectorText","parentSelectors":["student"],"selector":"div:nth-of-type(2) div.style__flex-item___1y-Re:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"graduation","type":"SelectorText","parentSelectors":["student"],"selector":"div:nth-of-type(2) div.style__flex-item___1y-Re:nth-of-type(1)","multiple":false,"regex":"","delay":0}]}

my bad actually I did something wrong! Sorry for the disturbance