Pagination - last page only

#1

In this scrape, there are three pages of data. I'm only receiving data for the third page when I scrape. What have I done wrong?

{"_id":"sjss","startUrl":["https://www.sjss.org.uk/whats-on?field_tag_tid=All&&keys=&field_date_value_1[min]&field_date_value_1[max]&combine=&page=[1-3]"],"selectors":[{"id":"More info","type":"SelectorLink","parentSelectors":["_root"],"selector":"p a.button:nth-of-type(1)","multiple":true,"delay":0},{"id":"Concert title","type":"SelectorText","parentSelectors":["More info"],"selector":"h1.whats-on-title","multiple":false,"regex":"","delay":0},{"id":"Byline","type":"SelectorText","parentSelectors":["More info"],"selector":".large-6 p","multiple":false,"regex":"","delay":0},{"id":"Concert date","type":"SelectorText","parentSelectors":["More info"],"selector":"div.ticket-date","multiple":false,"regex":"","delay":0}]}

#2

This should work:

{"_id":"sjss","startUrl":["https://www.sjss.org.uk/whats-on?field_tag_tid=All&&keys=&field_date_value_1[min]=&field_date_value_1[max]=&combine="],"selectors":[{"id":"More info","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":".result p a.button:contains('Info')","multiple":true,"delay":0},{"id":"Concert title","type":"SelectorText","parentSelectors":["More info"],"selector":"h1.whats-on-title","multiple":false,"regex":"","delay":0},{"id":"Byline","type":"SelectorText","parentSelectors":["More info"],"selector":".large-6 p","multiple":false,"regex":"","delay":0},{"id":"Concert date","type":"SelectorText","parentSelectors":["More info"],"selector":"div.ticket-date","multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"a[title='Go to next page']","multiple":true,"delay":0}]}

The range for pagination is not quite correct for this particular site

#3

Thank you so much. Why is that? I have a load of different sites to grab from and scrolling always seems to be different. I don't really know what I'm looking for to set it up correctly. Another example is:

https://www.barbican.org.uk/whats-on/classical-music - here I have an infinite scroll but it always seems to crash before I grab all the info, or it doesn't scroll all the way to the 'bottom' before it starts grabbing the rest of the info (I want all the details until July 2020, at the moment it's only getting as far as October, but the scrolling seems to work well, I just can't repeat the process).

{"_id":"barbican4","startUrl":["https://www.barbican.org.uk/whats-on/classical-music"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.views-infinite-scroll-content-wrapper","multiple":true,"delay":"20000"},{"id":"moreinfo","type":"SelectorLink","parentSelectors":["scroll"],"selector":"a.button","multiple":true,"delay":0},{"id":"Concerttitle","type":"SelectorText","parentSelectors":["moreinfo"],"selector":"h1.heading-group__primary","multiple":false,"regex":"","delay":0}]}

EDIT: Even though I have now made two parent selectors for the scroll as in the other example, it only ever collects data as late as 2 November and I don't know why :frowning: