Pagination, Page scroll

How to?
#1

Hi fellow scrapers,

Running into a problem and desperately need support. I am scraping results off of an online store and there is infinite scrolling which is hindering me from scraping off the data from each page. Other than that, I think I am also choosing wrong selectors/scraping flow.

Could someone please be kind enough to look into my sitemap and tell me where I am messing up.

Thanks so much.

Sitemap:
{"_id":"next","startUrl":["https://www2.next.co.uk/shop/department-homeware-productaffiliation-bedding-0"],"selectors":[{"id":"page_frame","type":"SelectorElement","parentSelectors":["_root","pagination"],"selector":"div.Page.loaded.Portrait.live","multiple":true,"delay":"5"},{"id":"product_link","type":"SelectorLink","parentSelectors":["page_frame"],"selector":"a.Image","multiple":true,"delay":0},{"id":"wrapper","type":"SelectorElement","parentSelectors":["product_link"],"selector":"article.Style.Selected section.StyleCopy","multiple":true,"delay":0},{"id":"desc","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#ToneOfVoice","multiple":true,"regex":"","delay":0},{"id":"desc 2","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#WashingInstructions","multiple":false,"regex":"","delay":0},{"id":"material_composition","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#Composition","multiple":true,"regex":"","delay":0},{"id":"composition_data","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#CompositionData","multiple":true,"regex":"","delay":0},{"id":"terms_cond","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#TermsAndConditions","multiple":true,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.nowPrice span","multiple":false,"regex":"","delay":0},{"id":"colour-list","type":"SelectorElementClick","parentSelectors":["wrapper"],"selector":"div.dk_options","multiple":false,"delay":0,"clickElementSelector":"div.dk_container.selectBoxId a.dk_toggle","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTMLText"},{"id":"colour_options","type":"SelectorText","parentSelectors":["colour-list"],"selector":"parent","multiple":true,"regex":"","delay":0},{"id":"first_color","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.dk_container.selectBoxId a.dk_toggle","multiple":true,"regex":"","delay":0},{"id":"title","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.Title","multiple":false,"regex":"","delay":0},{"id":"size_options","type":"SelectorElementClick","parentSelectors":["wrapper"],"selector":"div#dk_container_Size-820-852.dk_container div.dk_options","multiple":false,"delay":0,"clickElementSelector":"div#dk_container_Size-820-852.dk_container a.dk_toggle","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTMLText"},{"id":"sizes","type":"SelectorText","parentSelectors":["size_options"],"selector":"parent","multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorElementScroll","parentSelectors":["_root","pagination"],"selector":"div.ppContainer","multiple":true,"delay":"2000"}]}

#2

Try this

{"_id":"forum-fix-scroll-down","startUrl":["https://www2.next.co.uk/shop/department-homeware-productaffiliation-bedding-0"],"selectors":[{"id":"page_frame","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"section.Details","multiple":true,"delay":"2000"},{"id":"product_link","type":"SelectorLink","parentSelectors":["page_frame"],"selector":"a.TitleText","multiple":false,"delay":0},{"id":"wrapper","type":"SelectorElement","parentSelectors":["product_link"],"selector":"section.ProductDetail","multiple":true,"delay":0},{"id":"desc","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#ToneOfVoice","multiple":true,"regex":"","delay":0},{"id":"material_composition","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#Composition","multiple":true,"regex":"","delay":0},{"id":"composition_data","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#CompositionData","multiple":true,"regex":"","delay":0},{"id":"terms_cond","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div#TermsAndConditions","multiple":true,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.nowPrice span","multiple":false,"regex":"","delay":0},{"id":"colour-list","type":"SelectorElementClick","parentSelectors":["wrapper"],"selector":"div.dk_options","multiple":false,"delay":0,"clickElementSelector":"div.dk_container.selectBoxId a.dk_toggle","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTMLText"},{"id":"colour_options","type":"SelectorText","parentSelectors":["colour-list"],"selector":"parent","multiple":true,"regex":"","delay":0},{"id":"first_color","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.dk_container.selectBoxId a.dk_toggle","multiple":true,"regex":"","delay":0},{"id":"title","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.Title","multiple":false,"regex":"","delay":0},{"id":"size_options","type":"SelectorElementClick","parentSelectors":["wrapper"],"selector":"div#dk_container_Size-820-852.dk_container div.dk_options","multiple":false,"delay":0,"clickElementSelector":"div#dk_container_Size-820-852.dk_container a.dk_toggle","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTMLText"},{"id":"sizes","type":"SelectorText","parentSelectors":["size_options"],"selector":"parent","multiple":false,"regex":"","delay":0}]}
1 Like
#3

@bretfeig you are a genius. Thank you. It is working smoothly. Gracias :smile:

#4

Your Element-Scroll acts to both define each product and scroll down. You don't need another selector for pagination. While the element scroll down needs the multiple box checked. Any selector inside (child to) that doesn't need multiple.

1 Like
#5

Hi there,

I'm having a similar problem and tried replicating the steps here, but I'm pretty new to this.
I'd like to scrape all the data from my Spotify playlists but it only loads the first 50, can you help me figure out where I'm going wrong, please?

{"_id":"spotify","startUrl":["https://open.spotify.com/collection/playlists"],"selectors":[{"id":"playlist url","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.mo-info-name","multiple":true,"delay":"5"},{"id":"name","type":"SelectorText","parentSelectors":["playlist url"],"selector":".mo-info-name span","multiple":true,"regex":"","delay":0},{"id":"desc","type":"SelectorText","parentSelectors":["playlist url"],"selector":".TrackListHeader__entity-long-description","multiple":false,"regex":"","delay":0},{"id":"tracks","type":"SelectorText","parentSelectors":["playlist url"],"selector":".TrackListHeader__entity-additional-info","multiple":false,"regex":"","delay":0}]}

#8

Hi @kaiserpanorama ,
did you find a solution eventually?
I am stuck too, I can scape first 40 rows, but scrolling fails to scrape on Spotify.