Fail to scrape entire scroll down page (already applied element scroll down selector)

How to?
#1

Hi,

I'm new to web scraper and nice to meet you. Recently, I was really enjoy using the Element Scroll Down selector to scrape playlists or charts from streaming platform (though I know some of them has an open API). Things went quite well until I moved on to Spotify. I could only scrape around 25 items from any of the playlist even using the Element Scroll Down selector. Does anyone know how to overcome this issue? Thank you very much!

Here's one of the Spotify playlist example and it's sitemap:

URL:

Sitemap:
{"_id":"spotify-playlist","startUrl":["https://open.spotify.com/playlist/37i9dQZF1DXcBWIGoYBM5M"],"selectors":[{"id":"songlist","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.h4HgbO_Uu1JYg5UGANeQ","multiple":true,"delay":2000},{"id":"rank","parentSelectors":["songlist"],"type":"SelectorText","selector":"div.NZAU7CsuZsMeMQB8zYUu","multiple":false,"delay":0,"regex":""},{"id":"trackname","parentSelectors":["songlist"],"type":"SelectorText","selector":"div.gwYBEX","multiple":false,"delay":0,"regex":""},{"id":"artist","parentSelectors":["songlist"],"type":"SelectorText","selector":".eDbSCl a","multiple":false,"delay":0,"regex":""},{"id":"album","parentSelectors":["songlist"],"type":"SelectorText","selector":"a.standalone-ellipsis-one-line","multiple":false,"delay":0,"regex":""}]}

#2

@jchiu Hi, have you tried zooming out the page as well?

#3

Hi @ViestursWS,

Thanks for the tip! It really helps! Now I can scrape the whole 50 tracks from the playlists above.
However, when I try to scrape playlists with more songs, I fail again. For example, this one Melon Top 100 includes a hundred songs and the other one lofi beats includes even more (a thousand!). After I use the zoom out tip, I can still scrape only 60 tracks from each of them. Is there anything more I can do to scrape them all?

Thanks a lot for your reply!

#4

Hi jchiu,
did you find a solution eventually?
Peace & Prosper