Spotify : Scraping a scrolling list "Top 10,000 Songs of All Time"

How to?
#1

Hi Fantastic Forum Members,

I have searched the forum for the same keywords, and found many post alike, with one solution that does not cure the issue (trying to zoom out the spotify list).

While trying to scrape "Top 10,000 Songs of All Time"
" Top 10,000 Songs of All Time - playlist by Acclaimed Music | Spotify "

It scrapes OK first ones, but I can't get pass 32th row

For the record, I have tried every levels of selection to find a solution, using Parents of Parents, but it failed too.

Sitemap:
{"_id":"SPTF10K","startUrl":["https://open.spotify.com/playlist/1G8IpkZKobrIlXcVPoSIuf"],"selectors":[{"id":"SongRow","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.jEMA2gVoLgPQqAFrPhFw","multiple":true,"delay":500,"elementLimit":0},{"id":"Rank","parentSelectors":["SongRow"],"type":"SelectorText","selector":"div.NZAU7CsuZsMeMQB8zYUu","multiple":false,"regex":""},{"id":"Title","parentSelectors":["SongRow"],"type":"SelectorText","selector":"a.t_yrXoUO3qGsJS4Y6iXX","multiple":false,"regex":""},{"id":"Performer","parentSelectors":["SongRow"],"type":"SelectorText","selector":".bGROfl a","multiple":false,"regex":""},{"id":"Album","parentSelectors":["SongRow"],"type":"SelectorText","selector":"a.standalone-ellipsis-one-line","multiple":false,"regex":""},{"id":"Duration","parentSelectors":["SongRow"],"type":"SelectorText","selector":"div.bDHxRN","multiple":false,"regex":""}]}

Any clue?
TIA

#2

Spotify is a tricky one and you will need some advanced tricks. Notably, this page uses a separate scroller for the playlist section. You will need to use the un(der)documented "scrollElementSelector" scroller parameter to select the correct scroll element. This parameter must be added manually in the Json (it cannot be added via GUI).

Try this sitemap which will stop at around 75 results. You can adjust the number in Element limit.

{"_id":"spotify-2024","startUrl":["https://open.spotify.com/playlist/1G8IpkZKobrIlXcVPoSIuf"],"selectors":[{"delay":1500,"elementLimit":75,"id":"special scroller","multiple":true,"parentSelectors":["_root"],"scrollElementSelector":"div.main-view-container>div>div.os-padding>div.os-viewport","selector":"div#main div.main-view-container div[role='grid'] div[style^='transform'] > div[role='row']","type":"SelectorElementScroll"},{"id":"Row selectors","multiple":false,"parentSelectors":["special scroller"],"selector":"div[data-testid]","type":"SelectorElement"},{"id":"number","multiple":false,"parentSelectors":["Row selectors"],"regex":"","selector":"div[aria-colindex='1']","type":"SelectorText"},{"id":"track n link","linkType":"linkFromHref","multiple":false,"parentSelectors":["Row selectors"],"selector":"a[data-testid='internal-track-link']","type":"SelectorLink"},{"id":"artist","multiple":false,"parentSelectors":["Row selectors"],"regex":"","selector":"div[aria-colindex='2'] span","type":"SelectorText"},{"id":"album","multiple":false,"parentSelectors":["Row selectors"],"regex":"","selector":"div[aria-colindex='3']","type":"SelectorText"}]}

The scroller parameter I added is:
"scrollElementSelector": "div.main-view-container>div>div.os-padding>div.os-viewport",

#3

Hello @leemeng I'm trying to reproduce the same, how did you find that " div[data-testid]" is the selector for the special scroll ?

#4

You will need some knowledge of HTML and CSS. I found it manually by looking in the Elements tab