Previous data dissapears after scrolling down

How to?
#1

Hi, I was scraping this url until yesterday, and I was using scroll down feature for that. But now, the site pagination has been changed. Every time that users scroll down, upper data dissappears. it means always only 24 items are available. You can notice that on inspect (I want to get ".col-xs-12" class links)

Url: https://divar.ir/s/mashhad/mobile-phones/apple/honarestan?districts=430%2C445%2C470%2C494%2C431%2C488%2C429%2C468%2C437%2C492%2C456%2C442%2C438%2C433%2C459%2C497%2C498%2C441%2C463%2C491%2C435%2C464%2C462%2C458%2C460%2C484%2C428%2C461%2C472%2C479%2C432%2C434%2C467%2C480%2C444%2C465%2C439%2C447%2C482%2C457%2C489%2C443%2C449%2C455%2C448%2C446

Sitemap:
{"_id":"atew","startUrl":["https://divar.ir/s/mashhad/mobile-phones/apple/honarestan?districts=430%2C445%2C470%2C494%2C431%2C488%2C429%2C468%2C437%2C492%2C456%2C442%2C438%2C433%2C459%2C497%2C498%2C441%2C463%2C491%2C435%2C464%2C462%2C458%2C460%2C484%2C428%2C461%2C472%2C479%2C432%2C434%2C467%2C480%2C444%2C465%2C439%2C447%2C482%2C457%2C489%2C443%2C449%2C455%2C448%2C446"],"selectors":[{"id":"scr","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.col-xs-12","multiple":true,"delay":"2000"},{"id":"lnk","type":"SelectorLink","parentSelectors":["scr"],"selector":"parent","multiple":false,"delay":0}]}

#2

This is likely because the website uses React or Ajax. I don't think WS can scrape it, but Python + Selenium might be able to.

1 Like