Fixing Pagination Problem

How to?
#1

Hi Everyody,
I am scraping properties from a site(mentioned below). I have created sitemap but when I tested it in cloud scraper it shows "Finished" after scraping one page. Can you guys please check site map and let me know what I'm missing? Thanks

Url: https://www.immobilienscout24.at/resultlist?useType=RESIDENTIAL&transferType=BUY&estateType=APARTMENT&spot=Wien&region=009001&page=1&matchSubProperties=true&sort=LATEST

Sitemap:
{"_id":"rohnivfriends","startUrl":["https://www.immobilienscout24.at/resultlist?useType=RESIDENTIAL&transferType=BUY&estateType=APARTMENT&spot=Wien&region=009001&page=1&matchSubProperties=true&sort=LATEST"],"selectors":[{"id":"Element","type":"SelectorElement","parentSelectors":["_root","pagination"],"selector":"div.result-list-item.is-expose","multiple":true,"delay":0},{"id":"Name","type":"SelectorLink","parentSelectors":["Element"],"selector":"a.webkitbox","multiple":false,"delay":0},{"id":"Address","type":"SelectorText","parentSelectors":["Element"],"selector":"p.address","multiple":false,"regex":"","delay":0},{"id":"Area","type":"SelectorText","parentSelectors":["Element"],"selector":"div.keyfact-detail.area div.value","multiple":false,"regex":"","delay":0},{"id":"Rooms","type":"SelectorText","parentSelectors":["Element"],"selector":"div.keyfact-detail.rooms div.value","multiple":false,"regex":"","delay":0},{"id":"Purchase price","type":"SelectorText","parentSelectors":["Element"],"selector":"div.keyfact-detail div div.value","multiple":false,"regex":"","delay":0},{"id":"Estage","type":"SelectorText","parentSelectors":["Name"],"selector":"div#floor div.details-value","multiple":false,"regex":"","delay":0},{"id":"Anzahl Geschosse","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfFloors div.details-value","multiple":false,"regex":"","delay":0},{"id":"Objektzustand","type":"SelectorText","parentSelectors":["Name"],"selector":"div#conditionType div.details-value","multiple":false,"regex":"","delay":0},{"id":"Zimmer","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfRooms div.details-value","multiple":false,"regex":"","delay":0},{"id":"Badezimmer","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfBathrooms div.details-value","multiple":false,"regex":"","delay":0},{"id":"Separate WCs","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfToilets div.details-value","multiple":false,"regex":"","delay":0},{"id":"Keller","type":"SelectorText","parentSelectors":["Name"],"selector":"div#hasCellar div.details-value","multiple":false,"regex":"","delay":0},{"id":"Terrassen","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfTerrace div.details-value","multiple":false,"regex":"","delay":0},{"id":"Terrassenfläche","type":"SelectorText","parentSelectors":["Name"],"selector":"div#terraceArea div.details-value","multiple":false,"regex":"","delay":0},{"id":"Personenaufzug","type":"SelectorText","parentSelectors":["Name"],"selector":"div#hasLift div.details-value","multiple":false,"regex":"","delay":0},{"id":"Klimatisiert","type":"SelectorText","parentSelectors":["Name"],"selector":"div#hasAirConditioning div.details-value","multiple":false,"regex":"","delay":0},{"id":"ANBIETER KONTAKTIEREN","type":"SelectorText","parentSelectors":["Name"],"selector":"a.company-url","multiple":false,"regex":"","delay":0},{"id":"Immobiliendetailart","type":"SelectorText","parentSelectors":["Name"],"selector":"div#estateDetailType div.details-value","multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"ul.desktop.ng-dirty li.pagination-page:nth-of-type(n+2) a","multiple":true,"delay":0}]}

#2

You need to use element click selector and since you're using element click selector, you don't need the pagination link recursive set-up. Here is the sitemap. It's going to go through 99 pages until you see any results.

{"_id":"forum-broke-pag-immobilescout","startUrl":["https://www.immobilienscout24.at/resultlist?useType=RESIDENTIAL&transferType=BUY&estateType=APARTMENT&spot=Wien&region=009001&page=1&matchSubProperties=true&sort=LATEST"],"selectors":[{"id":"Element","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".is-expose","multiple":true,"delay":0,"clickElementSelector":"a.arrow-right","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Name","type":"SelectorLink","parentSelectors":["Element"],"selector":"a.webkitbox","multiple":false,"delay":0},{"id":"Address","type":"SelectorText","parentSelectors":["Element"],"selector":"p.address","multiple":false,"regex":"","delay":0},{"id":"Area","type":"SelectorText","parentSelectors":["Element"],"selector":"div.keyfact-detail.area div.value","multiple":false,"regex":"","delay":0},{"id":"Rooms","type":"SelectorText","parentSelectors":["Element"],"selector":"div.keyfact-detail.rooms div.value","multiple":false,"regex":"","delay":0},{"id":"Purchase price","type":"SelectorText","parentSelectors":["Element"],"selector":"div.keyfact-detail div div.value","multiple":false,"regex":"","delay":0},{"id":"Estage","type":"SelectorText","parentSelectors":["Name"],"selector":"div#floor div.details-value","multiple":false,"regex":"","delay":0},{"id":"Anzahl Geschosse","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfFloors div.details-value","multiple":false,"regex":"","delay":0},{"id":"Objektzustand","type":"SelectorText","parentSelectors":["Name"],"selector":"div#conditionType div.details-value","multiple":false,"regex":"","delay":0},{"id":"Zimmer","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfRooms div.details-value","multiple":false,"regex":"","delay":0},{"id":"Badezimmer","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfBathrooms div.details-value","multiple":false,"regex":"","delay":0},{"id":"Separate WCs","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfToilets div.details-value","multiple":false,"regex":"","delay":0},{"id":"Keller","type":"SelectorText","parentSelectors":["Name"],"selector":"div#hasCellar div.details-value","multiple":false,"regex":"","delay":0},{"id":"Terrassen","type":"SelectorText","parentSelectors":["Name"],"selector":"div#numberOfTerrace div.details-value","multiple":false,"regex":"","delay":0},{"id":"Terrassenfläche","type":"SelectorText","parentSelectors":["Name"],"selector":"div#terraceArea div.details-value","multiple":false,"regex":"","delay":0},{"id":"Personenaufzug","type":"SelectorText","parentSelectors":["Name"],"selector":"div#hasLift div.details-value","multiple":false,"regex":"","delay":0},{"id":"Klimatisiert","type":"SelectorText","parentSelectors":["Name"],"selector":"div#hasAirConditioning div.details-value","multiple":false,"regex":"","delay":0},{"id":"ANBIETER KONTAKTIEREN","type":"SelectorText","parentSelectors":["Name"],"selector":"a.company-url","multiple":false,"regex":"","delay":0},{"id":"Immobiliendetailart","type":"SelectorText","parentSelectors":["Name"],"selector":"div#estateDetailType div.details-value","multiple":false,"regex":"","delay":0}]}
1 Like
#3

Thank you so much for your help dear. I just add this sitemap in cloud scraper but it is showing finished after scarpering 950 properties but on the link, there are 11000+ properties. What might be the problem? Can you please help me fix this? I really appreciate.

Screenshot_9
Screenshot_9.jpg1120Ă—323 33.9 KB