Cannot scrape from multiple pages

drearystate · October 25, 2022, 3:50am

I cannot scrape from multiple pages. I tried following the tutorial and i did what it said without errors but it will only do 1 page then the browser closes. This is same issue with both sites.

https://www.autotrader.com/

Sitemap:
{"_id":"autotrader","startUrl":["https://www.autotrader.com/cars-for-sale/all-cars?zip=74101"],"selectors":[{"id":"pages","paginationType":"auto","parentSelectors":["_root","pages"],"selector":"a[aria-label='Next Page']","type":"SelectorPagination"},{"id":"vehicles","multiple":true,"parentSelectors":["pages"],"selector":".positioned-overlay-base .text-left .justify-content-between a","type":"SelectorLink"},{"id":"vehicle info","multiple":false,"parentSelectors":["vehicles"],"regex":"","selector":"h1","type":"SelectorText"}]}

ViestursWS · October 25, 2022, 9:50am

@drearystate Hello, it does not appear as the 2nd link would be accessible at this moment.

Could you, please, specify what exactly are you looking to scrape from cargurus.com?

drearystate · October 25, 2022, 1:15pm

vehicle's year, make, model, mileage, and price. It's the same from both sites.

ViestursWS · October 25, 2022, 1:47pm

@drearystate Hi, after performing tests it appears that everything should work properly.

Here's a sitemap example(pagination functions till the HTML displays 106-120 available results - if necessary you can change this value by your preference - html:has([data-cg-ft="page-navigation-number-of-listings"]:not(:contains("106 - 120"))) button[data-cg-ft="page-navigation-next-page"]:nth(0)):

{"_id":"cargurus-test","startUrl":["https://www.cargurus.com/Cars/spt_family_cars"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"html:has([data-cg-ft=\"page-navigation-number-of-listings\"]:not(:contains(\"106 - 120\"))) button[data-cg-ft=\"page-navigation-next-page\"]:nth(0)","type":"SelectorPagination"},{"id":"wrapper","multiple":true,"parentSelectors":["pagination"],"selector":"#cargurus-listing-search > div [data-cg-ft=\"car-blade\"]","type":"SelectorElement"},{"id":"title","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"h4:nth(0)","type":"SelectorText"},{"id":"price","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"h4:nth(1)","type":"SelectorText"},{"id":"mileage","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"p:has(span:contains(\"mi\"))","type":"SelectorText"},{"id":"year","multiple":false,"parentSelectors":["wrapper"],"regex":"^([\\w\\-]+)","selector":"h4:nth(0)","type":"SelectorText"},{"id":"make","multiple":false,"parentSelectors":["wrapper"],"regex":"^(?:\\w+ ){2}","selector":"h4","type":"SelectorText"}]}

Also, please, note that the scraped results will start to appear only once the 'Click'/'Click pagination' selector has finished executing.

drearystate · October 26, 2022, 1:02am

ViestursWS:

{"_id":"cargurus-test","startUrl":["https://www.cargurus.com/Cars/spt_family_cars"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"html:has([data-cg-ft="page-navigation-number-of-listings"]:not(:contains("106 - 120"))) button[data-cg-ft="page-navigation-next-page"]:nth(0)","type":"SelectorPagination"},{"id":"wrapper","multiple":true,"parentSelectors":["pagination"],"selector":"#cargurus-listing-search > div [data-cg-ft="car-blade"]","type":"SelectorElement"},{"id":"title","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"h4:nth(0)","type":"SelectorText"},{"id":"price","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"h4:nth(1)","type":"SelectorText"},{"id":"mileage","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"p:has(span:contains("mi"))","type":"SelectorText"},{"id":"year","multiple":false,"parentSelectors":["wrapper"],"regex":"^([\w\-]+)","selector":"h4:nth(0)","type":"SelectorText"},{"id":"make","multiple":false,"parentSelectors":["wrapper"],"regex":"^(?:\w+ ){2}","selector":"h4","type":"SelectorText"}]}

Thank you so very much. it works great and it is faster than I imagined. Ok I see what you did to make that work but that doesnt work on the autotrader website. It has pagination and a next button and it still just gives me the one page. Here's a pic of the elements and the element path. I really need to learn this stuff. I have about half a dozen more sites.