Hi, I'm new to webscraper so I'm sorry if this question is redundant -- I've tried to follow the directions on existing topics with no luck.
I'm trying to get the data of all 11,000 listings here for a course project:
Url: Apartments for Rent in Manhattan, Bronx, Brooklyn, Queens , NY | StreetEasy
Idk if it's because the listings open in new tabs, but it only works on the independent listing pages and not on the initial search results page above.
I also used this particular listing page to build the sitemap for each page: 750 Columbus Avenue #7L in Upper West Side, Manhattan | StreetEasy
The code works as I intended it to on this page. But for other listing pages, the sitemap sometimes doesn't get all the details like it's supposed to, especially elements that are supposed to retrieve multiple items like amenities.
The pagination doesn't seem to be working either. Any help is appreciated, thank you.
Here is the sitemap:
{"_id":"Streeteasy","startUrl":["https://streeteasy.com/for-rent/nyc/area:200,300,100,400,500"],"selectors":[{"id":"links","parentSelectors":["_root","pages"],"type":"SelectorLink","selector":"a.listingCard-globalLink","multiple":true,"linkType":"linkFromHref"},{"id":"Name","parentSelectors":["listing"],"type":"SelectorText","selector":"a.incognito","multiple":false,"regex":""},{"id":"price","parentSelectors":["listing"],"type":"SelectorText","selector":"div.price","multiple":false,"regex":""},{"id":"square_footage","parentSelectors":["listing"],"type":"SelectorText","selector":"li.first_detail_cell","multiple":false,"regex":""},{"id":"price_per_sqft","parentSelectors":["listing"],"type":"SelectorText","selector":"li.detail_cell:nth-of-type(2)","multiple":false,"regex":""},{"id":"rooms","parentSelectors":["listing"],"type":"SelectorText","selector":"li.detail_cell:nth-of-type(3)","multiple":false,"regex":""},{"id":"beds","parentSelectors":["listing"],"type":"SelectorText","selector":"li.detail_cell:nth-of-type(4)","multiple":false,"regex":""},{"id":"baths","parentSelectors":["listing"],"type":"SelectorText","selector":"li.last_detail_cell","multiple":false,"regex":""},{"id":"neighborhood","parentSelectors":["listing"],"type":"SelectorText","selector":".nobreak a","multiple":false,"regex":""},{"id":"nofee","parentSelectors":["listing"],"type":"SelectorText","selector":"a.NoFeeBadge","multiple":false,"regex":""},{"id":"amenities_highlights","parentSelectors":["listing"],"type":"SelectorGroup","selector":".AmenitiesBlock-highlightsLabel div","extractAttribute":""},{"id":"building_amenities","parentSelectors":["listing"],"type":"SelectorGroup","selector":"ul:nth-of-type(2) li","extractAttribute":""},{"id":"listing_amenities","parentSelectors":["listing"],"type":"SelectorGroup","selector":"div.AmenitiesBlock-item","extractAttribute":""},{"id":"outdoor_amenities","parentSelectors":["listing"],"type":"SelectorGroup","selector":"ul:nth-of-type(3) li","extractAttribute":""},{"id":"total_building_units","parentSelectors":["listing"],"type":"SelectorText","selector":"li.BuildingInfo-detail:nth-of-type(1) span.Text","multiple":false,"regex":""},{"id":"building_stories","parentSelectors":["listing"],"type":"SelectorText","selector":"li.BuildingInfo-detail:nth-of-type(2) span.Text","multiple":false,"regex":""},{"id":"year_built","parentSelectors":["listing"],"type":"SelectorText","selector":"li.BuildingInfo-detail:nth-of-type(3) span.Text","multiple":false,"regex":""},{"id":"subway","parentSelectors":["listing"],"type":"SelectorGroup","selector":".Nearby-transportationList p:nth-of-type(1)","extractAttribute":""},{"id":"subway_proximity","parentSelectors":["listing"],"type":"SelectorGroup","selector":".Nearby-transportationList p.Text","extractAttribute":""},{"id":"parks","parentSelectors":["listing"],"type":"SelectorGroup","selector":"div.Nearby-section:nth-of-type(2) p.Text:nth-of-type(1)","extractAttribute":""},{"id":"parks_proximity","parentSelectors":["listing"],"type":"SelectorGroup","selector":"div.Nearby-section:nth-of-type(2) p.u-nowrap","extractAttribute":""},{"id":"pages","parentSelectors":["_root"],"type":"SelectorLink","selector":"li.page:nth-of-type(n+2) a","multiple":true,"linkType":"linkFromHref"},{"id":"listing","parentSelectors":["links"],"type":"SelectorElement","selector":"div#content","multiple":true}]}