How to scrape a troublesome table

How to?
#1

I've been struggling to scrape the price history table details on the following page. My scraper code goes through a series of lists one by one and captures the details. When I create the text scrape element and preview the data, it all looks fine. When I run the scrape it captures NULL for most of the addresses I'm looking at even though there are values for each.

Url:
Root sitemap url: Georgina Real Estate for Sale | From $499,000 | Zoocasa

  • This will scrape through 43 pages.
    Specific url example of the price history table that is not being scraped.
    Georgina Real Estate for Sale | From $499,000 | Zoocasa
  • The code scrapes content from the main page then element clicks to description tab
  • The code clicks on the description tab and is supposed to get the price history information and store in the variables ListDate1, etc.

Sitemap:
{"_id":"testpropertylink","startUrl":["Georgina Real Estate for Sale | From $499,000 | Zoocasa div:nth-of-type(1) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Size","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"section div:nth-of-type(3) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Taxes","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"div:nth-of-type(5) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"description-click","type":"SelectorElementClick","parentSelectors":["linktoproperty"],"selector":"parent","multiple":true,"delay":"1000","clickElementSelector":"div.style_component__rpDkC:nth-of-type(2) li:nth-of-type(2)","clickType":"clickOnce","discardInitialElements":"discard","clickElementUniquenessType":"uniqueText"},{"id":"MLS Number","type":"SelectorText","parentSelectors":["description-click"],"selector":"section div:nth-of-type(1) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Central Vac","type":"SelectorText","parentSelectors":["description-click"],"selector":"section div:nth-of-type(3) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Acreage","type":"SelectorText","parentSelectors":["description-click"],"selector":"section div:nth-of-type(5) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Approx Age","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"div:nth-of-type(7) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Levels","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"section div:nth-of-type(2) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Lot Size","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"div:nth-of-type(6) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Type of Garage","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"div:nth-of-type(4) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Basement","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(9) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"AirCondition","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(13) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"LaundryLevel","type":"SelectorText","parentSelectors":["description-click"],"selector":"section div:nth-of-type(2) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Fireplace","type":"SelectorText","parentSelectors":["description-click"],"selector":"section div:nth-of-type(4) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Exterior","type":"SelectorText","parentSelectors":["description-click"],"selector":"section div:nth-of-type(6) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Driveway","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(10) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Heat","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(12) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"HeatingFuel","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(14) span:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Extras","type":"SelectorText","parentSelectors":["description-click"],"selector":".style_component__3IZXh > p section","multiple":false,"regex":"","delay":0},{"id":"Bedroomsnumber","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"span.style_stat__385D-:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"Bathroomnumber","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"span.style_stat__385D-:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"parkingspacesnumber","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"span.style_stat__385D-:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"PriceSoldFor","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":"div.style_price__e6A8i","multiple":false,"regex":"","delay":0},{"id":"PriceListedFor","type":"SelectorText","parentSelectors":["linktoproperty"],"selector":".style_list-price__1FQe2 span","multiple":false,"regex":"","delay":0},{"id":"ListDate1","type":"SelectorText","parentSelectors":["description-click"],"selector":".style_active__1aXT0 > div:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"ListPrice1","type":"SelectorText","parentSelectors":["description-click"],"selector":".style_active__1aXT0 div:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Status1","type":"SelectorText","parentSelectors":["description-click"],"selector":".style_active__1aXT0 p","multiple":false,"regex":"","delay":0},{"id":"EndDate1","type":"SelectorText","parentSelectors":["description-click"],"selector":".style_active__1aXT0 div:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"SoldPrice1","type":"SelectorText","parentSelectors":["description-click"],"selector":".style_active__1aXT0 div:nth-of-type(5)","multiple":false,"regex":"","delay":0},{"id":"ListDate2","type":"SelectorText","parentSelectors":["description-click"],"selector":"div.style_grid__1WWi7:nth-of-type(4) > div:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"ListPrice2","type":"SelectorText","parentSelectors":["description-click"],"selector":"div.style_grid__1WWi7:nth-of-type(4) div:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Status2","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(4) .style_terminated__1pxe5 p","multiple":false,"regex":"","delay":0},{"id":"EndDate2","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(4) div:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"SoldPrice2","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(4) div:nth-of-type(5)","multiple":false,"regex":"","delay":0},{"id":"ListDate3","type":"SelectorText","parentSelectors":["description-click"],"selector":"div.style_grid__1WWi7:nth-of-type(5) > div:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"ListPrice3","type":"SelectorText","parentSelectors":["description-click"],"selector":"div.style_grid__1WWi7:nth-of-type(5) div:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Status3","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(5) .style_status-label__3_pmx p","multiple":false,"regex":"","delay":0},{"id":"EndDate3","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(5) div:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"SoldPrice3","type":"SelectorText","parentSelectors":["description-click"],"selector":"div:nth-of-type(5) div:nth-of-type(5)","multiple":false,"regex":"","delay":0}]}

#2

@damon Hi. Unfortunately, the sitemap JSON you provided is not valid (most likely due to some pasting error), therefore, before pasting it make sure it is copiable.

Anyway, it seems that the property page can be accessed through a "Link" selector - a[itemprop='streetAddress'] and an "Element scroll down selector" is needed for the property page in order to load any additional information. To extract the price history table you should use an "Element" selector.

Sitemap example:

{"_id":"zoocasa-com","startUrl":["https://www.zoocasa.com/georgina-on-real-estate?page=1"],"selectors":[{"id":"property-link","parentSelectors":["_root"],"type":"SelectorLink","selector":"a[itemprop='streetAddress']","multiple":true,"delay":0},{"id":"property-page","parentSelectors":["property-link"],"type":"SelectorElementScroll","selector":"body:has(h1[itemprop=\"streetAddress\"])","multiple":true,"delay":2000},{"id":"property-title","parentSelectors":["property-page"],"type":"SelectorText","selector":"h1[itemprop='streetAddress']","multiple":false,"delay":0,"regex":""},{"id":"price-history-table","parentSelectors":["property-page"],"type":"SelectorElement","selector":"div[class*=\"style_component\"]:has(h1:contains(\"Price History\")) div[class*=\"style_component\"]:has(h1:contains(\"Price History\"))","multiple":true,"delay":0},{"id":"price-history-table-row","parentSelectors":["price-history-table"],"type":"SelectorElement","selector":"div[class*=\"style_grid\"]:not(:contains(\"List Date\"))","multiple":true,"delay":0},{"id":"List Date","parentSelectors":["price-history-table-row"],"type":"SelectorText","selector":"> div:nth-of-type(1)","multiple":false,"delay":0,"regex":""},{"id":"List Price","parentSelectors":["price-history-table-row"],"type":"SelectorText","selector":"div:nth-of-type(2)","multiple":false,"delay":0,"regex":""},{"id":"Status","parentSelectors":["price-history-table-row"],"type":"SelectorText","selector":"div:nth-of-type(3)","multiple":false,"delay":0,"regex":""},{"id":"End Date ","parentSelectors":["price-history-table-row"],"type":"SelectorText","selector":"div:nth-of-type(4)","multiple":false,"delay":0,"regex":""},{"id":"Sold Price ","parentSelectors":["price-history-table-row"],"type":"SelectorText","selector":"div:nth-of-type(5)","multiple":false,"delay":0,"regex":""}]}

Let me know if everything appears as expected. :blush:

#3

Thanks. The element scroll worked to get the data. Appreciate the help!! ; )