Table returning nulls

mistakenlcar · June 17, 2018, 9:58pm

I am trying to scrape this website that has multiple tables in a page. However, while everything goes smoothly the first column of every table after the first returns "null" instead of the contents of that cell. Is there a way to fix this? I've also tried doing a manual scrape on all the cells, but then everything is out of order.

Thanks in advance for any tips!

https://www.autotrader.com/Mercedes-Benz/C-Class/2018

Sitemap:
{"_id":"nmtest","startUrl":["https://www.autotrader.com/Mercedes-Benz/C-Class/2018"],"selectors":[{"id":"test","type":"SelectorTable","selector":"table.margin-vertical-sm","parentSelectors":["_root"],"multiple":true,"columns":[{"header":"Convertible","name":"Convertible","extract":true},{"header":"Original MSRP / Invoice","name":"Original MSRP / Invoice","extract":true},{"header":"Engine","name":"Engine","extract":true},{"header":"Horsepower","name":"Horsepower","extract":true},{"header":"City MPG / Hwy MPG","name":"City MPG / Hwy MPG","extract":true},{"header":"Transmission","name":"Transmission","extract":true},{"header":"Drive","name":"Drive","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr","tableHeaderRowSelector":"thead tr"}]}

iconoclast · June 19, 2018, 9:29pm

Hi there!

You're getting 'null' for tables №2 and №3 because header column contains different title (e.g. sedan/ coupe etc.)

The workaround for this is make a table row selector also pick header rows too. You will have to deselect unneded columns before scraping or else you're getting two extra columns of empty data.

Please try this sitemap:

{"_id":"autotrader","startUrl":["https://www.autotrader.com/Mercedes-Benz/C-Class/2018"],"selectors":[{"id":"table_aio","type":"SelectorTable","selector":"div.container > div.row:nth-of-type(3)","parentSelectors":["_root"],"multiple":true,"columns":[{"header":"Convertible","name":"Convertible","extract":true},{"header":"Original MSRP / Invoice","name":"Original MSRP / Invoice","extract":true},{"header":"Engine","name":"Engine","extract":true},{"header":"Horsepower","name":"Horsepower","extract":true},{"header":"City MPG / Hwy MPG","name":"City MPG / Hwy MPG","extract":true},{"header":"Transmission","name":"Transmission","extract":true},{"header":"Drive","name":"Drive","extract":true},{"header":"Coupe","name":"Coupe","extract":false},{"header":"Original MSRP / Invoice","name":"Original MSRP / Invoice","extract":true},{"header":"Engine","name":"Engine","extract":true},{"header":"Horsepower","name":"Horsepower","extract":true},{"header":"City MPG / Hwy MPG","name":"City MPG / Hwy MPG","extract":true},{"header":"Transmission","name":"Transmission","extract":true},{"header":"Drive","name":"Drive","extract":true},{"header":"Sedan","name":"Sedan","extract":false},{"header":"Original MSRP / Invoice","name":"Original MSRP / Invoice","extract":true},{"header":"Engine","name":"Engine","extract":true},{"header":"Horsepower","name":"Horsepower","extract":true},{"header":"City MPG / Hwy MPG","name":"City MPG / Hwy MPG","extract":true},{"header":"Transmission","name":"Transmission","extract":true},{"header":"Drive","name":"Drive","extract":true}],"delay":0,"tableDataRowSelector":"thead tr:nth-of-type(1), tbody tr","tableHeaderRowSelector":"thead tr:nth-of-type(1)"}]}