Hi,
I'm trying to scrape data from 10.000 URLs using this tutorial: 2019 - UnTECH # 17 - Scraping Multiple URL's with Webscraper.io - YouTube
I have no errors in the sitemap, but when the process is complete and I download the CSV, I have data from only the last 3 URLs in the list.
Any idea why this is happening, and why doesn't it scrape data from the whole 10K URLs?
My sitemap has this format (I've edited it just to get the idea):
{
"_id": "sitemap-name",
"startUrl": ["https://www.website.com/state-A/?s=average_rating", "https://www.website.com/state-B/?s=average_rating", "https://www.website.com/state-C/?s=average_rating","etc...for all 10K URLs"], "selectors": [{
"id": "1",
"parentSelectors": ["_root", "pagination"],
"type": "SelectorLink",
"selector": ".class",
"multiple": true,
"delay": 0
}, {"id": "2",
"parentSelectors": ["1"],
"type": "SelectorHTML",
"selector": "class",
"multiple": true,
"regex": "",
"delay": 0
}
]
}