I'm scraping a large amount of information. My scraper has been running 24 hours a day, for nearly two weeks. I have exported the data twice during this time to check how the scrape is going. Both times, this didn't seem to be a problem - the scraper continued to work; the second CSV was bigger than the first (as expected).
Today I exported the data for a third time. This time, the number of entries in the CSV was the same as when I exported it a week ago (19,597 rows), which implies the scraper hasn't collected any data since the last time I exported the CSV. However, the scraper still appears to be working in the pop-up window.
Is this a problem? Should my 3rd CSV have more entries than my 2nd CSV, if the scraper is working? I am expecting a final CSV of ~76,000 rows. Is this too much information to scrape?
Thank you in advance!