The web site I'm trying to scrape sometimes puts an index page before the data I want to scrape, and sometimes does not. I don't know how to handle this.
Simplified explanation:
There is a page with a list of reporters. Generally you click on a reporter's name to get to their report page, which contains all the fields I want to scrape. That much was easy.
But sometimes a reporter has filed multiple reports, and when you click on that reporter's name, you instead go to an index page with a list of links to each report. The fact that this page is sometimes there and sometimes not is what has me stumped.
I want to scrape the data from all the reports. But how?