I'm trying to scrape a multi-page table from http://healthinsuranceratings.ncqa.org/2018/search/Commercial/All/ . The pagination links are calls to a javascript function. And, for some bizarre reason, they've decided to include the page navigation within the last row of the data table.
I've tried both link and element click selectors, and can't seem to get either to work. Element Click seems to get closest, as I can see the scraper load the second page; but, then it immediately finishes and no data is retrieved. Here's a slim version of the scrape code that is closest to working, but I can't seem to get it working. Any assistance appreciated:
{"_id":"ncqa_data","startUrl":["http://healthinsuranceratings.ncqa.org/2018/search/Commercial/All/"],"selectors":[{"id":"table_rows","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".displaytable tr:nth-of-type(n+2)","multiple":true,"delay":"2000","clickElementSelector":".displaytable td td a","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"states","type":"SelectorText","parentSelectors":["table_rows"],"selector":"td:nth-of-type(3)","multiple":true,"regex":"","delay":0}]}