Can't Scrape Website with Pagination

bjohn087 · February 16, 2022, 2:55pm

As soon as I add Pagination nothing works, but first time trying to use this. I also want it to get the State column and click into each of the links and get the website. It scrapes the first page without pagination but does nothing with pagination added. I also can't get pagination to work properly with any of the methods I saw. I think it's because I can't use the selector to decide what to click.

Url: Find a Provider - Advanced

Current Sitemap, but I've tried adding the pagination 1000 Ways:
{"_id":"carf","startUrl":["404 Error > option:nth-child(13)","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":100,"discardInitialElements":"do-not-discard","multiple":false,"selector":"select#ctl00_body_ddlProgramFocus"},{"id":"age","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":"#ctl00_body_ddlPopulation > option:nth-child(5)","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":100,"discardInitialElements":"do-not-discard","multiple":false,"selector":"select#ctl00_body_ddlPopulation"},{"id":"search","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":"input#ctl00_body_btnAdvSearch","clickElementUniquenessType":"uniqueHTMLText","clickType":"clickOnce","delay":500,"discardInitialElements":"do-not-discard","multiple":false,"selector":"input#ctl00_body_btnAdvSearch"},{"id":"pagination","parentSelectors":["_root","pagination"],"type":"SelectorLink","selector":"#ctl00_body_gvProviderAdvSearch_ctl13_tblPager a","multiple":true,"delay":0},{"id":"links","parentSelectors":["_root","pagination"],"type":"SelectorLink","selector":".tableWithBorders tr:nth-of-type(n+2) > td > a","multiple":true,"delay":0}]}

ViestursWS · February 16, 2022, 3:48pm

@bjohn087 Hi, due to how this website is structured the best alternative would require using the predefined page range interval pagination for the start URL which can be based on the cid number.

Example:

{"_id":"carf-org-interval","startUrl":["http://www.carf.org/providerProfile.aspx?cid=[223800-223886]"],"selectors":[{"delay":0,"id":"title","multiple":false,"parentSelectors":["_root"],"regex":"","selector":"#ctl00_body_pnlProviderInformation h2","type":"SelectorText"},{"delay":0,"id":"website","multiple":false,"parentSelectors":["_root"],"regex":"","selector":"p a[target]","type":"SelectorText"},{"delay":0,"id":"details","multiple":false,"parentSelectors":["_root"],"regex":"","selector":"#ctl00_body_pnlParentInformation p","type":"SelectorText"}]}

bjohn087 · February 16, 2022, 4:02pm

Thank you, I did think of doing something like that, but the pages seem to range from 30000-500000 and are completely random. It appears it's just not a good site to do it for, I guess. Thank you for the help and input.