Pagination skipping pages

How to?
#1

Working on scraping a list of professionals, and pagination will jump from page 1, to 2, to 6, to 10, etc. it is not going page by page. I'd like to enter the pages in a URL, but am not sure how to do this.
In other words, I'd like to request:
https://www.northwell.edu/find-care?page=[1-855]
In other words, I want it to look at https://www.northwell.edu/find-care?page=1, then https://www.northwell.edu/find-care?page=2, then
https://www.northwell.edu/find-care?page=3
etc.

However when I add this as a link, it gives an error to me that it is not valid formatting.

Any guidance on how/where to enter something like
https://www.northwell.edu/find-care?page=[1-855] ?

Thanks so much

#2

This site is unable to be as easy as you wish...
To scrape it - you need something like this as an example:

{"_id":"notrhwell","startUrl":["https://www.northwell.edu/find-care?gender=male&language=cantonese"],"selectors":[{"id":"element","multiple":true,"parentSelectors":["pages"],"selector":"section[class*=\"Card__StyledCard\"]","type":"SelectorElement"},{"clickActionType":"real","clickElementSelector":"a.main-search-button","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard","id":"click","multiple":false,"parentSelectors":["_root"],"selector":"a.main-search-button","type":"SelectorElementClick"},{"id":"pages","paginationType":"clickMore","parentSelectors":["_root","pages"],"selector":"span:has(svg[data-icon=\"chevron-right\"])","type":"SelectorPagination"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["element"],"selector":"a.fad-query-doctor-profile-link","type":"SelectorLink"},{"id":"phone","multiple":false,"parentSelectors":["element"],"regex":"","selector":"a[class*=\"PhoneLink\"]","type":"SelectorText"}]}

There are many filters to make an exact query... check it...

1 Like