Pagination question

How to?
#1

Describe the problem.
I need to scrape the website in the URL below. The URL links to the first page, but the website is directory and contains a total of 708 individual pages. I want to scrape all 708 pages, but I am having issues with pagination

Url: https://directory.justice.org/SearchResult.asp?access=public&firstmiddlename=&middlename=&lastname=&maidenname=&firmname=&city=&provstateid=&zip=&countryid=USA&keyword=&areaofpractice=&areaofpractice2=&sectiontype=&memtype=&sb=&gender=Any

Below is the simple sitemap I created:
Sitemap:
{"_id":"scrape_full_list","startUrl":["https://directory.justice.org/SearchResult.asp?access=public&firstmiddlename=&middlename=&lastname=&maidenname=&firmname=&city=&provstateid=&zip=&countryid=USA&keyword=&areaofpractice=&areaofpractice2=&sectiontype=&memtype=&sb=&gender=Any"],"selectors":[{"id":"selector","type":"SelectorElement","parentSelectors":["_root"],"selector":"div#leftcolumn","multiple":true,"delay":0},{"id":"firm info","type":"SelectorText","parentSelectors":["selector"],"selector":"td[width] div:nth-of-type(1)","multiple":true,"regex":"","delay":0}]}

I don'y have pagination built into the sitemap above. But in previous attempts I have tried creating a pagination loop in the url as follows:
https://directory.justice.org/SearchResult.asp?access=public&gender=Any&firstmiddlename=&lastname=&maidenname=&firmname=&city=&provstateid=&zip=&countryid=USA&keyword=&areaofpractice=&areaofpractice2=&sectiontype=&memtype=&sb=&PID={2052D5B4-C15F-4CAE-AE35**-6D86552046F3}&PRC=17697&page=[0-708]**
But that hasn't worked full all 708 pages. Can anyone help or offer advice? Thanks in advance

#2

This:

{"_id":"scrape_full_list","startUrl":["https://directory.justice.org/SearchResult.asp?access=public&firstmiddlename=&middlename=&lastname=&maidenname=&firmname=&city=&provstateid=&zip=&countryid=USA&keyword=&areaofpractice=&areaofpractice2=&sectiontype=&memtype=&sb=&gender=Any"],"selectors":[{"id":"selector","type":"SelectorElement","parentSelectors":["_root","pagination"],"selector":"div#leftcolumn","multiple":true,"delay":0},{"id":"firm info","type":"SelectorText","parentSelectors":["selector"],"selector":"td[width] div:nth-of-type(1)","multiple":true,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"#ResultLinks a:has(img[alt=\"Next Page\"])","multiple":true,"delay":0}]}

1 Like
#3

Cool, I never thought of using img[alt.