Pagination issue in multiple page website

How to?
#1

I am trying to scrap list of companies from following website.

https://www.german-energy-solutions.de/GES/Navigation/EN/CompanyDB/Extern/Suchen/Unternehmen/Extern_Company_Liste_Formular.html?resourceId=616340&input_=616348&pageLocale=en&searchKeywords=&searchTypeOfBusiness=&searchTypeOfBusiness.GROUP=1&searchAreaOfApplication=&searchAreaOfApplication.GROUP=1&searchCompanyName=&searchTechnologyArea=&searchTechnologyArea.GROUP=1&searchApplicationGroup=&searchApplicationGroup.GROUP=1#form-616340

However, scraper select only first 3 pages using pagination. Please let me how I can scrap all linked pages in one sitemap.

Thanks a lot

#2

There is already a very good tutorial on pagination from the WS team:

For this type of site, you can use the second type of pagination, "Use link selector to navigate pagination pages" which is around 1:49

Pay attention to the part where he makes "pagination selector a child of itself". This is an important step which allows the paginator to discover new pages.

Actually, this website changes its URL with each page, so you can also use the first method, "Use one URL with a range definition." But you would need to know the exact number for the last page.

For example, page 22 is
https://www.german-energy-solutions.de/GES/Navigation/EN/CompanyDB/Extern/Suchen/Unternehmen/Extern_Company_Liste_Formular.html?searchTypeOfBusiness.GROUP=1&resourceId=616340&searchTechnologyArea.GROUP=1&input_=616348&searchAreaOfApplication.GROUP=1&searchApplicationGroup.GROUP=1&pageLocale=en&gtp=618726_list%253D22

#3

Nothing more to add :wink:

#4

Thanks, all working very good.