Pagination selector (new) for “Load more” - Only scraping first page. Pulling my hair out

[Snoopy]

I have watched several different videos in detail and tried everything, I just cannot get it to scrape more than one page. The final video I settled on to be my guide because it deals with the pagination(new) issue that was only released last year was The Pagination Selector Tutorial - YouTube

Site: Companies for Breweries - beverage-world.com

Here is the selector graph

Here is my sitemap

{"_id":"bev-pagination-element","startUrl":["https://www.beverage-world.com/en/business-directory/companies/1/0/0/0"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"linkFromInterceptableJavaScriptClick","selector":"button.subcategory__btn","type":"SelectorPagination"},{"id":"Element","parentSelectors":["pagination"],"type":"SelectorElement","selector":"article","multiple":true,"delay":0},{"id":"Link","parentSelectors":["Element"],"type":"SelectorLink","selector":"a","multiple":false,"delay":0},{"id":"Name","parentSelectors":["Link"],"type":"SelectorText","selector":"h1","multiple":false,"delay":0,"regex":""},{"id":"Website","parentSelectors":["Link"],"type":"SelectorLink","selector":".website a","multiple":false,"delay":0},{"id":"Email","parentSelectors":["Link"],"type":"SelectorLink","selector":".email a","multiple":false,"delay":0}]}

Since asking the question, I have at least managed to scrape the first page, but it does not go to the second.

Please please help. Thank you.

[ViestursWS]

@Snoopy Hello, It happens because you have set the pagination selector to the wrong pagination type, you should use the 'Click multiple times on next/more button' instead.

[Snoopy]

@Snoopy

{"_id":"bev-pagination-element","startUrl":["https://www.beverage-world.com/en/business-directory/companies/1/0/0/0"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"linkFromInterceptableJavaScriptClick","selector":"button.subcategory__btn","type":"SelectorPagination"},{"id":"Element","parentSelectors":["pagination"],"type":"SelectorElement","selector":"article","multiple":true,"delay":0},{"id":"Link","parentSelectors":["Element"],"type":"SelectorLink","selector":"a","multiple":false,"delay":0},{"id":"Name","parentSelectors":["Link"],"type":"SelectorText","selector":"h1","multiple":false,"delay":0,"regex":""},{"id":"Website","parentSelectors":["Link"],"type":"SelectorLink","selector":".website a","multiple":false,"delay":0},{"id":"Email","parentSelectors":["Link"],"type":"SelectorLink","selector":".email a","multiple":false,"delay":0}]}

@ViestursWS Thank you very much for your feedback. Your suggestion was my first choice, but the scraper does not move or do anything in that setting. That is why I tried this weird setting below, and it would actually scrape the first page.

The setting is not logical, but I know so little, it worked much better at least. Still cannot scrape further than the first 50 entries.

It is weird that the pagination would affect scraping before the first 50 entries were scraped.

Thank you very very much for your input. It is much appreciated.

[ViestursWS]

@Snoopy When launching the scrape try to increase the 'Page load delay' to at least 3000, or limit the click amount(because it can also happen due to your browser; OS and network speed); substitute the pagination selector with an 'Element click' selector(the delay set to at least 3'000).

Test results from Web Scraper Cloud; the click was limited till the record that contains - ''Brasserie de Bourbon S.A., Heineken Group''.
Click was limited using the following selector: html:not(:contains("Brasserie de Bourbon S.A., Heineken Group")) button.subcategory__btn

Test in Web Scraper Cloud:

[Snoopy]

@ViestursWS Thank you very much for your great insight and help. I learned a lot.

Biggest part of the problem was my ignorance. I thought:
• The scraper scrapes a page, then presses the "Load more" button, then scrapes the next
• I never gave it time to scrape, because I thought it was doing nothing

Real-world, what it actually does
• The scraper keeps pressing the "Load More" button until all the results are displayed
• Then it starts scraping the information from the end
• I never gave the scraper time to push the button enough, and I thought it was doing nothing. It should have been more patient.

When I implemented the correct button type ( Load More) and kept the delay at 3000, and just left it, it worked.

The positive thing is I managed to eventually scrape the website using both the Pagination (beta) method, en Element click with success. I learned a lot.

Thank you, everybody.

The code to use with delay of 3000:

[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"clickMore","selector":"button.subcategory__btn","type":"SelectorPagination"},{"id":"Element","parentSelectors":["pagination"],"type":"SelectorElement","selector":"article","multiple":true,"delay":0},{"id":"Link","parentSelectors":["Element"],"type":"SelectorLink","selector":"a","multiple":false,"delay":0},{"id":"Name","parentSelectors":["Link"],"type":"SelectorText","selector":"h1","multiple":false,"delay":0,"regex":""},{"id":"Website","parentSelectors":["Link"],"type":"SelectorLink","selector":".website a","multiple":false,"delay":0},{"id":"Email","parentSelectors":["Link"],"type":"SelectorLink","selector":".email a","multiple":false,"delay":0}]}