Webscraper (pagination) used to work but now stops after a few minutes

How to?
#1

Web Scraper version:
Chrome version: 121.0.6167.85 (Official Build) (arm64)
OS: Sonoma

Sitemap:

{"_id":"bol_product_list","startUrl":["https://www.bol.com/nl/nl/l/fitness-training/14767/"],"selectors":[{"id":"seller_name","multiple":false,"parentSelectors":["product_page"],"regex":"","selector":".product-seller--l span","type":"SelectorHTML"},{"id":"seller_url","multiple":false,"parentSelectors":["product_page"],"selector":".product-seller--l a","type":"SelectorLink"},{"id":"product_page","multiple":true,"parentSelectors":["_root"],"selector":"a.product-image","type":"SelectorLink"},{"id":"rating","multiple":false,"parentSelectors":["seller_url"],"regex":"","selector":"div.seller-rating--xl","type":"SelectorText"},{"id":"offers","multiple":false,"parentSelectors":["seller_url"],"regex":"([0-9])\\w+","selector":".large--is-visible a.nav-list__link","type":"SelectorText"},{"id":"handelsnaam","multiple":false,"parentSelectors":["seller_url"],"regex":"","selector":"dt:contains('Handelsnaam') + dd","type":"SelectorText"}]}

Error Message:

The scraper just finishes with 20-30 rows while there are 1000+ pages to scrape. The Console gives no error messages.

To access error messages follow these steps:

  1. Open chrome://extensions/ or go to manage extensions
  2. Enable “developer mode” at the top right
  3. Open Web Scrapers “background page”
  4. A new popup window should appear.
  5. Go to “Console” tab. You should see Web Scraper log messages and errors there.
#2

Hi, there does not seem to be any pagination configured and also it appears that the website design has changed and the selectors are no longer functional.

I have added page range (?page=[1-500]) to the start URL and adjusted the selectors.

See the following sitemap for reference:

{"_id":"bol_product_list","startUrl":["https://www.bol.com/nl/nl/l/fitness-training/14767/?page=[1-500]"],"selectors":[{"id":"seller_name","multiple":false,"parentSelectors":["product"],"regex":"","selector":"[class=\"product-seller__name\"]","type":"SelectorText"},{"id":"seller_url","linkType":"linkFromHref","multiple":false,"parentSelectors":["product"],"selector":"a.product-title","type":"SelectorLink"},{"id":"product","multiple":true,"parentSelectors":["_root"],"selector":"li.product-item--column","type":"SelectorElement"},{"id":"rating","multiple":false,"parentSelectors":["seller_url"],"regex":"","selector":"[data-test=\"seller-rating\"]","type":"SelectorText"},{"id":"offers","multiple":false,"parentSelectors":["seller_url"],"regex":"([0-9])\\w+","selector":".large--is-visible a.nav-list__link","type":"SelectorText"},{"id":"handelsnaam","multiple":false,"parentSelectors":["seller_url"],"regex":"","selector":"dt:contains('Handelsnaam') + dd","type":"SelectorText"}]}
#4

Thanks a bunch for looking into @JanAp t and the modifications! I'll try to test it and let you know the outcome asap :slight_smile:

#5

Thanks @JanAp ! It seems to work indeed, very grateful for your help!

One question though, as I didn't know you can solve the pagination through the url too. Is there documentation of this and when would you choose this type of solution over a pagination selector?