Scraping finishes while page is still loading

Bug Reports
#1

Hi,

I'm trying to scrape a website: https://www.nivea.nl/producten?sort=reviewNo with the Pagination selector (click multiple times on next / more button). At random times (apparently when the page loading takes a second longer) the scraper stops. I'm on a stable and fast internet connection but it seems to be in the page. Even though I've but page load time and request interval time to 5 or 10 seconds it happens (even though it doesn't load that long). Anyone knows how to optimize this?

Thank you in advance!! :pray:t2:

Web Scraper version: ?
Chrome version: 120.0.6099.199 (Official Build) (arm64)
OS: MacOS Sonoma 14.0

Sitemap:
{"_id":"Nivea_high_level_reviews","startUrl":["https://www.nivea.nl/producten?sort=reviewNo"],"selectors":[{"id":"Page","parentSelectors":["_root","Page"],"paginationType":"clickMore","type":"SelectorPagination","selector":"button.nx-btn--secondary"},{"id":"Product","parentSelectors":["Page"],"type":"SelectorElement","selector":".nx-teaser-grid a, [title='NIVEA Kersenbloesem & Jojoba Olie Body Soufflé 200ml'] div.nx-product-teaser__image","multiple":true},{"id":"Product name","parentSelectors":["Product"],"type":"SelectorText","selector":"h3","multiple":false,"regex":""},{"id":"Reviews","parentSelectors":["Product"],"type":"SelectorText","selector":"span.nx-rating__count","multiple":false,"regex":""},{"id":"Rating","parentSelectors":["Product"],"type":"SelectorElementAttribute","selector":"span.nx-rating__total","multiple":false,"extractAttribute":"data-rating"}]}

Error Message:
Scraping finished but not all is loaded

#2

Hi, you can try to use the Element click selector instead and set a longer delay. See below sitemap for reference:

{"_id":"Nivea_high_level_reviews","startUrl":["https://www.nivea.nl/producten?sort=reviewNo"],"selectors":[{"clickActionType":"real","clickElementSelector":"button.nx-btn--secondary","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":7000,"discardInitialElements":"do-not-discard","id":"Page","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"id":"Product","multiple":true,"parentSelectors":["_root"],"regex":"","selector":".nx-teaser-grid article:not([data-test=\"categoryTeaser\"])","type":"SelectorText"}]}
1 Like
#3

Thanks I'll try right away!