Web scraper crashes/fails to collect data

How to?
#1

I am using the free web extension. I want to scrape product data (price & name of product foremost) from a german website with "Load More" buttons. To test it out, I started with a subcategory "gesichtsreinigung" which has 212 products in total and did everything shown in this video: Web Scraper << How to >> Scrape products using "Load More" button

When I press "Data Preview" everything is fine, price and name shown. But when I run the scrape, a seperate chrome browser is opened and the scrape runs. Then the seperate browser either crashes or I don't get any data after clicking "refresh data". I had one case where roughly half of the product names and prices were shown but I can't recreate that.

Down below is the URL im scraping from and my Sitemap.

How can I fix that issue? Perhaps the website blocks me from scraping? It is a really big german retailer.

Down below is the URL im scraping from and my Sitemap.

Url: dm-drogerie markt - dauerhaft günstig online kaufen

Sitemap:
{"_id":"dm-neuladen-gesichtsreinigung-212p","startUrl":["dm-drogerie markt - dauerhaft günstig online kaufen span","multiple":false,"delay":0,"regex":""},{"id":"price","parentSelectors":["product-wrapper"],"type":"SelectorText","selector":"span.qj","multiple":false,"delay":0,"regex":""}]}

#2

@duc Hello. It most likely happens because the selector classes are re-generated each time a new session is launched(serves as a anti-scraping mechanism), therefore you have to pick something more stable.

Also, if you may not have noticed this website has an additional 'Vorherige laden' button, therefore if you haven't specified the 'Load more' selector it will run to an infinite loop.

Vorherige laden
Vorherige laden1758×779 161 KB

Updated sitemap example:

{"_id":"dm-de","startUrl":["https://www.dm.de/search?query=gesichtsreinigung&searchType=product"],"selectors":[{"clickElementSelector":"button[data-dmid=\"load-more-products-button\"]:contains(\"Mehr laden\")","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":1200,"discardInitialElements":"do-not-discard","id":"wrapper-click","multiple":true,"parentSelectors":["_root"],"selector":"div[data-dmid=\"product-tile-container\"]","type":"SelectorElementClick"},{"delay":0,"id":"Title","multiple":false,"parentSelectors":["wrapper-click"],"regex":"","selector":"div[data-dmid=\"product-description\"]","type":"SelectorText"}]}

#3

Websites may implement IP blocking to prevent scraping for crm data enrichment. If you're running the scraper too frequently or making too many requests in a short period, the website might block your IP. Consider adding more delays between requests. Some websites use JavaScript to load content dynamically. Ensure that your web scraper is capable of handling JavaScript-rendered content. If not, you might need a more advanced tool that supports JavaScript rendering.