Hello!
I encounter a problem in the scraping process, that after a certain number of scraped products it gives me this error, "502 Bad Gateway
nginx", and from that moment it no longer collects data. Has anyone else had this problem? Did you manage to solve it?
Thank you.
@robert Hi, that most likely happens due to the fact that the website has a broken link or a bot detection mechanism that tries to interrupt the scraper.
Are you able to provide us with the sitemap?
Hi! the site is https://blogsrl.it/
When you ask to provide the sitemap, this is what you want:
{"_id":"blogsrl-food","startUrl":["https://blogsrl.it/gb/"],"selectors":[{"id":"categorie","parentSelectors":["_root"],"type":"SelectorLink","selector":"#czcategory-3 a[data-depth='0']","multiple":true},{"id":"subcategorie","parentSelectors":["categorie"],"type":"SelectorLink","selector":"a.subcategory-name","multiple":true},{"id":"extra-subcategorie","parentSelectors":["subcategorie"],"type":"SelectorLink","selector":"a.subcategory-name","multiple":true},{"id":"link-produse","parentSelectors":["extra-subcategorie"],"type":"SelectorLink","selector":".h5 a","multiple":true}]}
?