How to understand why sometimes webscraper stop to scrape?

#1

How to understand why sometimes webscraper stop to scrape? On soem websites is all right, but into other websites, webscraper works for few pages and then it stop by itself, but to my eyes it should go on and on scraping again and again.. but I have no idea about how to investigate and change things to make it to not interrupt scraping..

#2

maybe something wrong with your sitemaps...?

1 Like
#3

I don't think so, this happen to me only when the scraper is just passing to the next page, clicking on the NEXT button.

#4

or my configuration in general..

#5

how many pages passed via pagination clicks?
how many left ?

#6

Super thanks for the interest, when I will get again the issue I will send here the script and the real example, hoping it will not happen again. maybe it was just my hurry, we'll see the future living it.

#7

For example here I got a similar issue where the webscraper was going like in pause, not closing, but not going on scraping data: Can't surf into the detail page of the search results - #6 by ViestursWS

#8

Ok I solved the issue changing computer from my chromebook (that maybe I should reset) to windows 10 and with windows 10 I have no interruptions, thanks for the patience.

Web Scraper on Firefox
Can't surf into the detail page of the search results
#9

Ok I have found a casuality, with chrome is all right, instead with firefox it block without scrape one page, why?
url: https://vendiamos.com/uc/1-3-50k.html
with this sitemap:

{"_id":"ufficiocamerale-it-vendiamos","startUrl":["https://vendiamos.com/uc/1-3-50k.html"],"selectors":[{"id":"name","parentSelectors":["links"],"type":"SelectorText","selector":"strong#field_denominazione","multiple":false,"regex":""},{"id":"pec","parentSelectors":["links"],"type":"SelectorText","selector":"strong#field_pec","multiple":false,"regex":""},{"id":"links","parentSelectors":["_root"],"type":"SelectorLink","selector":"a","multiple":true,"linkType":"linkFromHref"},{"id":"dipendenti","parentSelectors":["links"],"type":"SelectorText","selector":"strong#field_addetti","multiple":false,"regex":""},{"id":"fatturato","parentSelectors":["links"],"type":"SelectorText","selector":".OLDmb-3 li:nth-of-type(6) strong","multiple":false,"regex":""},{"id":"provincia-cciaa","parentSelectors":["links"],"type":"SelectorText","selector":"strong#field_ccia","multiple":false,"regex":""},{"id":"cod-ateco","parentSelectors":["links"],"type":"SelectorText","selector":"strong#field_ateco","multiple":false,"regex":""},{"id":"ateco","parentSelectors":["links"],"type":"SelectorText","selector":"strong#field_desc_ateco","multiple":false,"regex":""}]}```