Web scraper is not working properly

Hi there,

Thanks for reading this post. I'm new to web scraper and build my first scraper that works, unfortunately not perfect. As you can see the scraper below is made for bol.com. The idea is to scrape titles descriptions images etc etc.

So when i start scraping with the extension its doing its job. But if you look at the data there are a lot information missing.

  • Titles
  • Descriptions
  • More

Now i have no clue why its missing this data. Maybe its scraping to fast? I tried to run it slower but also same results.

Hopefully someone can help me out!

Url: http://bol.com

Sitemap:

{"_id":"bol","startUrl":["https://www.bol.com/nl/l/puzzels/N/10560/?page=1"],"selectors":[{"id":"title","type":"SelectorText","parentSelectors":["_root"],"selector":"div.product-title--inline","multiple":true,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["_root"],"selector":"span.promo-price","multiple":true,"regex":"","delay":0},{"id":"image","type":"SelectorImage","parentSelectors":["_root"],"selector":"a.product-image","multiple":true,"delay":0},{"id":"description","type":"SelectorText","parentSelectors":["_root"],"selector":"p.medium--is-visible","multiple":true,"regex":"","delay":0},{"id":"voorraad","type":"SelectorText","parentSelectors":["_root"],"selector":"div.product-delivery-highlight","multiple":true,"regex":"","delay":0}]}

When it's scrapes it misses a lot of titles and description which are there. Anyone any idea what went wrong?

Not a bug. This site uses lazy-loading, so you will need a scroller. E.g.:

{"_id":"forum-bol","startUrl":["https://www.bol.com/nl/l/puzzels/N/10560/?page=1"],"selectors":[{"id":"title","type":"SelectorText","parentSelectors":["Scroller"],"selector":"div.product-title--inline","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["Scroller"],"selector":"span.promo-price","multiple":false,"regex":"","delay":0},{"id":"image","type":"SelectorImage","parentSelectors":["Scroller"],"selector":"a img[alt]","multiple":false,"delay":0},{"id":"description","type":"SelectorText","parentSelectors":["Scroller"],"selector":"p.medium--is-visible","multiple":false,"regex":"","delay":0},{"id":"voorraad","type":"SelectorText","parentSelectors":["Scroller"],"selector":"div.product-delivery-highlight","multiple":false,"regex":"","delay":0},{"id":"Scroller","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"li.product-item--row","multiple":true,"delay":"2250"}]}

1 Like