Error with Image links and scraping the whole site

[gregnd]

Hi All,
I am new and having 2 issues, I have tries a few thinks but now need help.
Issue 1. When I use Scraper it does not scrape all of the site it miss out information. The record count does not match. so I have to go through Category by category and do it one by one,and check that each product has been captured.
Issue 2 is the image urls for images are not downloading correctly. But when you check on the preview it correct.

This is what it should be: https://hermalabels.com.au/wp-content/uploads/2018/05/10017_pack_front_2015_72dpi_1692x1135-247x296.jpeg

And this is what i get: data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20247%20296'%3E%3C/svg%3E

I apricate any help to sort this out and understand what i am doing wrong.
Thank you in advance

Url: https://hermalabels.com.au/

Sitemap:
{"_id":"herma-cat-prod-v1","startUrl":["https://hermalabels.com.au/"],"selectors":[{"delay":0,"id":"cat-link","multiple":true,"parentSelectors":["_root"],"selector":".product-categories > li > a","type":"SelectorLink"},{"delay":0,"id":"product-link","multiple":true,"parentSelectors":["cat-link"],"selector":"a.woocommerce-LoopProduct-link","type":"SelectorLink"},{"delay":0,"id":"part-number","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"span.sku","type":"SelectorText"},{"delay":0,"id":"part-desc","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"h1","type":"SelectorText"},{"delay":0,"id":"body-desc","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"div.product-section:nth-of-type(1) div.panel","type":"SelectorText"},{"delay":0,"id":"barcode","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"span.product_code","type":"SelectorText"},{"delay":0,"id":"tags-cat","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"span.posted_in","type":"SelectorText"},{"delay":0,"id":"tags-tags","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"span.tagged_as","type":"SelectorText"},{"delay":0,"id":"weight","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":".woocommerce-product-attributes-item--weight td","type":"SelectorText"},{"delay":0,"id":"addtional-desc","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":"div.product-section:nth-of-type(2) div.panel","type":"SelectorText"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["product-link"],"regex":"","selector":".price bdi","type":"SelectorText"},{"delay":0,"id":"image-1","multiple":false,"parentSelectors":["product-link"],"selector":"img.wp-post-image.skip-lazy","type":"SelectorImage"},{"delay":0,"id":"image-2","multiple":false,"parentSelectors":["product-link"],"selector":".col.first img","type":"SelectorImage"},{"delay":0,"id":"image-3","multiple":false,"parentSelectors":["product-link"],"selector":"div:nth-of-type(2) img.attachment-woocommerce_thumbnail","type":"SelectorImage"},{"delay":0,"id":"image-4","multiple":false,"parentSelectors":["product-link"],"selector":"div:nth-of-type(3) img.attachment-woocommerce_thumbnail","type":"SelectorImage"},{"delay":0,"id":"image-5","multiple":false,"parentSelectors":["product-link"],"selector":"div:nth-of-type(4) img.attachment-woocommerce_thumbnail","type":"SelectorImage"}]}

[ViestursWS]

@gregnd Hi, after having a closer look at your sitemap and the targeted website it seems that new product links are discovered after a scroll down is performed, therefore should update your sitemap by using an 'Element scroll' selector for the product link page.

As for the images, it seems that if you disable the JavaScript for example with this plugin - Toggle JS you should be able to access the 'noscript' image source. For example to extract the second image you could use the following selector - div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(0), third - div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(1) and etc.

[gregnd]

Thank you for your help @ViestursWS

I have tried to instant the element Scroll to the product link page as surgested but now i get a error Error:

• TypeError: Cannot read properties of null (reading 'getElements')

when i check out the scrap txt
and Parent element not found!

Hope you can help me

{"_id":"herma-cat-prod-v1","startUrl":["https://hermalabels.com.au/"],"selectors":[{"id":"cat-link","parentSelectors":["_root"],"type":"SelectorLink","selector":".product-categories > li > a","multiple":true,"delay":0},{"id":"product-link","parentSelectors":["cat-link"],"type":"SelectorElementScroll","selector":".product_tag-address-labels a.woocommerce-LoopProduct-link","multiple":true,"delay":2000},{"id":"part-number","parentSelectors":["product-link"],"type":"SelectorText","selector":"span.sku","multiple":false,"delay":0,"regex":""},{"id":"part-desc","parentSelectors":["product-link"],"type":"SelectorText","selector":"h1","multiple":false,"delay":0,"regex":""},{"id":"body-desc","parentSelectors":["product-link"],"type":"SelectorText","selector":"div.product-section:nth-of-type(1) div.panel","multiple":false,"delay":0,"regex":""},{"id":"barcode","parentSelectors":["product-link"],"type":"SelectorText","selector":"span.product_code","multiple":false,"delay":0,"regex":""},{"id":"tags-cat","parentSelectors":["product-link"],"type":"SelectorText","selector":"span.posted_in","multiple":false,"delay":0,"regex":""},{"id":"tags-tags","parentSelectors":["product-link"],"type":"SelectorText","selector":"span.tagged_as","multiple":false,"delay":0,"regex":""},{"id":"weight","parentSelectors":["product-link"],"type":"SelectorText","selector":".woocommerce-product-attributes-item--weight td","multiple":false,"delay":0,"regex":""},{"id":"addtional-desc","parentSelectors":["product-link"],"type":"SelectorText","selector":"div.product-section:nth-of-type(2) div.panel","multiple":false,"delay":0,"regex":""},{"id":"price","parentSelectors":["product-link"],"type":"SelectorText","selector":".price bdi","multiple":false,"delay":0,"regex":""},{"id":"image-1","parentSelectors":["product-link"],"type":"SelectorImage","selector":"img.wp-post-image.skip-lazy","multiple":false,"delay":0},{"id":"image-2","parentSelectors":["product-link"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(0)","multiple":false,"delay":0},{"id":"image-3","parentSelectors":["product-link"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(1)","multiple":false,"delay":0},{"id":"image-4","parentSelectors":["product-link"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(2)","multiple":false,"delay":0},{"id":"image-5","parentSelectors":["product-link"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(4)","multiple":false,"delay":0}]}

[ViestursWS]

Hi @gregnd

You have misunderstood something the scroll selector should be applied in the product page, not the product listing page.

Example -

{"_id":"herma-cat-prod-v-11","startUrl":["https://hermalabels.com.au/"],"selectors":[{"id":"cat-link","parentSelectors":["_root"],"type":"SelectorLink","selector":".product-categories > li > a","multiple":true,"delay":0},{"id":"product-link","parentSelectors":["cat-link"],"type":"SelectorLink","selector":"a.woocommerce-LoopProduct-link","multiple":true,"delay":0},{"id":"part-number","parentSelectors":["product-page"],"type":"SelectorText","selector":"span.sku","multiple":false,"delay":0,"regex":""},{"id":"part-desc","parentSelectors":["product-page"],"type":"SelectorText","selector":"h1","multiple":false,"delay":0,"regex":""},{"id":"body-desc","parentSelectors":["product-page"],"type":"SelectorText","selector":"div.product-section:nth-of-type(1) div.panel","multiple":false,"delay":0,"regex":""},{"id":"barcode","parentSelectors":["product-page"],"type":"SelectorText","selector":"span.product_code","multiple":false,"delay":0,"regex":""},{"id":"tags-cat","parentSelectors":["product-page"],"type":"SelectorText","selector":"span.posted_in","multiple":false,"delay":0,"regex":""},{"id":"tags-tags","parentSelectors":["product-page"],"type":"SelectorText","selector":"span.tagged_as","multiple":false,"delay":0,"regex":""},{"id":"weight","parentSelectors":["product-page"],"type":"SelectorText","selector":".woocommerce-product-attributes-item--weight td","multiple":false,"delay":0,"regex":""},{"id":"addtional-desc","parentSelectors":["product-page"],"type":"SelectorText","selector":"div.product-section:nth-of-type(2) div.panel","multiple":false,"delay":0,"regex":""},{"id":"price","parentSelectors":["product-page"],"type":"SelectorText","selector":".price bdi","multiple":false,"delay":0,"regex":""},{"id":"image-1","parentSelectors":["product-page"],"type":"SelectorImage","selector":"img.wp-post-image.skip-lazy","multiple":false,"delay":0},{"id":"image-2","parentSelectors":["product-page"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(0)","multiple":false,"delay":0},{"id":"image-3","parentSelectors":["product-page"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(1)","multiple":false,"delay":0},{"id":"image-4","parentSelectors":["product-page"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(2)","multiple":false,"delay":0},{"id":"image-5","parentSelectors":["product-page"],"type":"SelectorImage","selector":"div.product-thumbnails noscript img.attachment-woocommerce_thumbnail:nth(4)","multiple":false,"delay":0},{"id":"product-page","parentSelectors":["product-link"],"type":"SelectorElementScroll","selector":"body:has(h1.entry-title)","multiple":true,"delay":2000}]}