All elements are selected but only limited results returned

I am trying to scrape multiple pages. It appears that I have correctly selected elements when I preview data, however when I scrape only a small subset of results are returned. Any ideas what might be going on?

Url: https://shop.coles.com.au/a/a-national/everything/browse/pantry/breakfast?pageNumber=1

Sitemap:
{"_id":"trial2","startUrl":["https://shop.coles.com.au/a/a-national/everything/browse/pantry?pageNumber=1"],"selectors":[{"id":"element","type":"SelectorElement","parentSelectors":["_root"],"selector":"[data-ng-switch-when='product-citrus'] div.product, .widget .clear div.product","multiple":true,"delay":"2000"},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":".product-title a","multiple":false,"regex":"","delay":0},{"id":"current price","type":"SelectorText","parentSelectors":["element"],"selector":"span.product-pricing-info","multiple":false,"regex":"","delay":0},{"id":"product weight","type":"SelectorText","parentSelectors":["element"],"selector":"p span.package-size","multiple":false,"regex":"","delay":0},{"id":"unit price","type":"SelectorText","parentSelectors":["element"],"selector":"span.package-price","multiple":false,"regex":"","delay":0},{"id":"pageination","type":"SelectorLink","parentSelectors":["_root","pageination"],"selector":"li:nth-of-type(n+2) a.button","multiple":true,"delay":0}]}

Almost there.

Your "element" selector also needs to be a child of "pageination". Currently, "element" is only a child of _Root so it will only scrape the first page, even though the pagination will continue.

Capture

Hi leemeng thank you for that.
I have updated, but the scrape is still only returning 5 results when there are more than 5 on the first page alone. Is there something wrong with the way I have setup the element?

{"_id":"trial2","startUrl":["https://shop.coles.com.au/a/a-national/everything/browse/pantry?pageNumber=1"],"selectors":[{"id":"element","type":"SelectorElement","parentSelectors":["_root","pageination"],"selector":"[data-ng-switch-when='product-citrus'] div.product, .widget .clear div.product","multiple":true,"delay":"2000"},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":".product-title a","multiple":false,"regex":"","delay":0},{"id":"current price","type":"SelectorText","parentSelectors":["element"],"selector":"span.product-pricing-info","multiple":false,"regex":"","delay":0},{"id":"product weight","type":"SelectorText","parentSelectors":["element"],"selector":"p span.package-size","multiple":false,"regex":"","delay":0},{"id":"unit price","type":"SelectorText","parentSelectors":["element"],"selector":"span.package-price","multiple":false,"regex":"","delay":0},{"id":"pageination","type":"SelectorLink","parentSelectors":["_root","pageination"],"selector":"li:nth-of-type(n+2) a.button","multiple":true,"delay":0}]}

Turns out it also needs a scroller, and longer delays. Try this with Page load delay 9000:

{"_id":"coles-test2","startUrl":["https://shop.coles.com.au/a/a-national/everything/browse/pantry/breakfast?pageNumber=1"],"selectors":[{"id":"Item wrappers n scroller","type":"SelectorElementScroll","parentSelectors":["_root","pageination"],"selector":"header.product-header","multiple":true,"delay":"4000"},{"id":"name","type":"SelectorText","parentSelectors":["Item wrappers n scroller"],"selector":".product-title a","multiple":false,"regex":"","delay":0},{"id":"current price","type":"SelectorText","parentSelectors":["Item wrappers n scroller"],"selector":"span.product-pricing-info","multiple":false,"regex":"","delay":0},{"id":"product weight","type":"SelectorText","parentSelectors":["Item wrappers n scroller"],"selector":"p span.package-size","multiple":false,"regex":"","delay":0},{"id":"unit price","type":"SelectorText","parentSelectors":["Item wrappers n scroller"],"selector":"span.package-price","multiple":false,"regex":"","delay":0},{"id":"pageination","type":"SelectorLink","parentSelectors":["_root","pageination"],"selector":"li.page-number:nth-of-type(n+2) a.button","multiple":true,"delay":0}]}

Thanks LeeMeng! Works very well now, you're a legend!

I tried to do a similar setup to get multiple prices within one element on a different website... I can see the details when i do data preview, but not all of them show when I perform the scrape!

{"_id":"woolworths-trial","startUrl":["https://www.woolworths.com.au/shop/browse/pantry?pageNumber=2"],"selectors":[{"id":"element","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"wow-shelf-bundle-tile","multiple":true,"delay":"2000"},{"id":"element2","type":"SelectorElement","parentSelectors":["element2"],"selector":"a","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":"div.shelfBundleTile-title","multiple":false,"regex":"","delay":0},{"id":"details","type":"SelectorText","parentSelectors":["element"],"selector":"li:nth-of-type(1) div.shelfProductVariant-variant, div.shelfProductVariant-details","multiple":true,"regex":"","delay":0}]}

Hi invasian,

do you endup with a success with Woolworth scrapping?
i can not get the link selector or elememt link selector to work on woolworth.com.au

Trying to select product link immediately diverts to the product page.
Unfortunately, there is no data in the html source code if you do view source code.

if you can shed some light that would be fantastic.

I am also experiencing the same behaviour, I can't get it work. :frowning: