Export doesn't show all data

How to?
#1

I'm new to the site and non-technical, but after watching tutorials have managed to work out how to scrape on online retailer site. There's 64 records in total. The page initially shows the first 24 ... you then need to click a "More" button to load another 24, followed by another "More" button to load the remaining 16.

I've used the pagination options with the 'Click multiple times ...' type as auto didn't work. When I then do a Data Preview is works perfectly (although the 3 data fields I want are listed seperately rather than on one line, but I can work around that).

However, when I do the scrape and export, it only downloads returns from the first "More" button ... 48 returns instead of 64.

I've done a workaround (by copying the data from the Preview and pasting into a spreadsheet), but would be interesting in understanding what the problem is.

Appreciate any suggestions and remember ... I'm not technical. So quoting complicated code won't help me :slight_smile:

Url: https://www.danmurphys.com.au/whisky/australian-whisky

Sitemap:
{"_id":"dans4","startUrl":["https://www.danmurphys.com.au/whisky/australian-whisky?sort=PriceAsc"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"clickMore","selector":".dans-button span.mat-button-wrapper","type":"SelectorPagination"},{"id":"Company","parentSelectors":["element-wrapper"],"type":"SelectorText","selector":"span.title","multiple":true,"regex":""},{"id":"Product","parentSelectors":["element-wrapper"],"type":"SelectorText","selector":"span.subtitle","multiple":true,"regex":""},{"id":"Price","parentSelectors":["element-wrapper"],"type":"SelectorText","selector":"span.value","multiple":true,"regex":""},{"id":"element-wrapper","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"shop-product-card-find-nearby-wrapper div.product","multiple":true,"delay":2000,"elementLimit":500}]}

#2

@timl Hello, it appears that the issue arises due to an incorrect pagination setup. The 'element-wrapper' has to be set as the child of the pagination in order to execute continous loop.

Here's a sitemap example:

{"_id":"dans4","startUrl":["https://www.danmurphys.com.au/whisky/australian-whisky?sort=PriceAsc"],"selectors":[{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":".dans-button span.mat-button-wrapper","type":"SelectorPagination"},{"id":"Company","multiple":false,"parentSelectors":["element-wrapper"],"regex":"","selector":"span.title","type":"SelectorText"},{"id":"Product","multiple":false,"parentSelectors":["element-wrapper"],"regex":"","selector":"span.subtitle","type":"SelectorText"},{"id":"Price","multiple":false,"parentSelectors":["element-wrapper"],"regex":"","selector":"span.value","type":"SelectorText"},{"id":"element-wrapper","multiple":true,"parentSelectors":["pagination"],"selector":"shop-product-card-find-nearby-wrapper div.product","type":"SelectorElement"}]}
#3

Hi @ViestursWS

You're a legend - thank you so much! Works perfectly. Shame the tutorials don't provide the level of accurate detail required to make this tool work, especially for non-tech people like myself.

Thanks again,

Tim