I've created a scraper that successfully loads every page of the paginated section of the site I want to scrape - however the scraped data export only contains data from the last page of the scraped pages. What am I doing wrong?
Any help is appreciated! Thanks
check this:
{"_id":"new-scraper","startUrl":["https://www.costco.com/s?keyword=whilesupplieslast&langId=-1&sortBy=item_location_pricing_salePrice%2Basc&dept=All&dept=All&refine=item_location_availability-in%20stock%7C%7Citem_program_eligibility-ShipIt"],"selectors":[{"id":"element","multiple":true,"parentSelectors":["pagination"],"selector":"div.MuiGrid2-direction-xs-row:has(div[data-testid*=\"PriceGroup_\"]):has(input[id*=\"CompareProductCheckbox\"])","type":"SelectorElement"},{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":"[aria-label='Go to next page']","type":"SelectorPagination"},{"id":"Price","multiple":false,"parentSelectors":["element"],"regex":"","selector":"div[data-testid*=\"Text_Price\"]","type":"SelectorText"},{"id":"Link","linkType":"linkFromHref","multiple":false,"parentSelectors":["element"],"selector":".MuiTypography-root.MuiTypography-inherit.MuiLink-root","type":"SelectorLink"}]}
Thanks for trying to help! But it doesn't work, unfortunately. The web page shows an error when trying to navigate to the second page (and nothing was scraped from the first page).