Trying to scrape multiple pages on Home Depot

#1

I've tried for a couple days to get this to work but I'm just not getting it. I want to get the Price, title, and image for multiple products but it only pulls results for a few products. And even that varies-sometimes it will scrape 14 products sometimes 1 or 2.

{"_id":"Home-Depot","startUrl":["https://www.homedepot.com/b/Milwaukee/Pick-Up-Today/Special-Values/N-5yc1vZ7ZzvZ1z175a5?NCNI-[24-216:24]"],"selectors":[{"id":"Single","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":".browse-search__pod .pod-spacer__top--2g3md div a","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Single"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"Price","multiple":false,"parentSelectors":["Single"],"regex":"","selector":"div.price-format__large","type":"SelectorText"},{"id":"Image","multiple":false,"parentSelectors":["Single"],"selector":".mediagallery__thumbnail--selected img","type":"SelectorImage"}]}
#2

Hi, this page requires a combination of pagination and scroll down in order to render all products. You can refer to the sitemap below

{"_id":"Home-Depot","startUrl":["https://www.homedepot.com/b/Milwaukee/Pick-Up-Today/Special-Values/N-5yc1vZ7ZzvZ1z175a5?NCNI-[24-216:24]"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":".hd-pagination__button:has([href*=\"Nao=\"]):has([aria-label=\"Next\"]) a","type":"SelectorPagination"},{"delay":2000,"elementLimit":50,"id":"scrolldown","multiple":true,"parentSelectors":["pagination"],"selector":"[class=\"results-wrapped\"] [data-testid=\"product-pod\"]","type":"SelectorElementScroll"},{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["scrolldown"],"selector":"[data-testid=\"ssku-min-swatch\"] a","type":"SelectorLink"},{"id":"title","multiple":false,"parentSelectors":["link"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"price","multiple":false,"parentSelectors":["link"],"regex":"","selector":".price-detailed__unit-price span, div.price-format__large","type":"SelectorText"},{"id":"image","multiple":false,"parentSelectors":["link"],"selector":"[data-testid=\"small-image\"]","type":"SelectorImage"}]}
#3

Thank you. I'll give that a try.

#4

Works great! Thank you.