I have been trying to web scrape all the 214 results on this page. https://www.homedepot.com/b/Hardware-Door-Hardware-Door-Locks-Keyless-Door-Locks/N-5yc1vZc2bd?sortorder=desc&sortby=price&&
It has a sort filter, because of which the pagination goes back and forth and takes a lot of time. Also, I am not able to scrape all results.
Thanks in advance!
This is the sitemap used by me:
{"_id":"attempt2","startUrl":["https://www.homedepot.com/b/Hardware-Door-Hardware-Door-Locks-Keyless-Door-Locks/N-5yc1vZc2bd?sortorder=desc&sortby=price&&"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"type":"SelectorLink","selector":".hd-pagination__button a","multiple":true},{"id":"element","parentSelectors":["_root","pagination"],"type":"SelectorElement","selector":".browse-search__pod div.product-pod--padding","multiple":true},{"id":"productname","parentSelectors":["element"],"type":"SelectorText","selector":"div.product-pod__title","multiple":false,"regex":""},{"id":"price","parentSelectors":["element"],"type":"SelectorText","selector":".price-format__main-price span:nth-of-type(2)","multiple":false,"regex":""}]}