I'm trying to scrape the names and links of products on a website with a 'load more' button, which is similar to the test sites. It was successful on test sites, but on the websites I need to scrape, some records were duplicated, for example, there are 96 products on this site, but it returns 177 records when I choose 'never discard' and 237 records when I choose 'discard when click element exists'.
Hence, I have no idea how to modify my sitemap.
Url: ALL VAPE TANKS - Mesh & Sub-Ohm
Sitemap:
{"_id":"elementvapelink","startUrl":["https://www.elementvape.com/tanks"],"selectors":[{"clickElementSelector":"div.amscroll-load-button","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":500,"discardInitialElements":"discard-when-click-element-exists","id":"product-wrapper","multiple":true,"parentSelectors":["_root"],"selector":".product > div.product-item-inner","type":"SelectorElementClick"},{"delay":0,"id":"name","multiple":false,"parentSelectors":["product-wrapper"],"regex":"","selector":"a.product-item-link","type":"SelectorText"},{"delay":0,"id":"product_link","multiple":false,"parentSelectors":["product-wrapper"],"selector":"a.product-item-link","type":"SelectorLink"}]}