Duplicate records while Scraping ecommerce sites

How to?
#1

As part of a competitor analysis - I am trying to extract all product titles and prices from different categories from Online Pharmacies. These also include pagination. It looks my sitemap is scraping duplicate records. From what I have read online, it looks like it could be an issue with element selector but would appreciate an expert advice on this.

Url: https://asteronline.com/personal-care/nail-care.html (The site is slow)

Sitemap:

{
"_id": "aster_1",
"startUrl": [
"https://asteronline.com/personal-care/nail-care.html"
],
"selectors": [
{
"id": "pagination",
"paginationType": "clickMore",
"parentSelectors": [
"_root",
"pagination"
],
"selector": ".product-list-container .pages-items a",
"type": "SelectorPagination"
},
{
"delay": 0,
"id": "parent",
"multiple": true,
"parentSelectors": [
"pagination"
],
"selector": "li.product",
"type": "SelectorElement"
},
{
"delay": 0,
"id": "title",
"multiple": false,
"parentSelectors": [
"parent"
],
"regex": "",
"selector": "a.product-item-link",
"type": "SelectorText"
},
{
"delay": 0,
"id": "price",
"multiple": false,
"parentSelectors": [
"parent"
],
"regex": "",
"selector": "div.price-box",
"type": "SelectorText"
}
]
}

#2

@mrsid2201 Hi, there does not seem to be a particular issue with your sitemap configuration, however, I would recommend using the 'Link' type pagination instead of a 'Click' in this case.

{"_id":"aster_1","startUrl":["https://asteronline.com/personal-care/nail-care.html"],"selectors":[{"id":"pagination","paginationType":"linkFromHref","parentSelectors":["_root","pagination"],"selector":".product-list-container a.next","type":"SelectorPagination"},{"delay":0,"id":"parent","multiple":true,"parentSelectors":["pagination"],"selector":"li.product","type":"SelectorElement"},{"delay":0,"id":"title","multiple":false,"parentSelectors":["parent"],"regex":"","selector":"a.product-item-link","type":"SelectorText"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["parent"],"regex":"","selector":"div.price-box","type":"SelectorText"}]}

#3

Thank you for your reply. I have tried this but the duplication still seems to be happening unfortunately. In the below image, you can see that it shows the same product from two different pages. However, on checking this manually, that does not seem to be the case

image
image1407×55 6.19 KB

Moreover, I have also noticed that it is not picking up all the products from all the pages.

#4

@mrsid2201 You can try extending the scrape to the product page(Web Scraper will not scrape the same link twice).

{"_id":"aster_1","startUrl":["https://asteronline.com/personal-care/nail-care.html"],"selectors":[{"id":"pagination","paginationType":"linkFromHref","parentSelectors":["_root","pagination"],"selector":".product-list-container a.next","type":"SelectorPagination"},{"delay":0,"id":"parent","multiple":true,"parentSelectors":["pagination"],"selector":"li.product","type":"SelectorElement"},{"delay":0,"id":"title","multiple":false,"parentSelectors":["parent"],"regex":"","selector":"a.product-item-link","type":"SelectorText"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["parent"],"regex":"","selector":"div.price-box","type":"SelectorText"},{"delay":0,"id":"link-to-product","multiple":false,"parentSelectors":["parent"],"selector":"a.product-item-link","type":"SelectorLink"},{"delay":0,"id":"stock","multiple":false,"parentSelectors":["link-to-product"],"regex":"","selector":".stock span","type":"SelectorText"}]}

Afterwards, you can run the duplicate test here: https://dedupelist.com/