Unable to scrap multiple page

How to?
#1

Unable to scrap item from one ecommerce site.

Url: http://lazada.sg/skincare

Sitemap: {"_id":"lazskincare","startUrl":["https://www.lazada.sg/skincare/"],"selectors":[{"id":"Link","type":"SelectorLink","parentSelectors":["_root","page"],"selector":"div.c5TXIP a","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["Link"],"selector":"div.c2prKC:nth-of-type(1) div.c16H9d a","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["Link"],"selector":"div.c2prKC:nth-of-type(1) span.c13VH6","multiple":false,"regex":"","delay":0},{"id":"review","type":"SelectorText","parentSelectors":["Link"],"selector":"div.c2prKC:nth-of-type(1) span.c3XbGJ","multiple":false,"regex":"","delay":0},{"id":"page","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.ant-pagination-item:nth-of-type(n+2) a","multiple":true,"delay":0}]}

#2

Hey I tried this site map but it does seems to scrap every page.

{"_id":"lazskin","startUrl":["https://www.lazada.sg/skincare/"],"selectors":[{"id":"Link","type":"SelectorLink","parentSelectors":["_root","page"],"selector":"div.c16H9d a","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["Link"],"selector":"h1.pdp-product-title","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["Link"],"selector":"span.pdp-price","multiple":false,"regex":"","delay":0},{"id":"rating","type":"SelectorText","parentSelectors":["Link"],"selector":"div.pdp-review-summary a.pdp-link","multiple":false,"regex":"","delay":0},{"id":"page","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.ant-pagination-item:nth-of-type(n+2) a","multiple":true,"delay":0}]}

#3

Hi Chounmin!

After having the same issue with pagination on one of my projects I poked around and learned about bracketing the page numbers for the original URL. By doing that, it moves through each page without a pagination selector as part of the scrape.

The sitemap below should work for you. It scraped over 6 pages for me before I stopped it.

I also noticed in your original sitemap that the title, price and review columns weren't pulling from a valid field so I reset them and also added a function to close down a popup that comes up on some products when opened. Feel free to modify as needed.

{"_id":"lazada","startUrl":["https://www.lazada.sg/skincare/?page=[1-102]"],"selectors":[{"id":"Link","type":"SelectorLink","parentSelectors":["_root"],"selector":"div.c16H9d a","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["Link"],"selector":"h1.pdp-product-title","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["Link"],"selector":"span.pdp-price","multiple":false,"regex":"","delay":0},{"id":"review","type":"SelectorText","parentSelectors":["Link"],"selector":"div.mod-rating div.content","multiple":false,"regex":"","delay":0},{"id":"close popup","type":"SelectorElementClick","parentSelectors":["Link"],"selector":"span.sfo__close","multiple":false,"delay":0,"clickElementSelector":"span.sfo__close","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"}]}

#4

Thanks a million, great support from this forum.