Unable to scrape-No product Link

[januaryrain77]

Describe the problem.
Hi Webscraper Forum Admin

I am trying to scrape a website that is apparently has a high degree of anti scraping mechanism

Kategori Kamar Tidur

As you can see the pagination in the bottom is only an arrow, and when I try to create product link, I can not select any product link that can be used for pagination.

That's the first problem.

The second problem , when we click one of the product link
https://tokokasur.com/catalog/spring-bed-royal-romance-120x200-komplit-set_5304/?

You can see on the right hand side there are several size of mattress "Matras 120x200" , "Komplit Set 120x200" , "Komplit Set 160x200", etc etc
whose each mattress size has different corresponding size in the bottom
I first thought this is an element click or attribute click. But I can not select field in webscraper that can extract the price data for each mattress size.
Everytime we click on each mattress size, it trigger the webpage to reload and display the new price for that corresponding mattress size.
This is a new anti scraping strategy that I saw, I never saw this before.

Is webscraper has a way to solve these two problems above?

Thank you so much for your help in advance

[januaryrain77]

This is the sitemark that I used to scrape this website and unable to get pagination and scrape each product page by page

{"_id":"tokokasur-kamartidur","startUrl":["https://tokokasur.com/catalog/category/kamar-tidur_1/?page=[43-44]"],"selectors":[{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"auto","selector":"i.text-blue","type":"SelectorPagination"},{"id":"productlink","parentSelectors":["pagination"],"type":"SelectorLink","selector":".text-left a","multiple":true,"delay":0},{"id":"productname","parentSelectors":["productlink"],"type":"SelectorText","selector":"h1.product-title","multiple":false,"delay":0,"regex":""},{"id":"productcat","parentSelectors":["productlink"],"type":"SelectorText","selector":"ol li:nth-of-type(3) a","multiple":false,"delay":0,"regex":""},{"id":"produkmerek","parentSelectors":["productlink"],"type":"SelectorText","selector":"a.brand-name","multiple":false,"delay":0,"regex":""},{"id":"productdesc","parentSelectors":["productlink"],"type":"SelectorText","selector":"p:nth-of-type(2)","multiple":false,"delay":0,"regex":""},{"id":"productdesc1","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(1)","multiple":false,"delay":0,"regex":""},{"id":"productdesc2","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(2)","multiple":false,"delay":0,"regex":""},{"id":"productdesc3","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(3)","multiple":false,"delay":0,"regex":""},{"id":"productdesc4","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(4)","multiple":false,"delay":0,"regex":""},{"id":"productdesc5","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(5)","multiple":false,"delay":0,"regex":""},{"id":"productdesc6","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(6)","multiple":false,"delay":0,"regex":""},{"id":"productdesc7","parentSelectors":["productlink"],"type":"SelectorText","selector":".desc-tab-content li:nth-of-type(7)","multiple":false,"delay":0,"regex":""},{"id":"productprice1","parentSelectors":["productlink"],"type":"SelectorText","selector":".gdn-product-detail > aside button[data-attr_val='117']","multiple":false,"delay":0,"regex":""},{"id":"productprice1-1","parentSelectors":["productlink"],"type":"SelectorText","selector":".gdn-product-detail > aside h3","multiple":false,"delay":0,"regex":""},{"id":"productpriceall","parentSelectors":["productlink"],"type":"SelectorText","selector":".gdn-product-detail > aside div.full","multiple":false,"delay":0,"regex":""},{"id":"productprice2-1","parentSelectors":["productlink"],"type":"SelectorText","selector":".gdn-product-detail > aside h3","multiple":false,"delay":0,"regex":""},{"id":"spekupc","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('UPC') td","multiple":false,"delay":0,"regex":""},{"id":"spekjenisproduk","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Jenis Produk') td","multiple":false,"delay":0,"regex":""},{"id":"spekketersediaan","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Ketersediaan') td","multiple":false,"delay":0,"regex":""},{"id":"spekbantalan","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Bantalan') td","multiple":false,"delay":0,"regex":""},{"id":"spekberat","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Berat') td","multiple":false,"delay":0,"regex":""},{"id":"speklebar","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Lebar') td","multiple":false,"delay":0,"regex":""},{"id":"spekmaterialtype","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Material Type') td","multiple":false,"delay":0,"regex":""},{"id":"spekpanjang","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Panjang') td","multiple":false,"delay":0,"regex":""},{"id":"spektinggi","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Tinggi') td","multiple":false,"delay":0,"regex":""},{"id":"spekukuran","parentSelectors":["productlink"],"type":"SelectorText","selector":"tr:contains('Ukuran') td","multiple":false,"delay":0,"regex":""},{"id":"productimage","parentSelectors":["productlink"],"type":"SelectorImage","selector":"figure img","multiple":false,"delay":0},{"id":"productprice","parentSelectors":["productlink"],"type":"SelectorText","selector":".gdn-product-detail > aside h3","multiple":false,"delay":0,"regex":""}]}

[leemeng]

Your sitemap does not seem to need pagination 'cos you're already handling that in the Start URL. Try this:

{"_id":"forum-tokokasur-kamartidur","startUrl":["https://tokokasur.com/catalog/category/kamar-tidur_1/?page=[43-44]"],"selectors":[{"id":"productlink","multiple":true,"parentSelectors":["_root"],"selector":"li figure a[title]","type":"SelectorLink"},{"id":"productname","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"h1.product-title","type":"SelectorText"},{"id":"productcat","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"ol li:nth-of-type(3) a","type":"SelectorText"},{"id":"produkmerek","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"a.brand-name","type":"SelectorText"},{"id":"productdesc","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"p:nth-of-type(2)","type":"SelectorText"},{"id":"productdesc1","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(1)","type":"SelectorText"},{"id":"productdesc2","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(2)","type":"SelectorText"},{"id":"productdesc3","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(3)","type":"SelectorText"},{"id":"productdesc4","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(4)","type":"SelectorText"},{"id":"productdesc5","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(5)","type":"SelectorText"},{"id":"productdesc6","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(6)","type":"SelectorText"},{"id":"productdesc7","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".desc-tab-content li:nth-of-type(7)","type":"SelectorText"},{"id":"productprice1","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".gdn-product-detail > aside button[data-attr_val='117']","type":"SelectorText"},{"id":"productprice1-1","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".gdn-product-detail > aside h3","type":"SelectorText"},{"id":"productpriceall","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".gdn-product-detail > aside div.full","type":"SelectorText"},{"id":"productprice2-1","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".gdn-product-detail > aside h3","type":"SelectorText"},{"id":"spekupc","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('UPC') td","type":"SelectorText"},{"id":"spekjenisproduk","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Jenis Produk') td","type":"SelectorText"},{"id":"spekketersediaan","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Ketersediaan') td","type":"SelectorText"},{"id":"spekbantalan","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Bantalan') td","type":"SelectorText"},{"id":"spekberat","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Berat') td","type":"SelectorText"},{"id":"speklebar","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Lebar') td","type":"SelectorText"},{"id":"spekmaterialtype","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Material Type') td","type":"SelectorText"},{"id":"spekpanjang","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Panjang') td","type":"SelectorText"},{"id":"spektinggi","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Tinggi') td","type":"SelectorText"},{"id":"spekukuran","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":"tr:contains('Ukuran') td","type":"SelectorText"},{"id":"productimage","multiple":false,"parentSelectors":["productlink"],"selector":"figure img","type":"SelectorImage"},{"id":"productprice","multiple":false,"parentSelectors":["productlink"],"regex":"","selector":".gdn-product-detail > aside h3","type":"SelectorText"}]}

[januaryrain77]

Hi leemeng,

Thank you so much for your help. Now I am having problem with scraping prices for different mattress size

https://tokokasur.com/catalog/spring-bed-royal-romance-120x200-komplit-set_5304

as I can not use element click selector to scrape the prices

I can use the url in the sitemap as start url , but how can I use webscraper to scrape the entire url in the sitemap, since all the url in http://tokokasur.com/catalog/xxxxxx
are not clickable url like the usual product list

I try to create Product Link selector with link type but I can not select any url since there no clickable url? How can I scrape all the url after
https://tokokasur.com/static/sitemaps/sitemap-products-1.xml

Thank you so much for your help
Jan