Scrape pagination or sub-sub-categories

Hello everyone,

I'm looking for help because I'm stuck.
The website I'm trying to scrape is done a bit strangely and I cannot make it work.

What I want to do is scrape the title and brand on the product page, that's it.
> like on this page

So for that it seems I would need to have the category menu "opened" but it doesn't seem to be clickable via the selector on the main page. However if you go on a random product page it works?!

https://www.selver.ee/oun-paulared-kg

Then, some categories have sub-categories which then depending on the amount of products there could be some pagination with up to 10 pages.

> like on this one

But, on some categories there also are sub-sub-categories (see photo because here it's not clickable)
Capture d’écran 2023-11-16 à 13.04.28

Honeslty this is really hard to map, I'm not sure anyone will figure it out.
But if there is a genius around here, maybe he or she could help me!

Thanks in advance! :pray:

Url: https://www.selver.ee/

Sitemap:
{"_id":"slver","startUrl":["https://www.selver.ee/oun-kanzi-kg"],"selectors":[{"extractAttribute":"class","id":"click","parentSelectors":["_root"],"selector":"li:nth-of-type(n+7) > button","type":"SelectorGroup"},{"id":"sub-category","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root","sub-category"],"selector":".SidebarMenu__item.is-open a","type":"SelectorLink"},{"id":"page","paginationType":"auto","parentSelectors":["sub-category","page"],"selector":".Category__toolbar a.sf-pagination__item","type":"SelectorPagination"},{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["page"],"selector":".ProductCard > a","type":"SelectorLink"},{"id":"title","multiple":false,"parentSelectors":["page"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"SKU","multiple":false,"parentSelectors":["page"],"regex":"","selector":"tr:contains('SKU') td","type":"SelectorText"}]}

@don2010 would you have an idea maybe?

Hi, I will check it a bit later...

You can scrape all items by each category separately, not as a big batch task. Here is all category list:

You can use this sitemap with request interval around 4000 ms:

{"_id":"slver","startUrl":["https://www.selver.ee/ru/kastmed-olid/ketsupid-tomatipastad-kastmed?limit=96"],"selectors":[{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["pages"],"selector":"div.ProductCard h3 a","type":"SelectorLink"},{"id":"pages","paginationType":"clickMore","parentSelectors":["_root","pages"],"selector":"header .sf-pagination__item--next a","type":"SelectorPagination"},{"id":"category","multiple":false,"parentSelectors":["link"],"regex":"","selector":"span.Breadcrumbs__node:nth-last-child(2)","type":"SelectorText"},{"id":"price","multiple":false,"parentSelectors":["link"],"regex":"\\d+,\\d{1,2}\\s€","selector":".ProductPrice","type":"SelectorText"},{"id":"unit_price","multiple":false,"parentSelectors":["link"],"regex":"","selector":"div.Product__prices span.ProductPrice__unit-price","type":"SelectorText"},{"id":"suppl","multiple":false,"parentSelectors":["link"],"regex":"","selector":"tr:contains('Поставщик') td","type":"SelectorText"},{"id":"origin","multiple":false,"parentSelectors":["link"],"regex":"","selector":"tr:contains('Происхождение') td","type":"SelectorText"},{"id":"sku","multiple":false,"parentSelectors":["link"],"regex":"","selector":"tr:contains('SKU') td","type":"SelectorText"},{"id":"img","multiple":false,"parentSelectors":["link"],"selector":".product-image-container img","type":"SelectorImage"}]}

Thanks a lot Don!

However, I'm not sure I got it right, are you saying that I should launch the scraping individually using this category list?

So I should do it like 241 times?

Hi,
Of course you can add all category links at once and start scraping. But it will last too long to check if all is scraped properly.

Hi,

Oh ok I see what you mean.
But thanks a lot for your big help, it works great! :pray:

You are welcome.... That was not too hard to solve