ElementClick losing parenting

How to?
#1

Hey, This is going to be a bit tricky because i'm trying to scrape the shop from the MarketPlaces of the Mirakl group. As an authentication is needed to access those, you will not be able to tryout the sitemap.

I've litteraly been pulling my hair over this for 2 weeks now. I've looked up many topics from this forum who seemed close enough. But i didn't manage to have this work properly.

Sans titre
Sans titre1855×845 139 KB

The website is designed like this. It should be fairly simple but i'm not an expert.

the fact is that those are not links. I think that those elements trigger an API Call to load the next page.

the Html looks like this.

Sans titre
Sans titre1269×508 27.1 KB

So as I've not been able to get links out of those i'm using this sitemap.

Sitemap:

{"_id":"rdc","startUrl":["https://mirakl-web.groupe-rueducommerce.fr/mmp/shop/sell/product?query=T440"],"selectors":[{ "id":"filtre","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.filters-body","multiple":false,"delay":"1000","clickElementSelector":"input[name='category.Ordinateurs portables']","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{ "id":"product","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"tr.clickable","multiple":true,"delay":2000,"clickElementSelector":"tr.clickable","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{ "id":"content","type":"SelectorText","parentSelectors":["_root","next","next2"],"selector":"div[data-e2e='offer-product-panel']","multiple":true,"regex":"","delay":0},{ "id":"next","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.btn","multiple":false,"delay":0},{ "id":"next2","type":"SelectorLink","parentSelectors":["next","next2"],"selector":"a.btn:nth-of-type(2)","multiple":false,"delay":0}]}

The problem is that, as you can see it's working on ElementClick and as the new Elements are opened in a new page the parenting is lost in the process so i'm not able to travel all the Elements. I use the "Suivant" (Next) Button to try to get some out of it but it disapear at the first void page so i can only get like 10 out of 50 products per page.

If you could help me sort this out. i'd gladly appreciate it.
If i did not give precise enough explanations, don't hesitate to tell me.

Thanks

#2

It is probably easier to scrape this site in two stages, where in stage 1, you get all the URLs of the product pages, and then in stage 2 you have a different sitemap which uses all those stage 1 URLs as Starturls.

You can get the URLs (stage 1) with something like:

Type: HTML
Selector: div[class$='mui-row'] > div[class$='mui-info-media']
Regex: (?<=url:')[^']+

#3

Thank you very much for your insight.

I managed to work with this. as a matter of fact the actual URL was the in the "vertical-align-middle" class but i got the gist of it.

If you have any idea about how to concate those two step so that i can save some time, that would be marvelous.