Incomplete scraping

How to?
#1

Hi, I'm trying to extract all the items in the following website. but the extraction stops at around 60 each time. Seems there are multiple type of items if that matters.

Any help appreciated. thanks

https://store.nike.com/au/en_gb/pw/mens-shoes/7puZoi3?ipp=120

Sitemap:

{"_id":"nike-au-men-shoe","startUrl":["https://store.nike.com/au/en_gb/pw/mens-shoes/7puZoi3"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.exp-product-wall","multiple":true,"delay":"10000"},{"id":"popup","type":"SelectorLink","parentSelectors":["_root"],"selector":"div.exp-product-wall > div.grid-item.fullSize div.grid-item-image-wrapper a","multiple":true,"delay":0}]}

#2

To make to sitemap work as it supposed to, you need to pick the item that will get populated after scroll, so that the scroll selector keeps on scrolling. In this case that would be the item 'card'. After that, you would need to select the link that is within the card itself and position it as a child selector to the scroll element.

This should work:

{"_id":"nike-au-men-shoe","startUrl":["https://store.nike.com/au/en_gb/pw/mens-shoes/7puZoi3"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".grid-item-box","multiple":true,"delay":"1500"},{"id":"popup","type":"SelectorLink","parentSelectors":["scroll"],"selector":".grid-item-image a","multiple":false,"delay":0}]}

#3

Thanks. it works.

Wondering how I can find out where an item (for example "grid-item-box") refers to?

#4

By going through the Element section of your console. The selectors that you are hovering over in the console will get highlighted in the page itself, so you can drill down to the element that you are after and see what selectors you need to use to select the correct item that you are after.

#5

I do the same but incomplete scraper on this bwin website, you can help me?

Bwin

{"_id":"win","startUrl":["https://sports.m.bwin.es/en/sports/sport/5/filter/today"],"selectors":[{"id":"item","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".grid-event","multiple":true,"delay":"1500"},{"id":"winlink","type":"SelectorLink","parentSelectors":["item"],"selector":"a.grid-event-wrapper","multiple":false,"delay":0}]}

#6

solution, i did it

{"_id":"wwin","startUrl":["https://sports.m.bwin.es/en/sports/sport/5/filter/today"],"selectors":[{"id":"item","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".grid-event","multiple":true,"delay":"1500","clickElementSelector":".grid-footer","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTML"},{"id":"winlink","type":"SelectorLink","parentSelectors":["item"],"selector":"a.grid-event-wrapper","multiple":false,"delay":0}]}

#7

Hi, I have a further questions.

I want to extract the link for the buttons "Regular" and "Extra Wide" on
https://www.nike.com/au/t/air-zoom-pegasus-35-flyease-running-shoe-d1fvSc/AV2312-016

Seems they are not standard links, and I can't extract the link with the following code. Should I pick another selector type for them? thanks

{"_id":"nike-au-men-shoe","startUrl":["https://store.nike.com/au/en_gb/pw/mens-shoes/7puZoi3"],"selectors":[{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".grid-item-box","multiple":true,"delay":"1500"},{"id":"popup","type":"SelectorLink","parentSelectors":["scroll"],"selector":".grid-item-image a","multiple":true,"delay":0},{"id":"link-fit","type":"SelectorLink","parentSelectors":["popup"],"selector":"label.css-9kpo6p","multiple":true,"delay":0}]}