Initial itens on second page are not scraped

#1

Hello,

I need some help, please.

I have this sitemap:

{"_id":"petlove_example","startUrl":["https://www.petlove.com.br/peixes/filtros-de-aquario"],"selectors":[{"id":"subcategoria","type":"SelectorLink","parentSelectors":["_root"],"selector":"section.catalog-menu a","multiple":true,"delay":0},{"id":"item","type":"SelectorElementClick","parentSelectors":["subcategoria"],"selector":"div.catalog-list-item","multiple":true,"delay":"2000","clickElementSelector":"div.col-xs-10 a.button","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"nome","type":"SelectorText","parentSelectors":["item"],"selector":"h2.product-name","multiple":false,"regex":"","delay":0},{"id":"imagem","type":"SelectorImage","parentSelectors":["item"],"selector":"img.lazyloaded","multiple":false,"delay":0},{"id":"desconto","type":"SelectorText","parentSelectors":["item"],"selector":"div.flag","multiple":false,"regex":"","delay":0},{"id":"marca","type":"SelectorText","parentSelectors":["item"],"selector":"span a","multiple":false,"regex":"","delay":0},{"id":"preco_cheio","type":"SelectorText","parentSelectors":["item"],"selector":"div.catalog-list-price:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"rodape","type":"SelectorText","parentSelectors":["item"],"selector":"div.catalog-list-variant","multiple":false,"regex":"","delay":0}]}

There is a "load more" button at the end of the page. The initial 3-4 products that appear after the button is clicked for the first time are not scraped (it works just fine after the button is clicked for the second time).

Any thoughts?

Thank you in advance!

#2

You're very close! It's not a click selector, it's an element-scroll selector. As you scroll down it auto-loads.

Make this one change and you should be good

#3

Thanks for your help!

There is another issue though. I'm trying to add one more child selector to "item" selector. But when I click the button "Select" to choose the field in the page, it appears this message: "Parent element not found!".

I have already tried to create the sitemap again from the scratch (petlove_example2) and I couldn't add any child selector to this "item" parent. And, as you can possibly notice, it worked before, since I was able to create child selectors to "item" in the original sitemap I posted before (petlove_example).

This is the new sitemap where I cannot create child selectors to "item":

{"_id":"petlove_example2","startUrl":["https://www.petlove.com.br/peixes/filtros-de-aquario"],"selectors":[{"id":"subcategoria","type":"SelectorLink","parentSelectors":["_root"],"selector":"section.catalog-menu a","multiple":true,"delay":0},{"id":"item","type":"SelectorElementScroll","parentSelectors":["subcategoria"],"selector":"div.catalog-list-item","multiple":true,"delay":"6000"},{"id":"","type":"SelectorText","parentSelectors":["item"],"selector":"","multiple":false,"regex":"","delay":0}]}

#4

What info are you trying to get?

this works for me.

{"_id":"petlove_example2","startUrl":["https://www.petlove.com.br/peixes/filtros-de-aquario"],"selectors":[{"id":"subcategoria","type":"SelectorLink","parentSelectors":["_root"],"selector":"section.catalog-menu a","multiple":true,"delay":0},{"id":"item","type":"SelectorElementScroll","parentSelectors":["subcategoria"],"selector":"div.catalog-list-item","multiple":true,"delay":"6000"},{"id":"Name","type":"SelectorText","parentSelectors":["item"],"selector":"h2.product-name","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["item"],"selector":"div.catalog-list-price.catalog-list-price-subscription","multiple":false,"regex":"","delay":0}]}
#5

Yes, this sitemap does work.

But I would like to also scrape "Cod" data on the product page, so I need to add another child selector to "item". The issue is that after I select "item", click on "Add new selector" and then click on the "Select" button it shows the message "Parent element not found!"

[UPDATE] It seems to be a known error, just found this forum topic: Parent element not found!

img1
img1.png875×642 61.2 KB
img2
img2.png870×666 62.7 KB
img3
img3.png863×571 59.1 KB
img4
img4.png915×573 59.8 KB

#6

Ah easy fix.

The site map I created was pulling all the details from the main search page and is contained under an element selector (the one used to scroll down). We didn't add any link to tell it to go into the summary page.

{"_id":"petlove_example2","startUrl":["https://www.petlove.com.br/peixes/filtros-de-aquario"],"selectors":[{"id":"subcategoria","type":"SelectorLink","parentSelectors":["_root"],"selector":"section.catalog-menu a","multiple":true,"delay":0},{"id":"item","type":"SelectorElementScroll","parentSelectors":["subcategoria"],"selector":"div.catalog-list-item","multiple":true,"delay":"6000"},{"id":"Link In ","type":"SelectorLink","parentSelectors":["item"],"selector":"div.catalog-info-product > a:nth-of-type(1)","multiple":false,"delay":0},{"id":"Name","type":"SelectorText","parentSelectors":["Link In "],"selector":"h1.h1","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["Link In "],"selector":"div.product-price-subscription div.product-price","multiple":false,"regex":"","delay":0},{"id":"Code #","type":"SelectorText","parentSelectors":["Link In "],"selector":"div.code-label","multiple":false,"regex":"","delay":0}]}
1 Like
#7

OMG, thank you so much, it's working just fine!

I have been strugling with that since yesterday because I am new in this scraping thing and have minimum HTML knowledge.

Thank you again for your help!!! :heart_eyes:

#8