Can't get all the links of the page

#1

Hi All,

I want to scrape all the vegan product from this site (all the links of the products).

Sitemap:

{"_id":"target","startUrl":["Meatless Alternatives : Target":0}]}

There is something strange, the first page will select all the urls, but when I go to the next page. 10/20 or 30% of the urls will be done (also visible in element preview). How can I fix this? It seems that some products need another selection. Hopefully anyone could help me! Thnx in advice!

(so the products of these urls)

https://www.target.com/s?searchTerm=vegan+meat
https://www.target.com/s?searchTerm=vegan+meat&Nao=24
https://www.target.com/s?searchTerm=vegan+meat&Nao=48
https://www.target.com/s?searchTerm=vegan+meat&Nao=72
https://www.target.com/s?searchTerm=vegan+meat&Nao=96
https://www.target.com/s?searchTerm=vegan+meat&Nao=120
https://www.target.com/s?searchTerm=vegan+meat&Nao=144

#2

Hi @Wouter_Rikkink It most likely happens due to the slow loading pace of this website or some items are discovered only after a scroll is performed. Tested in the Cloud Scraper using one of the community sitemaps template and it seem to work.

target com
target com1780×563 33.3 KB

#3

thnx for your feedback! problem solved!