Challenges with Web Scraping Amazon Best Seller Rank: Unable to Retrieve Full Product List Even with Element Scroll Down

How to?
#1

Hello everyone,

I'm currently facing a hurdle while attempting to scrape product names from the Amazon Best Seller Rank page. Despite using the Element Scroll Down method, I am still unable to retrieve the complete list of products beyond the initial 30. The remaining 30-50 products only become visible after scrolling down, posing a challenge for my web scraping efforts.

I am reaching out to seek guidance on how to successfully retrieve the full list of 50 products using web scraping techniques. Any advice or alternative approaches to address this limitation would be greatly appreciated. Thank you for your support and insights!

Web link: Amazon Best Sellers: Best Furniture-Style Dog Crates

Sitemap
{"_id":"Test_scrollor","startUrl":["https://www.amazon.com/Best-Sellers-Pet-Supplies-Furniture-Style-Dog-Crates/zgbs/pet-supplies/17602453011/ref=zg_bs_nav_pet-supplies_4_2975401011"],"selectors":[{"id":"scr","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.sif-observe","multiple":true,"delay":2000,"elementLimit":500},{"id":"name","parentSelectors":["scr"],"type":"SelectorText","selector":"div._cDEzb_p13n-sc-css-line-clamp-3_g3dy1","multiple":true,"regex":""}]}

#2

You don't understand how to set scrolling...
Check this sitemap:

{"_id":"Test_scrollor","startUrl":["https://www.amazon.com/Best-Sellers-Pet-Supplies-Furniture-Style-Dog-Crates/zgbs/pet-supplies/17602453011/ref=zg_bs_nav_pet-supplies_4_2975401011"],"selectors":[{"delay":700,"elementLimit":500,"id":"scr","multiple":true,"parentSelectors":["_root"],"selector":"div#gridItemRoot","type":"SelectorElementScroll"},{"id":"name","multiple":false,"parentSelectors":["scr"],"regex":"","selector":"a.a-link-normal span div","type":"SelectorText"},{"extractAttribute":"href","id":"link","multiple":false,"parentSelectors":["scr"],"selector":"a.a-link-normal:has(span)","type":"SelectorElementAttribute"}]}
#3

Thank you so much! It works well.

#4

I would also greatly appreciate learning the correct method for setting the selector to "div#gridItemRoot." Thank you.

#5

seems like you should learn CSS to understand what is id and class )))

image

#6

Hi,

You can learn the basics about CSS selectors on this site:

1 Like
#7

Thank you, this helps me a lot.