Issues scraping a site that loads on scroll

Hi there, I can't seem to figure out how to set up a scrape for a page that loads on scroll. The page is a list of 70 books on StoryGraph and I want to collect details from the page for each books (title, ratings, reviews). Any help would be greatly appreciated! Thank you.

Url: 70 Best of the Best Romance Books of All Time (Romancing the Data) | The StoryGraph

Sitemap:
{"_id":"StoryGraphTop70Ratings","startUrl":["Reading Challenge Not Found | The StoryGraph a.dark\:hover\:text-cyan-500","type":"SelectorText"},{"id":"Rating1","multiple":false,"parentSelectors":["Click"],"regex":"","selector":".col-span-2 span.average-star-rating","type":"SelectorText"},{"delay":1000,"elementLimit":100,"id":"Scroll","multiple":true,"parentSelectors":["_root"],"selector":".col-span-3 div.text-center","type":"SelectorElementScroll"},{"id":"Click","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":".font-bold > a","type":"SelectorLink"},{"id":"Reviews","multiple":true,"parentSelectors":["Click"],"regex":"","selector":".col-span-2 .font-normal a.standard-link","type":"SelectorText"}]}

Hi, here is a reference sitemap to get you started:

{"_id":"thestorygraph","startUrl":["https://app.thestorygraph.com/reading_challenges/0ff3b0af-07b6-4948-bb79-80a1e1c8b91a"],"selectors":[{"delay":2000,"elementLimit":500,"id":"element-wrapper","multiple":true,"parentSelectors":["_root"],"selector":"div.books-panes > div","type":"SelectorElementScroll"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["element-wrapper"],"selector":"a[href*=\"/books/\"]","type":"SelectorLink"},{"id":"title","multiple":false,"parentSelectors":["link"],"regex":"","selector":"h3.text-2xl","type":"SelectorText"},{"id":"rating","multiple":false,"parentSelectors":["link"],"regex":"","selector":".col-span-2 span.average-star-rating","type":"SelectorText"}]}
1 Like

Thank you so much! This worked perfectly for me. I really appreciate your help.

1 Like

I have an additional question - I set up a similar sitemap that has two pages, but the scrolling isn't happening on the second page. Any suggestions?

URL: Amazon Best Sellers: Best Paranormal Romance

Sitemap:
{"_id":"AmazonBestSellerParanormalPaid","startUrl":["https://www.amazon.com/Best-Sellers-Kindle-Store-Paranormal-Romance/zgbs/digital-text/6190484011/ref=zg_bs_pg_1_digital-text?_encoding=UTF8&pg=1"],"selectors":[{"delay":1000,"elementLimit":100,"id":"Scroll","multiple":true,"parentSelectors":["_root"],"selector":"div._cDEzb_iveVideoWrapper_JJ34T","type":"SelectorElementScroll"},{"id":"Pagination","paginationType":"auto","parentSelectors":["_root","Pagination"],"selector":".a-last a","type":"SelectorPagination"},{"id":"Link","linkType":"linkFromHref","multiple":true,"parentSelectors":["Pagination"],"selector":".zg-grid-general-faceout .p13n-sc-uncoverable-faceout > div > div > a","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"span.a-size-large","type":"SelectorText"},{"id":"Author","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".author a","type":"SelectorText"},{"id":"Rating","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"#averageCustomerReviews span.a-color-base","type":"SelectorText"},{"id":"Ratings","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"#averageCustomerReviews span#acrCustomerReviewText","type":"SelectorText"},{"id":"Publisher","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".detail-bullet-list li:nth-of-type(2) span:nth-of-type(2)","type":"SelectorText"},{"id":"PubDate","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".detail-bullet-list li:nth-of-type(3) span:nth-of-type(2)","type":"SelectorText"},{"id":"Length","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"li:nth-of-type(11) span:nth-of-type(2)","type":"SelectorText"},{"id":"Category1","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".zg_hrsr li:nth-of-type(1) a","type":"SelectorText"},{"id":"Category2","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".zg_hrsr li:nth-of-type(2) a","type":"SelectorText"},{"id":"Category3","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".zg_hrsr li:nth-of-type(3) a","type":"SelectorText"}]}

Hi, you have to set the scroll selector as a child to the pagination selector.

{"_id":"AmazonBestSellerParanormalPaid","startUrl":["https://www.amazon.com/Best-Sellers-Kindle-Store-Paranormal-Romance/zgbs/digital-text/6190484011/ref=zg_bs_pg_1_digital-text?_encoding=UTF8&pg=1"],"selectors":[{"delay":1000,"elementLimit":100,"id":"Scroll","multiple":true,"parentSelectors":["Pagination"],"selector":"div._cDEzb_iveVideoWrapper_JJ34T","type":"SelectorElementScroll"},{"id":"Pagination","paginationType":"auto","parentSelectors":["_root","Pagination"],"selector":".a-last a","type":"SelectorPagination"},{"id":"Link","linkType":"linkFromHref","multiple":true,"parentSelectors":["Pagination"],"selector":".zg-grid-general-faceout .p13n-sc-uncoverable-faceout > div > div > a","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"span.a-size-large","type":"SelectorText"},{"id":"Author","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".author a","type":"SelectorText"},{"id":"Rating","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"#averageCustomerReviews span.a-color-base","type":"SelectorText"},{"id":"Ratings","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"#averageCustomerReviews span#acrCustomerReviewText","type":"SelectorText"},{"id":"Publisher","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".detail-bullet-list li:nth-of-type(2) span:nth-of-type(2)","type":"SelectorText"},{"id":"PubDate","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".detail-bullet-list li:nth-of-type(3) span:nth-of-type(2)","type":"SelectorText"},{"id":"Length","multiple":false,"parentSelectors":["Link"],"regex":"","selector":"li:nth-of-type(11) span:nth-of-type(2)","type":"SelectorText"},{"id":"Category1","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".zg_hrsr li:nth-of-type(1) a","type":"SelectorText"},{"id":"Category2","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".zg_hrsr li:nth-of-type(2) a","type":"SelectorText"},{"id":"Category3","multiple":false,"parentSelectors":["Link"],"regex":"","selector":".zg_hrsr li:nth-of-type(3) a","type":"SelectorText"}]}