Bloglovin scrape only returning results above fold?

How to?
#1

Describe the problem.

Hi, I'm trying to scrape a bloglovin page, but can't figure out the how-to-use element click to select the bottom page spinner to load more results. Do I need to create an element scroll parent slector before the element click?

Url: https://www.bloglovin.com/search?q=road%20trip&search_term=road%20trip&context=search_page&referrer_context=autocomplete_enter

Sitemap: {"_id":"bloglovinscraperoadtrip","startUrl":["https://www.bloglovin.com/search?q=road%20trip&search_term=road%20trip&context=search_page&referrer_context=autocomplete_enter"],"selectors":[{"id":"Blog wrapper","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div:nth-of-type(6) ol","multiple":true,"delay":"500","clickElementSelector":"div.page-bottom-spinner","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"blog Links","type":"SelectorLink","parentSelectors":["Blog wrapper"],"selector":".mod-large-img-mode a.grid-post-content-blog-name","multiple":true,"delay":0},{"id":"URLS","type":"SelectorText","parentSelectors":["blog Links"],"selector":"a.header-card-url","multiple":false,"regex":"","delay":0}]}

#2

No, your data wrapper element and data scrapers need to reside within (be a child of) the scroller element.

{"_id":"bloglovinscraperoadtrip-v2","startUrl":["https://www.bloglovin.com/search?q=road%20trip&search_term=road%20trip&context=search_page&referrer_context=autocomplete_enter"],"selectors":[{"id":"Scroller and Blog selectors","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"li:nth-of-type(-n+50) div.grid-post-content","multiple":true,"delay":"3000"},{"id":"Title","type":"SelectorText","parentSelectors":["Scroller and Blog selectors"],"selector":"p.grid-post-content-title-p","multiple":false,"regex":"","delay":0},{"id":"Link","type":"SelectorLink","parentSelectors":["Scroller and Blog selectors"],"selector":"a.grid-post-content-title","multiple":false,"delay":0}]}

This'll scrape the first 50 items. Modify as needed. To scrape more than 50, change the number in li:nth-of-type(-n+50) div.grid-post-content

#3

Thanks, this worked! For anyone trying to also scrape and extract the domain name I've included the updated site map below:

{"_id":"bloglovinscrapetraveltips","startUrl":["https://www.bloglovin.com/search?q=road%20trip&search_term=road%20trip&context=search_page&referrer_context=autocomplete_enter"],"selectors":[{"id":"Scroller and Blog selectors","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"li:nth-of-type(-n+200) div.grid-post-content","multiple":true,"delay":"3000"},{"id":"Title","type":"SelectorText","parentSelectors":["Scroller and Blog selectors"],"selector":"p.grid-post-content-title-p","multiple":false,"regex":"","delay":0},{"id":"Link","type":"SelectorLink","parentSelectors":["Scroller and Blog selectors"],"selector":"a.grid-post-content-blog-name","multiple":false,"delay":0},{"id":"URLS","type":"SelectorText","parentSelectors":["Link"],"selector":"a.header-card-url","multiple":false,"regex":"","delay":0}]}