Scrolling and Show more button which does not work

#1

I try to scrape this page Articles on Self-Knowledge, Relationships and Calm, but the problem is that Pagination does not work, because new articles appear just when you scroll on the "More" button, but you do not need to push it, just scroll to it. Pagination tries to push it, and it does not work. I tried to use Element (scroll), it scrolls down to the limit of page, and not stopping long enough on More button for expansion to start working. PLease help

Screenshot 2025-06-08 224713
Screenshot 2025-06-08 2247131897×747 68.6 KB 

{"_id":"LastPAginSchool","startUrl":["https://www.theschooloflife.com/articles/"],"selectors":[{"id":"Articles","parentSelectors":["_root"],"type":"SelectorLink","selector":".animated a.article-block__content__more, a.btn-dark","multiple":true,"linkType":"linkFromHref"},{"id":"Scroll","parentSelectors":["Articles","Scroll"],"paginationType":"auto","type":"SelectorPagination","selector":"button.load-more-articles"},{"id":"Date","parentSelectors":["Scroll"],"type":"SelectorText","selector":"time","multiple":false,"regex":""},{"id":"Title","parentSelectors":["Scroll"],"type":"SelectorText","selector":"h1","multiple":false,"regex":""},{"id":"Views","parentSelectors":["Scroll"],"type":"SelectorText","selector":".article-main__content__counter span","multiple":false,"regex":""},{"id":"Body","parentSelectors":["Scroll"],"type":"SelectorText","selector":".article-main__content__wysiwyg p","multiple":true,"regex":""}]}
#2

I gathered all unique article links, so you can walk through all of them...
source file

#3

THank you. PLease help me to understand how to program this, especially how to use scrolling correctly. How to measure the amount of scrolling to reach the button, not the bottom of the page? I programed that in UI.Vision, but can not program algorithm of articls retrieaval.

#4

I didn't set scrolling correctly eaither. I just scroll manually grabbed all article links, so you can visit each lick to scrape all you need.

#5

Hi,

The pagination mechanics on this website are a bit weird. You can try this setup:

{"_id":"LastPAginSchool","startUrl":["https://www.theschooloflife.com/articles/"],"selectors":[{"clickActionType":"real","clickElementSelector":"button.load-more-articles","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":2000,"discardInitialElements":"do-not-discard","id":"load-more","multiple":true,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"elementLimit":0,"id":"Scroll","multiple":true,"parentSelectors":["load-more"],"scroll":true,"selector":".articles-with-filters .column","type":"SelectorElement"},{"id":"Date","multiple":false,"parentSelectors":["link"],"regex":"","selector":"time","type":"SelectorText"},{"id":"Title","multiple":false,"parentSelectors":["link"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"Views","multiple":false,"parentSelectors":["link"],"regex":"","selector":".article-main__content__counter span","type":"SelectorText"},{"id":"Body","multiple":true,"parentSelectors":["link"],"regex":"","selector":".article-main__content__wysiwyg p","type":"SelectorText"},{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":"\t.article-block__content__title a","type":"SelectorLink"}]}
#6

Thank you very much. Can you please advise some documentation on scrolling in WebScraper?

#7

The basic concept is described here Scroll selector | Web Scraper How To

However, in real life, there are countless nuances that would have to be considered as each website has slightly different mechanics.