How to scrape data that requires scrolling down

How to?
#1

Describe the problem.
I'm trying to scraped data but it requires scrolling down. The issue I'm facing is that the data that doesn't requires scrolling down to load gets extracted but the data that requires scrolling down to load doesn't get scraped
The data that requires scrolling down to load would come under 'couplet-wrapper' which is the wrapper that captures the whole card, then couplet and tags are scraped together
Url: https://www.rekhta.org/poets?contentFilter=couplets&lang=ur

Sitemap:
{"_id":"couplet_data_urdu","startUrl":["https://www.rekhta.org/poets?contentFilter=couplets&lang=ur"],"selectors":[{"id":"pages","paginationType":"auto","parentSelectors":["_root","pages"],"selector":"div:nth-of-type(4) li.poetAlphLtr:nth-of-type(n) a","type":"SelectorPagination"},{"id":"poet-link","multiple":true,"parentSelectors":["pages"],"selector":".poetNameDatePlace a.clearfix","type":"SelectorLink"},{"id":"sher-link-bar","multiple":true,"parentSelectors":["poet-link"],"selector":"ul.searchCategory","type":"SelectorElement"},{"id":"sher-link","multiple":false,"parentSelectors":["sher-link-bar"],"selector":"a:contains('شعر')","type":"SelectorLink"},{"id":"couplet-wrapper","multiple":true,"parentSelectors":["sher-link"],"selector":"div.sherSection","type":"SelectorElement"},{"id":"Couplet","multiple":false,"parentSelectors":["couplet-wrapper"],"regex":"","selector":"div.c","type":"SelectorText"},{"id":"Tag","multiple":false,"parentSelectors":["couplet-wrapper"],"regex":"","selector":".tagHighlt a","type":"SelectorText"}]}

#2

hi @andrew48 did you try playing around with loading delay ? i.e load scrap delay load... ?

#3

Hi, if you're talking about page load delay, its's currently 2000 and I haven't changed it. What should I change it to?

#4

@ViestursWS can you please look into this?

#5

@andrew48 Hello, to load all of the available listings you will have to create the pagination and scroll down the loop.

Here's a sitemap example:

{"_id":"rekhta-test","startUrl":["https://www.rekhta.org/poets?contentFilter=couplets&lang=ur"],"selectors":[{"id":"pages","parentSelectors":["_root","pages"],"paginationType":"auto","selector":"div:nth-of-type(4) li.poetAlphLtr:nth-of-type(n) a","type":"SelectorPagination"},{"id":"poet-link","parentSelectors":["wrapper"],"type":"SelectorLink","selector":".poetNameDatePlace a.clearfix","multiple":false},{"id":"wrapper","parentSelectors":["pages"],"type":"SelectorElementScroll","selector":"div.poetColumn","multiple":true,"delay":4000,"elementLimit":0}]}