[HELP] Pagination Scraping Issued [Doesn't show page no. tried with Element Click]

https://www.kvs-demo.com/latest-updates/
Please help me with this pagination scrape

{"_id":"kvsdemo","startUrl":["https://www.kvs-demo.com/latest-updates/"],"selectors":[{"id":"pagi111","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.margin-fix","multiple":true,"delay":2000,"clickElementSelector":".pagination-holder a","clickType":"clickOnce","discardInitialElements":"discard-when-click-element-exists","clickElementUniquenessType":"uniqueText"},{"id":"click","type":"SelectorLink","parentSelectors":["pagi111"],"selector":"a","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["click"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"duration","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"video:duration\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"image-src","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"og:image\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"embed-link","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"og:video\"]","multiple":false,"extractAttribute":"content","delay":0}]}

@ScrapeSite Hi.
Try this:

{"_id":"kvsdemo","startUrl":["https://www.kvs-demo.com/latest-updates/"],"selectors":[{"id":"wrapper","type":"SelectorElement","parentSelectors":["_root","pagination"],"selector":".margin-fix div.item","multiple":true,"delay":0},{"id":"click","type":"SelectorLink","parentSelectors":["wrapper"],"selector":"a","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["click"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"duration","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"video:duration\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"image-src","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"og:image\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"embed-link","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"og:video\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":".next a","multiple":true,"delay":0}]}

@ViestursWS How about limiting to visit certain pages say 1-5.. As it keep visiting the pages without getting the data being scraped.

@ScrapeSite Then you can manually change the start URL interval [1-5], it doesn't work on every website though, fortunately on this it works.
Like this:

{"_id":"kvsdemo","startUrl":["https://www.kvs-demo.com/latest-updates/[1-5]"],"selectors":[{"id":"wrapper","type":"SelectorElement","parentSelectors":["_root"],"selector":".margin-fix div.item","multiple":true,"delay":0},{"id":"click","type":"SelectorLink","parentSelectors":["wrapper"],"selector":"a","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["click"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"duration","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"video:duration\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"image-src","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"og:image\"]","multiple":false,"extractAttribute":"content","delay":0},{"id":"embed-link","type":"SelectorElementAttribute","parentSelectors":["click"],"selector":"meta[property=\"og:video\"]","multiple":false,"extractAttribute":"content","delay":0}]}

Start URL with Range definition: Web Scraper pagination tutorial - YouTube starts @ 0:37

Hope it helps.

Yeah I try that already.. the thing is it goes to only pages.. doesn't click and visit.. to collect data..

@ScrapeSite I just ran through it and got 120 out of 120 records.


1 Like