Pagination works but does not scrape data (auto mode) or scrapes data only on the active page (when other pagination types selected

How to?
#1

Hello,

I use pagination (beta) together with a link which scrapes texts in the target file. Pagination is based on next button.

Pagination works smoothly in auto mode but does not open addressed pages.

If i choose one of the other pagination types it works only in the active page, it opens links and scrapes data.

Here is the sitemap

{"_id":"mawa","startUrl":["https://www.mawa.om/en/rent"],"selectors":[{"id":"pag","paginationType":"auto","parentSelectors":["_root","pag"],"selector":"li:nth-of-type(9) a.page-link","type":"SelectorPagination"},{"delay":0,"id":"link","multiple":true,"parentSelectors":["pag"],"selector":"#PropertyList .inner_feature_cntnt_heading a","type":"SelectorLink"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["link"],"regex":"","selector":".inner_wrapper h4","type":"SelectorText"},{"delay":0,"id":"sqm","multiple":false,"parentSelectors":["link"],"regex":"","selector":".inner_wrapper p","type":"SelectorText"},{"delay":0,"id":"loc","multiple":false,"parentSelectors":["link"],"regex":"","selector":".detail-wrapper-body p","type":"SelectorText"}]}

#2

@emrekko Hi, it appears that in some cases the 'pagination' stops prematurely because the next page does not load in time. Try to update the 'pagination' using an 'Element click' selector with the delay set to at least 3'000 - 5'000. Also, the results will start to appear once the scraper has gone through all of the pagination pages.

#3

Thank you very much, but unfortunately it did not work. Firstly i cannot retrieve the price data in the element, secondly element click does not paginate, stopped in the active page.

{"_id":"mawa4","startUrl":["https://www.mawa.om/en/rent"],"selectors":[{"id":"element","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":"li:nth-of-type(9) a.page-link","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":5000,"discardInitialElements":"do-not-discard","multiple":true,"selector":"#PropertyList div.inner_feature_cntnt"},{"id":"title","parentSelectors":["element"],"type":"SelectorText","selector":"h3","multiple":false,"delay":0,"regex":""},{"id":"loc","parentSelectors":["element"],"type":"SelectorText","selector":".view_map_flag p","multiple":false,"delay":0,"regex":""},{"id":"type","parentSelectors":["element"],"type":"SelectorText","selector":".square > p","multiple":false,"delay":0,"regex":""},{"id":"bedroom","parentSelectors":["element"],"type":"SelectorText","selector":"li:nth-of-type(1) p","multiple":false,"delay":0,"regex":""}]}

#4

@emrekko Hi, that happens because the 'Click type' is set to 'Click once' instead of 'Click more' + you can also change the 'click element uniqueness' to 'unique CSS Selector' and set the 'Selector' to - #PropertyList div.feature_inner

Example:

{"_id":"mawa-om","startUrl":["https://www.mawa.om/en/rent"],"selectors":[{"clickElementSelector":"a.page-link:contains(\"Next\")","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":5000,"discardInitialElements":"do-not-discard","id":"element-click","multiple":true,"parentSelectors":["_root"],"selector":"#PropertyList div.feature_inner","type":"SelectorElementClick"},{"delay":0,"id":"title","multiple":false,"parentSelectors":["element-click"],"regex":"","selector":"h3","type":"SelectorText"},{"delay":0,"id":"loc","multiple":false,"parentSelectors":["element-click"],"regex":"","selector":".view_map_flag p","type":"SelectorText"},{"delay":0,"id":"type","multiple":false,"parentSelectors":["element-click"],"regex":"","selector":".square > p","type":"SelectorText"},{"delay":0,"id":"bedroom","multiple":false,"parentSelectors":["element-click"],"regex":"","selector":"li:nth-of-type(1) p","type":"SelectorText"},{"delay":0,"id":"price","multiple":false,"parentSelectors":["element-click"],"regex":"","selector":".price p","type":"SelectorText"}]}

#5

Fantastic! Thank you very much, well appreciated.