Pagination and element scroll

How to?
#1

Hello,

I am unable to figure out how to extract data from this website. To extract all elements I need element scrolling and there are multiple pages hence I need pagination and I am unable to figure out how to use both simultaneously. The data to be extracted is product name and price range. Thank you.

Url: https://www.johnlewis.com/browse/home-garden/cushions/_/N-cr9

Sitemap:
{"id":"jlp4","startUrl":["https://www.johnlewis.com/browse/home-garden/cushions//N-cr9"],"selectors":[{"id":"element scroll","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.GridColumn_col-sm-6__Ba6rz","multiple":true,"delay":2000},{"id":"Product Name","parentSelectors":["element scroll","pagination1"],"type":"SelectorLink","selector":"h2","multiple":false,"delay":0},{"id":"Name","parentSelectors":["Product Name"],"type":"SelectorText","selector":".show-on-desktop h1","multiple":false,"delay":0,"regex":""},{"id":"price","parentSelectors":["Product Name"],"type":"SelectorText","selector":".show-on-desktop p","multiple":false,"delay":0,"regex":""},{"id":"pagination1","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":".Pagination_c-pagination--header__QM1i5 a.Pagination_c-pagination__next-btn__3g_DG","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":2000,"discardInitialElements":"do-not-discard","multiple":true,"selector":"div.GridColumn_col-sm-6__Ba6rz"}]}

#2

@fabiha Hi. There are two ways to do it:

  • Using a predefined page range interval pagination in the start URL;

  • Using the pagination selector - ul[data-test="pagination-footer"] a[aria-label="Next"]set as a 'parent' for the scroll selector.

Example:

{"_id":"john-lewis","startUrl":["https://www.johnlewis.com/browse/home-garden/cushions/_/N-cr9"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"ul[data-test=\"pagination-footer\"] a[aria-label=\"Next\"]","type":"SelectorPagination"},{"delay":2000,"id":"scroll","multiple":true,"parentSelectors":["pagination"],"selector":"section[class*=\"product-card_c-product-card\"]","type":"SelectorElementScroll"},{"delay":0,"id":"Title","multiple":false,"parentSelectors":["scroll"],"regex":"","selector":"h2","type":"SelectorText"}]}

Helpful resources:

https://webscraper.io/how-to-video/pagination-next