I'm currently working on scraping this website https://mad.coop.dk/dagligvarer. I wanna scrape all the items in the category -> subcategory. The page needs to be scrolled to the bottom to reveal all the elements. I got that working, check out the sitemap below. Some pages have a second page. The second page seems to only be able to refer to using element click. Is there a way to combine the scroll with 'element click' or another way to work around this?
I don't have a working code with pagination, but i'm thinking it's should be something like: Scroll -> Scrape Items -> Page -> Scroll -> Scrape Items.
Url: https://mad.coop.dk/dagligvarer
Sitemap:
{"_id":"coop_english","startUrl":["https://mad.coop.dk/dagligvarer"],"selectors":[{"id":"category","type":"SelectorLink","parentSelectors":["_root"],"selector":"h5 a","multiple":true,"delay":0},{"id":"sub_category","type":"SelectorLink","parentSelectors":["category"],"selector":"div.catalogProducts h5 a","multiple":true,"delay":0},{"id":"Scroll","type":"SelectorElementScroll","parentSelectors":["sub_category"],"selector":"div.background-container","multiple":false,"delay":"2000"},{"id":"Products","type":"SelectorElement","parentSelectors":["Scroll"],"selector":"div.product-inner","multiple":true,"delay":0},{"id":"Name","type":"SelectorText","parentSelectors":["Products"],"selector":"h5 a","multiple":false,"regex":"","delay":0},{"id":"Price_per_unit","type":"SelectorText","parentSelectors":["Products"],"selector":"span.price-unit","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["Products"],"selector":"span.price span","multiple":false,"regex":"","delay":0}]}