Describe the problem.
Hey, having some more issues with pagination where the URL does not change. I am trying to scrape all 40 pages of Meijer locations (Name, Address) in Michigan.
Sitemap:
Would appreciate any help here. Thanks!
Firstly, you need a delay to you Click Selector, otherwise, almost certainly, will fail, as the page does not have enough time to render the new items.
Secondly, if you are selecting text from a contained Element area that only has the selectors that you are after, you do not need to have the 'Multiple' checkbox selected, otherwise, the data will have empty cells.
Thirdly, when using the Click Selector, you need to select the element, that triggers the next page, not click on the current page. For this reason the Click Selector doesn't paginate.
Here is a working version of the sitemap:
{"_id":"meijermichigan","startUrl":["https://www.meijer.com/custserv/locate_store_by_state.cmd?form_state=locateStoreByStateForm&state=Michigan"],"selectors":[{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".store","multiple":true,"delay":"2000","clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueCSSSelector"},{"id":"street","type":"SelectorText","parentSelectors":["pagination"],"selector":"div.storeAddress:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"citystatezip","type":"SelectorText","parentSelectors":["pagination"],"selector":"div.storeAddress:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"name","type":"SelectorText","parentSelectors":["pagination"],"selector":"div.storeName","multiple":false,"regex":"","delay":0}]}