Pagination stops working after 3 pages

Dear Web Scraper Community

I'm trying to get a list of businesses to contact through this directory: IIABCal > Membership > Membership Directory > Find an Agent/Broker

The algorithm is not working as-is and something is going wrong after page 3 loads. I expected it to click the "Next" button but it quits.

I have tried the youtube tutorials, and read the pagination documentation. This website is a special case that was not covered.

Any help will be appreciated :slight_smile:

{"_id":"iiabcal","startUrl":["https://member.iiabcal.org/Membership/Membership-Directory/Find-an-Agent-Broker?search="],"selectors":[{"clickElementSelector":"input.btn-primary","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"Search","multiple":false,"parentSelectors":["_root"],"selector":"div.pane-contentpane","type":"SelectorElementClick"},{"delay":0,"id":"Card","multiple":true,"parentSelectors":["Search","Next"],"selector":"div.card-body","type":"SelectorElement"},{"id":"Next","paginationType":"auto","parentSelectors":["Search","Next"],"selector":"_parent_","type":"SelectorPagination"},{"delay":0,"id":"Name","multiple":false,"parentSelectors":["Card"],"regex":"","selector":"h5","type":"SelectorText"},{"delay":0,"id":"Address","multiple":false,"parentSelectors":["Card"],"regex":"","selector":"> p:nth-of-type(1)","type":"SelectorText"},{"delay":0,"id":"Phone","multiple":false,"parentSelectors":["Card"],"regex":"","selector":"span.d-block","type":"SelectorText"},{"delay":0,"id":"Website","multiple":false,"parentSelectors":["Card"],"regex":"","selector":"a:nth-of-type(2)","type":"SelectorText"}]}

I expected the Next button to be clicked over and over until there is no more "Next" button. The script ends after clicking "Next" only twice.

@csabour Hi, after performing tests using Web Scraper Cloud i successfully managed to scrape the desired data.


1 Like

Thanks I'll try using Web Scraper Cloud!

Are we running the same sitemap because I only got 60 records?
Thanks for your help.

I abandon this project.

@csabour Hi. Try to update your sitemap using the following example:

{"_id":"iiabcal","startUrl":["https://member.iiabcal.org/Membership/Membership-Directory/Find-an-Agent-Broker?search="],"selectors":[{"clickElementSelector":"html:not(:has(div.card)) input.btn-primary","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"Search","multiple":false,"parentSelectors":["_root"],"selector":"div.pane-contentpane","type":"SelectorElementClick"},{"id":"Next","paginationType":"auto","parentSelectors":["_root","Next"],"selector":"a.mx-next-hyperlink","type":"SelectorPagination"},{"delay":0,"id":"Name","multiple":false,"parentSelectors":["Wrapper"],"regex":"","selector":"h5","type":"SelectorText"},{"delay":0,"id":"Address","multiple":false,"parentSelectors":["Wrapper"],"regex":"","selector":".card-body > p:nth-of-type(1)","type":"SelectorText"},{"delay":0,"id":"Phone","multiple":false,"parentSelectors":["Wrapper"],"regex":"","selector":"span.d-block","type":"SelectorText"},{"delay":0,"id":"Website","multiple":false,"parentSelectors":["Wrapper"],"regex":"","selector":"a:nth-of-type(2)","type":"SelectorText"},{"delay":0,"id":"Wrapper","multiple":true,"parentSelectors":["Next"],"selector":"div.card","type":"SelectorElement"}]}