Deal with the AJAX paginagion problem

Can anyone help me how to deal with the AJAX problem? It has perplex me for a long time, expecially these days I need to deal with some organizations data. Even though I set "click more", it just don't use and droped after getting one page data. Thank you all very much for pay attention to that!
The website is "http://mzj.beijing.gov.cn/wssbweb/wssb/dc/searchOrgList.do?action=searchOrgList"

And my sitemap is:
{"_id":"beijingcharity","startUrl":["http://mzj.beijing.gov.cn/wssbweb/wssb/dc/searchOrgList.do?action=searchOrgList"],"selectors":[{"id":"changepage","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"table:nth-of-type(3)","multiple":true,"delay":0,"clickElementSelector":"a:nth-of-type(3)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"url","type":"SelectorLink","parentSelectors":["changepage"],"selector":"tr:nth-of-type(n+2) a","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["url"],"selector":"table:nth-of-type(8) tr:nth-of-type(1) td","multiple":true,"regex":"","delay":0}]}

Hi there!

It can be easily set-up and running until you reach the last page. When you reach page number 733/733 the 'Next page' button will be still present on a page and WebScraper won't stop clicking it.

Therefore, you will need either to open Developer tab within Scrape window, and use Inspect tool to select the 'next page' button and delete it for WebScraper to successfully continue to navigate through table links, OR use Tampermonkey extension to delete it automatically after you reach last page.

1 Like