Stop scraping and still get the data?

How to?
#1

I am scraping using a "Element click" selector.

For click type I used "click more" to scrape customer reviews and go to next review page automatically.

The problem is that there are 11000 review pages.

I have to choose between scraping only one page ("click once" option) or scraping ALL the reviews ("click more" option).

I found no way to stop the scraping after 200 clicks for example, or to pull the data during scraping to be able to export it after 200 pages.

How can I do?

Here is the sitemap :

{"_id":"LMNT","startUrl":["https://drinklmnt.com/products/lmnt-recharge-electrolyte-drink?variant=16358367199266"],"selectors":[{"id":"Selector","parentSelectors":["_root"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"div.cursor-pointer:nth-of-type(7)","clickElementUniquenessType":"uniqueText","clickType":"clickMore","delay":2000,"discardInitialElements":"do-not-discard","multiple":true,"selector":"div.review-body"},{"id":"Text","parentSelectors":["Selector"],"type":"SelectorText","selector":"_parent_","multiple":false,"regex":""}]}

#2

Try the sitemap below which will stop at page 11.
The click selector to do this is fairly complex, but you only need to worry about this section:
not(:contains('<910111213>'))

It means "keep on clicking right arrow until you see this number sequence". So for page 11, the sequence is 9 10 11 12 13 (with no space). Change only the numbers; do not edit or delete any of the surrounding punctuation. To stop at page 200, you would need to experiment to find out what sequence would appear then.

image
image939×600 22.9 KB

In this example, I made it stop at page 11 and that is what the number sequence looks like:

image
image1130×669 68.8 KB

You may also need to increase the Delay (ms) if the site is slow in loading more reviews.

Revised sitemap:

{"_id":"drinklmnt","startUrl":["https://drinklmnt.com/products/lmnt-recharge-electrolyte-drink?variant=16358367199266"],"selectors":[{"id":"Title","parentSelectors":["_root"],"type":"SelectorText","selector":"div > h1","multiple":false,"regex":""},{"id":"Reviews section","parentSelectors":["_root","Next page with limit"],"type":"SelectorElement","selector":"div.reviews","multiple":false},{"id":"Review rows","parentSelectors":["Reviews section"],"type":"SelectorElement","selector":"div.review","multiple":true},{"id":"Review title","parentSelectors":["Review rows"],"type":"SelectorText","selector":"div.review-title","multiple":false,"regex":""},{"id":"Review text","parentSelectors":["Review rows"],"type":"SelectorText","selector":"div.review-body","multiple":false,"regex":""},{"id":"Reviewer","parentSelectors":["Review rows"],"type":"SelectorText","selector":"div.review-author","multiple":false,"regex":""},{"id":"Next page with limit","parentSelectors":["_root"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"div div.text-center:not(:contains('<910111213>')) > div.cursor-pointer:nth-of-type(7)","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":1800,"discardInitialElements":"discard-when-click-element-exists","multiple":false,"selector":"div.bg-gradient-to-b"}]}