Ability to modify hidden form variables when submitting URL

I have come across instances where pagination is controlled by hidden form variants sent along with the request (and not part of the URL)

To illustrate an example of how the form variables are used,
NO_OF_RESULTS - Represents max. no. of results that can be displayed on a page
CURRENT_PAGE_NO - Displays the current page
(The website uses an implicit sort order for the results)

So what happens is (say there are 400 entries in the database)

  • Clicking page 1 - shows you the first 25 results (if NO_OF_RESULTS = 25 and CURRENT_PAGE_NO = 1)
  • Clicking page 1 - shows you the next 25 results (if NO_OF_RESULTS = 25 and CURRENT_PAGE_NO = 2)

So, while pagination by URL won't work, it is possible to modify the hidden form variable when making the URL request, to retrieve all results (by setting the NO_OF_RESULTS to 500 .. ) and the iterating through retrieved keys or links ... and then scraping the details for each.

It'd be great to have this feature