Unable to get required data from paginated data while scraping a website for analysis

#1

I'm scraping a website for analysis purpose which uses request verification token in its payload. I'm using JsoupBrowser() (library: "net.ruippeixotog" %% "scala-scraper" % "3.0.0") and Scala (version 2.13.1).

First I send a Get Request to the homepage and got request verification token from home page.

Code:

val browser = JsoupBrowser()
  val homepageDoc = browser.get("example.com")

  val requestVerificationToken = homepageDoc >> element("input[name='__RequestVerificationToken']") >> attr("value")

Then using this requestVerificationToken I send request to homepage to get the data.

Code:

val formData = Map(
    "searchtype" -> "C",
    "__RequestVerificationToken" -> requestVerificationToken
  )

    val doc = browser.post(url, formData)

At this stage everything is working fine. But the data is paginated and there is no request verification token in this request's payload. Code for that is:

val nextPageFormData = Map(
  "undefined" -> "",
  "sortby" -> "",
  "stype" -> "a",
  "pidx" -> "2"
)

val doc = browser.post(url, nextPageFormData)

But this time it is not giving required data, instead it is giving some login form. I've set cookies with the request and tried to do same thing with cookies but this is also not working.

Guide me how can I do this, where everything is working except pagination.

Update:

I've tried to use http for above work.

Code:

implicit val actorSystem: ActorSystem = ActorSystem()
  implicit val executionContext: ExecutionContextExecutor = actorSystem.dispatcher

  val browser = JsoupBrowser()
  val homepageDoc = browser.get("example.com")

  val requestVerificationToken = homepageDoc >> element("input[name='__RequestVerificationToken']") >> attr("value")

  val url = "searchpage.com"
  val postBody = "undefined=&sortby=&stype=a&pidx=2"

  val request = HttpRequest(HttpMethods.POST, url, Seq(RawHeader("__requestverificationtoken",requestVerificationToken)), HttpEntity(ContentTypes.`application/x-www-form-urlencoded`, postBody))
  val responseFuture = Http(actorSystem).singleRequest(request)
  responseFuture
    .onComplete {
      case Success(res) => println(res)
        Unmarshal(res.entity.toStrict(5 seconds)).value.map { result =>
          val htmlStr = result.data.utf8String
          val browser = JsoupBrowser()
          val doc = browser.parseString(htmlStr)
          println(doc)
        }

      case Failure(e) => println(e)
    }

Output:

HttpResponse(302 Found,List(Cache-Control: private, Location: /BFS/Online/Account/Index?ReturnUrl=%2FBFS%2Fonline%2FNotarySearch%2FGetNotaryListWithOutCriteria, x-frame-options: SAMEORIGIN, x-frame-options: SAMEORIGIN, Access-Control-Allow-Origin: *, Date: Tue, 25 Jul 2023 12:42:00 GMT, Via: 1.1 dca1-bit16043, Strict-Transport-Security: max-age=63072000),HttpEntity.Strict(text/html; charset=UTF-8,214 bytes total),HttpProtocol(HTTP/1.1))
JsoupDocument(<html>
 <head>
  <title>Object moved</title>
 </head>
 <body>
  <h2>Object moved to <a href="/BFS/Online/Account/Index?ReturnUrl=%2FBFS%2Fonline%2FNotarySearch%2FGetNotaryListWithOutCriteria">here</a>.</h2>
 </body>
</html>)

It is giving 302 means it is not giving actual data instead it is moving to some other page.