Website returns 500 error in html form while scraping it for learning purpose

#1

I'm trying to scrape a website which uses viewstate, eventvalidation and viewstategenerator in its payload for searchpage post request.

Code:

  val homepageDoc = Jsoup.connect("abc.com").timeout(0).get()

  val cookies = Jsoup.connect("abc.com").execute().cookies()

  println("Cookies: " + cookies)

  val viewState = homepageDoc.select("input[name='__VIEWSTATE']").attr("value")
  val viewStateGenerator = homepageDoc.select("input[name='__VIEWSTATEGENERATOR']").attr("value")
  val eventValidation = homepageDoc.select("input[name='__EVENTVALIDATION']").attr("value")

  println("View State: " + viewState)
  println("View State Generator: " + viewStateGenerator)
  println("Event Validation: " + eventValidation)

  val searchpageDoc = Jsoup.connect(url).timeout(0)
    .data("ctl00$ctl09", "ctl00$MainContent$UpdatePanel1|ctl00$MainContent$butt_Search")
    .data("__EVENTTARGET", "")
    .data("__EVENTARGUMENT", "")
    .data("__VIEWSTATE", viewState)
    .data("__VIEWSTATEGENERATOR", viewStateGenerator)
    .data("__EVENTVALIDATION", eventValidation)
    .data("ctl00$MainContent$txt_SearchName", "aa")
    .data("ctl00$MainContent$dd_SearchCounty", "0")
    .data("__ASYNCPOST", "true")
    .data("ctl00$MainContent$butt_Search", "Search")
    .cookies(cookies)
    .post()

  println("Search Page Document: " + searchpageDoc)

Output:

Cookies: {ASP.NET_SessionId=pgbhlui3nsfmlmjuf1uyxjcp, __AntiXsrfToken=be66fdf80c16479f911e3257ec5a53e1}
View State: 1IbN0ppapDzTBLZUJOXQKq+dNwv9uvZkDht+N7nMqjVIR8j2qq4M00JwpzwgDFBd9Rg1Y2qh8OCT0FeVD9pQEXDiMC4aOIE+JqCztnlj7fg5+1nEGgTbdHYf1qiWUpf5PsrxC2tb1r8CT1hfb7zaf7vyrI1G7BNHWtzh6EfVp5t01qOlGWO5Z4rd82uPD5CY+Kmd/IF2nzngFTNhl5lEIXsTXYZnx1EpEldP3KbAUc8I5XtGfe6wDMf2F2McGHXJXNLZmL7i3/giwFRxQpMNbjYQijmc5B4DKJfGLr9p0tiTinPogU1H76bDhgYmY2lNA8HZaf176xVWriVwDQ9r5TvwGS76df8X71PHMtuWalvtAWYMfu5ovBsPw2bmkGbm0DknNA3/7aYUSMEj5za0itd7LV10zecCSDkksAMa7+0snI+wTUHghkWCH3kwkfpQJETlatN8FvuTrw/T6FtktxlxxYYxTXnH9Kwgj0wPS/4MGOi4tV2g7UIlRiV2PncOSmtGYQ58+YVCWhT3cfSmSdR7ygAchPcoueRlLtDZH+SLqlyQRH2FmOJo/C5c9S8Alru+/sDB6cSs0ThfyryAHEbYzw/lrbjHerjrPJdNWMFiqnmSjGKBrRegdvaxYswIDtlUjdtr83U5PbqdqcJ7XcRZH05KmcJuWQ0fv3xN+Pyt4boS+w/RqU3s3DbECq/HOn9ul3kbjdY5XX52Q/vM3GEpiFY7G2SzcKIDlOGMkK14/ypUIfA9tEoWhHIXrk/g4ZNLrcyyDa8UbmougoAg+vVtlmi6HHoh5qoKCTUOab3ZspKJh2nKIkk3CvO9J2x2UyV2UNjJXIFqwOSm57RkN8gw7N9VJ3t8YNCm6l8PMI+1pcLkLvshjCTjAMZfnklmvvJsBTaqpA/pdt1Lq+t8zXKXd99s6xPBS3abBHjuSc3XbPJfqPzZBzeWbakx5a+QfiLb1Ov0CFxAUX0siq/sHCgLayo1saV5w9KAbJ+BLJxDq3cm2GaiHZ4kOfkrsLSXd1sazpOvW8INpGRSI6jQAtjSGnO2s5CtmB9XtsrnQ+0WqdzGnlX497GX8KcbPgqjbW3FlcvMTwNtPXlmrfenluPtXjbB5GnBSGSixJTQSNLfvRMH767L42IDf8IjPCRNBXg1/TRe9lLi17JrNElyqk0wflA7y8HYD0H/oDO51BUgNuX6lUp1ItBr1k0NUltqImzKRvTyZlYwwvaWo6bkK2h39o5+2SH5yJiRwG7qU94LSwBNbn8I4bD52lNAy50b3/v4kZvXbsjwWhIp8mr6mR8dAkJiio1A9FlTBFFCbIqv8HNFwK8iY3eqUMXwr1nYViatOsGAaOYRwQQbpMZW7I3N1Lea/ADC/IFVMPV6mobdnO5K5wFXc6LjvVZizqiWwrLp34WKh9G/eIYRA4HLgmNft7f39eY/eUzfgUvrZQrScKBaYDNQhLAwn94U5loyHXJ51VAfLLLRC8bF8Er1TZmJfIq1VtoitDinqS6+BVSBZIyIsc66QmvIHrP+t1her0Um5CUzaVSC3NeiTsTPnj+ji4IGd+NsNz2Y1TgbjvvYXQxlttV/cwDkNNGLF5FdtyU1H8uOOVApHww7VqcS5eJE1a278S2bchkh+oMbyzDxRp3xNn35c5mSuPZzFpxYRT71SRf31vPOqgFL4pbQHNzUSUOgM/5Y03/KbBeuBBO2PoiIbPZPFWr1sQWn+s3m16ePRfyhL9A8u52H1AbNv4jPStV5TAYvDmh8FSJfYRokvByRCN6sIlUJ47VcmDl/m9SvhlT4XVXAdOqY4ALxANb2CmaFn5je29fzbwETZgkS1EQ4YqSnscQ198mMak7IeurbDDoZ3MIGeN7Ec6wfCvMfgDEZ0gegwCZ/e7i9ssEP2/e4OwuVy410huphOq1Nvdwm/5zI+TL6urQwY5LcvU1vL8K2j93unwnMhT/u/JH/ksJLIGEtE+CB+lGqPoHQbIK6o/PWCaUPd9UQLJCZGyGzZKNIidDz/vdwXhltCSupPsrmOWOFTJIOGR/tKCdBbk+yH8BaVW6q//4JC/zy6tr5B7C7HqWl7fR+cm5+RsCtrf492zXgmhhxuH73mPykPDxAfkH3LTQq7QSxNfGl6r3SkFUcUZY6CohGVPt4vaTdEiLTg2f/c/zA+7MQ4pDUypDCdwuuvOTnStOzuKCT25hed2SpP/Dm8qEulgo9gfM2cXwU7EZM6HkrOX8Lsa8m/znWxr8dHhsMHIISQ2UyF6GI9hWlVAZsqSnKV2/Xu17Ek2+j7azwUpHTIIhrgjnmpMIaxJHL+nduHJLZALFh/B/WzT3ZaObYDvoGbhCEU3pKky/oHppnzzfPqsSTUENL+RkwKqDHu88AZkylaw80VCI9Wugg5idf9vhwehLNjFpl1ObCKZMV5yVLchn1/vzgSgbqaE2XhMuaUYDGBSvxsydU2dzpg6WJifpJhhPAM5qlkMQzG+GgIaE6BWQpFKkWRYoVi+ytKbomYD1p27ZIHQkLpAwswuzEleSaOZNh2Q/ILI5iPeb57EiovcUHc7s65wVmuTltcvI+vLpAcvQXm7l5L5tcDjBG0Xxo4s7aXeB6G1JuMAWDJ0A/nws0zjyvpjZ9ozGjwWG7vdWgCnnTGPD9HE/z4KqKkNwJOIomb6HfGvNRqNOV6iKwOaszdzbYStizjUHlgdoYplXUYPnuHhqZMJG+suIxaVuRWtu0UNiRKqbSr4VWHyBoSLvwSLJvgYqNqfom3TC2OPGTNG1fAl+YoYWtuOil5En/cl2L5pNbZjCg3Mg7yKNtUJa8FFKhiGv7W42ZxDqiYCg4eqAeyw1eDG3oPoZ2vkNbSzNlwKG+zLslbSu2JyFyScEWCROtALOL1pOOj6clDGr2cXaDOdi4h3TUl7bPKrDL4PODNLX236RqVSuHJAA02NJkZ2MQIaL4Sih07/njia1GPdu0GFS25mySpnJmCNm29zijhg3vaBIdPiXVX8l86OrlluzYkYWOM3KHQQz9zIePt6xO/x/kVlaBKLhL9sJUA7shx6vEW1sf4/Xh1WCVtPGC7ZsqC/po7hm+r6s4qOzsRaPoV6HV+VnpSJxfmA6K9xcsPFEnzyr1IH3/2UvEglkR21es2lwdeGCKNdObsa8NO44YOO7gpst6qCqWvCub9fEtu8VAo4n+Fzu0GsbhyT0QmoZ2FR2DCFpJAojnRMXzSdVr+WHwPk7mVXAt6L8WzBLhLYvkWQiKGs0/O1fEMSmTWw9OHbjUZ2iHKGFZZ0lp7dR8tV2m3S99ekOmkMpDeTVn79gbjRQtSKcQnOpEjFqfAVidZpSVJq0tPaB7YRlC1q5gYKJTMQx3qGYpWqqYc6hOia+GlMWYCCH/N3SX3M4koVgtcjSYprqGu88SOiKCXr6XYx+h9qbh+LvjnFfwbk7Gcsmnevozq5ac8aLgdmsN5kHKrSXDOtBXKsZQLAqkBUkPVA==
View State Generator: E42FE49D
Event Validation: tojLErdTZW8zFWrJHRIpsbtf3XZmYANPNt01mOFfxjp8VT2J7Fcsk9Tx6BPOMbWrUKIuVdWIe+lpP7Nakc9CPBC8prdE01+PnBBcT6uPjTbJj4fIAzt9SC2OWc/Yh9vTuHmCm5pSk3yctzMFBd79naxpY/5V1Q8d+5/JWKIFWJAWSFLKfi5aFFaYW6bttpF2bDyQatq+BXFK2P4ESkAf9iO4pLOYorz5s2cL1TZAvCmx+2E5YFkRAH8joDS419agjCnZYYIHQTVGVSq/kdJrhIG3jWLyMq7fqDqvsRrjMyH2GfKDTRHLdKTC32z9yPN1JY9CYE9qJoIio5x1kfBuE89ybOTgmi4vb+byXRFfLB/+zXsOB0Wfs1S4T5LXXRngypmGNA34qpdviKDARwQ9zJGehpCpS2KU/xYDQV6PBdnm10JA7QDrWYykwgGLfuMmfOMi51ZTVrGmg2Bm8l+ikhhwYdt0dxixL9oSWUB2LKIxWFMp7uFspF7gjBNYTs1QU59l/JBfGZhdKpTpEN759U2T5fGdsE4SbUODOu+2mRThMYvCFyzB7aTH+jQGkzGHPcZNbH2G2rNrcnmc63d2mSDydrU0Kq2LybrlFxThP18CN3PmiUQedHgkGVk5NABuSywzsdpDqMDNVaoriTybbBceVaSPIGCt3Q4XAOUdhlsF6ILYrGrwXlXGKd6sqtvhi+OtDqHJZWrP9E7ei4UhNsKQNQap2QlAL6gs+hScSzx31+HvwlcdFbc+INALRGdo8tHHBkRprHb52VPxylfONh9scwELjrmGSaYzKbIb4D7ujZBDqtfnjB2e6v7vWQQ1s0nS2fURvfuQJKJP8haMCO7luYlwUnTquwSD2FMNf7Ov+cIA0I8pgwu4MqpYUyHfoatYth55Ma9TQfUupJFuQ5DmEMAyhJfmRXsnEnmyfMxrtnk6NtmaxAZaAriV7RxYBBxDvyKYNEQAsBjt7g2/PgCzc2CBjpAVsagmqVx9ZSUaSIoyJhkp9s4U+gXWzzSa0eIETX7duIKfHGoHas1/T7d1Y9faOi0qdPdrLjTCa3Fafy7aPQ+TL0uGMikvFE2+ryxD5M3BNJJT2OL01/nGv+WWJK3shwnsvUMNYtunTEFK/dkZHH69RnyV9URktuFH
Search Page Document: <html>
 <head></head>
 <body>
  0|error|500||
 </body>
</html>

I've tried this, took viewstate, and other from homepage and set it on the searchpage request. I've also set cookies for searchpage but still its not giving required data. I've tried with JsoupBrowser as well but it is giving me same error.

Guide me what I'm doing wrong?

#2

This is probably not the right forum for your product. But anyway, a 500 error usually means a general problem with the website's server. There's probably an issue or temporary glitch with the website's programming. So not that much you can do on your end.

1 Like