Incorrect data when the quantity increases

Web Scraper version: 0.4.2
Chrome version: 83.0.4103.116
OS: Windows 10 64 Bit


{"_id":"propwala","startUrl":[""],"selectors":[{"id":"city click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"body","multiple":true,"delay":"2000","clickElementSelector":"select.ddlBox option[value=new_delhi],option[value=noida_uttar_pradesh],option[value=greater_noida_uttar_pradesh],option[value=gurgaon_haryana],option[value=faridabad_haryana],option[value=ghaziabad_uttar_pradesh],option[value=mumbai_maharashtra],option[value=thane_maharashtra],option[value=navi_mumbai_maharashtra],option[value=bangalore_karnataka],option[value=chennai_tamil_nadu],option[value=hyderabad_telangana],option[value=pune_maharashtra],option[value=kolkata_west_bengal],option[value=ahmedabad_gujarat],option[value=chandigarh_city],option[value=mohali_chandigarh],option[value=panchkula_chandigarh],option[value=zirakpur_chandigarh],option[value=north_goa],option[value=south_goa]","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"localities","type":"SelectorText","parentSelectors":["city click"],"selector":"select#searchLocalities option","multiple":true,"regex":"","delay":0},{"id":"city","type":"SelectorText","parentSelectors":["city click"],"selector":"select.ddlBox option[selected]","multiple":false,"regex":"","delay":0}]}


The website has a city and a location dropdown field. On selecting a city the values of the location dropdown changes. I am trying to dump this values in the form Cities(1st Column) and Locations(2nd Column). It works fine if I try it for like 2-3 cities but as soon as I increase the amount of data, web scrapper starts to show incorrect city names with cities. And If I run it for all the cities together than it doesn't scrape anything at all. I have tried to increase timeout aswell but that doesn't help.

Extra Info : The City field has around 620 values and the Location field has around 1500 values for each City.

Not really a bug but rather the site is using server-side redirects which WS and most scrapers will have trouble navigating.

It is still possible to get the info in two stages, where in stage 1, you get all the City strings from the main page, and you create a list of URLs from those strings.
Then in stage 2 you have a different sitemap which uses all those URLs created from stage 1 as Starturls and you can access the Localities.

You can get all the city strings with this selector:

Type: Element attribute
Selector: select[class*='CityDropdown'] option
Multiple: Yes (checked)
Attribute name: value

Then you build the city URLs with those strings. For example, the url for Noida, residential properties for sale, is: