Scraping data from BSE India website

How to?
#1

Hi,

I am trying to scrape data from the Indian stock market website - BSE India. I want to scrape several values of various indices here (such as market value, PE ratio etc). The data of various indices is available at this link. basically, there are 56 indices available at this site (pls see the heat map boxes). I want to scrape data from each of theses indices/ boxes.I tried by creating several sitemaps but was able to scrape data of only 14-15 indices. What can I do to scrape the data from all the indices/boxes pls.

Url: LIVE Stock/Share Market | Indian Stock/Share Market LIVE | S&P BSE SENSEX | BSE (formerly Bombay Stock Exchange)

Sitemap:
{"_id":"third_try","startUrl":["https://www.bseindia.com/markets.html"],"selectors":[{"id":"Index","multiple":true,"parentSelectors":["_root","pages"],"selector":".active .boxred a","type":"SelectorLink"},{"id":"index","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"h2","type":"SelectorText"},{"id":"PE_ratio","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"td.ng-binding[align='left']","type":"SelectorText"},{"id":"PB_ratio","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"div.whitebox:nth-of-type(3) td.ng-binding:nth-of-type(2)","type":"SelectorText"},{"id":"Dividend Yield","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"div.whitebox:nth-of-type(3) td.ng-binding:nth-of-type(3)","type":"SelectorText"},{"id":"pages","multiple":true,"parentSelectors":["_root"],"selector":".carousel-indicators a:nth-of-type(n+2)","type":"SelectorLink"}]}

Looking forward to your kind response.

Thanks and regards,
Arvind Gupta

#2

@garvind25 Hi, it appears that all of the targeted elements are natively embedded into the HTML of this page, therefore the pagination selector is redundant.

Example:

{"_id":"third_try","startUrl":["https://www.bseindia.com/markets.html"],"selectors":[{"id":"Index","multiple":true,"parentSelectors":["_root"],"selector":".boxred a","type":"SelectorLink"},{"id":"index","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"h2","type":"SelectorText"},{"id":"PE_ratio","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"td.ng-binding[align='left']","type":"SelectorText"},{"id":"PB_ratio","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"div.whitebox:nth-of-type(3) td.ng-binding:nth-of-type(2)","type":"SelectorText"},{"id":"Dividend Yield","multiple":false,"parentSelectors":["Index"],"regex":"","selector":"div.whitebox:nth-of-type(3) td.ng-binding:nth-of-type(3)","type":"SelectorText"}]}

#3

@ViestursWS Thanks for the response. Yes, I was able to scrape all data.

Just wanted to understand that if I dont want to scrape the data from all the pages but only a few of them (say 7-10), how do I do it pls? As of now I am scraping data of all 56 indices at the site.

Thanks again,
Arvind Gupta.

#4

@garvind25 Hello, could you, please provide an example of such page/-s?