Trying to scrap pricing info from cell phone provider

How to?
#1

Hello,

I am trying to scrap pricing and related plan (term) information for cell phones from a provider in Canada. I have started my search (my starting URL) right at the page where the devices are listed. It would seem as simple as creating a link selector that will click on each device, and then text selectors for title, price, and term, but I am not always receiving pricing or term information.

Am I missing something basic? TIA!

Url: https://www.bell.ca/Mobility/Smartphones_and_mobile_internet_devices2

Sitemap:
{"_id":"bell_mobility","startUrl":["https://www.bell.ca/Mobility/Smartphones_and_mobile_internet_devices"],"selectors":[{"id":"device","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.rsx-product-hotspot","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["device"],"selector":"h1.rsx-bell-font","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["device"],"selector":"div.rsx-price","multiple":true,"regex":"","delay":0},{"id":"term","type":"SelectorText","parentSelectors":["device"],"selector":"p.rsx-txt-size-18","multiple":true,"regex":"","delay":0}]}

#2

Hey, Adam,

Unfortunately, this site is quite complicated. If you check the links that didn't return valid results, you will see that there are different page structures with varying selectors scattered around. For you to be able to extract the correct information from the site, you will have to write custom selectors for each of the various pages dividing each selector with a comma and double-checking that the selectors do not overlap.

#4

Hello Adam,

I believe I have this working, by using a link selector to click on each phone, and then by using an element selector to grab the wrapper for the data points you want (set to multiple for each section of pricing, term, and details), and then using child text selectors to retrieve the data. See below:

{"_id":"bell_test_again","startUrl":["https://www.bell.ca/Mobility/Smartphones_and_mobile_internet_devices"],"selectors":[{"id":"price","type":"SelectorText","parentSelectors":["wrapper"],"selector":"div.rsx-price","multiple":false,"regex":"","delay":0},{"id":"term","type":"SelectorText","parentSelectors":["wrapper"],"selector":"p.rsx-txt-size-18","multiple":false,"regex":"","delay":0},{"id":"details","type":"SelectorText","parentSelectors":["wrapper"],"selector":"p.\31 rsx-note","multiple":false,"regex":"","delay":0},{"id":"wrapper","type":"SelectorElement","parentSelectors":["phone"],"selector":"div.bcx-order-now-box-inner","multiple":true,"delay":0},{"id":"phone","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.rsx-product-hotspot","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["phone"],"selector":"h1.rsx-bell-font","multiple":false,"regex":"","delay":0}]}

The one thing I haven't been able to figure out is how to select from the drop down for the memory variants on each device page, which should go before the element selector / child text selectors in the selector tree.

Possibly @webber knows how to approach this? I have seen drop downs navigated in other example site maps in this forum, but have not got it to work for myself yet.