Help with Octopus Energy

Hi guys,

I am trying to scrape this site (Fine-tune your estimate | Solar Panels | Octopus Energy) such that I end up with an excel where for each # of solar panels (integer values from 2 to 20), I have the estimated cost for each battery capacity (none to 13.5kwh) and also if the EV charger is included.

For more clarity, the end data should look something like...

Apologies if this is super simple, but I have exhausted my (limited) scraping skills. The main challenges I am having are:

  1. The website starts at 10 panels, and when i used pagination to change the # of panels, it only goes in one direction i.e., from 10 to 20, when i want it to go from 2 to 20 (which is the full range offered)

  2. I am getting odd results when i try and push beyond just gathering # panels and price estimates i.e., when i try and incorporate the EV charger being included, or different battery capacity

Huge thanks in advance for any help, and please let me know if i can help you help me!

Hi,

You can try this one:

{"_id":"octopus-energy","startUrl":["https://octopus.energy/order/solar/estimate/"],"selectors":[{"id":"price","multiple":false,"parentSelectors":["EV-charger-click"],"regex":"","selector":"h2:contains('Your estimate') + div p","type":"SelectorText"},{"clickActionType":"real","clickElementSelector":"input[name='evCharger']","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"discard-when-click-element-exists","id":"EV-charger-click","multiple":true,"parentSelectors":["Battery-click"],"selector":"_parent_","type":"SelectorElementClick"},{"id":"EV charger","multiple":false,"parentSelectors":["EV-charger-click"],"regex":"","selector":"[name=\"evCharger\"]:checked ++","type":"SelectorText"},{"id":"panels","multiple":false,"parentSelectors":["EV-charger-click"],"regex":"","selector":"div:has(> h1)","type":"SelectorText"},{"id":"pagination-up","paginationType":"clickMore","parentSelectors":["_root","pagination-up"],"selector":"button:nth-of-type(2) h1","type":"SelectorPagination"},{"clickActionType":"real","clickElementSelector":"input[name='battery']","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"discard-when-click-element-exists","id":"Battery-click","multiple":true,"parentSelectors":["pagination-up"],"selector":"body","type":"SelectorElementClick"},{"id":"Battery","multiple":false,"parentSelectors":["EV-charger-click"],"regex":"","selector":"[name=\"battery\"]:checked ++","type":"SelectorText"}]}

Start the scraping with a little longer delay (5000 for example) and in the pop-up window manually click the nr. of panels down to 2.