Hello, my research calls for some heavy data but I am a novice web scraper. I am trying to scrape (w.log) four pre-entered strings in a form (data entries), nested deep within my web scrape**. I don't know how to write the code for these things, but the select tool is only getting me so far. Finding no guides in the forum or elsewhere, and assuming some beginner mistakes in my sitemap, I query ye. >>This<< is the closest query.
--- Root --- link 1a --- Link 1b --- #Problem obtaining data from entries here (see code below)
--- Data (Table selection) #This ok
--- Number of Table Elements (Text selection) #This ok
--- link 2a --- "repeat above data tree for this unique page" - **problem repeats**
--- link 3a --- "repeat above data tree for this unique page" - ...
...
--- link na --- "repeat above data tree for this unique page" - ...
...
I have some 300+ links off of the root I need to get structured for analysis.
My Sitemap:
{"_id":"scrape4","startUrl":["http://kombuchadaddy.com/admin/batches?order=id_desc&page=1"],"selectors":[{"id":"batch_link","type":"SelectorLink","parentSelectors":["_root"],"selector":"strong a","multiple":true,"delay":0},{"id":"Batch_Details","type":"SelectorLink","parentSelectors":["batch_link"],"selector":"span.action_item:nth-of-type(1) a","multiple":false,"delay":0},{"id":"Sample_Data","type":"SelectorTable","parentSelectors":["batch_link"],"selector":"div.column:nth-of-type(2) div.panel_contents > table","multiple":true,"columns":[{"header":"Date","name":"Date","extract":true},{"header":"Brew Day","name":"Brew Day","extract":true},{"header":"pH","name":"pH","extract":true},{"header":"specific_gravity","name":"specific_gravity","extract":true},{"header":"temperature","name":"temperature","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr","tableHeaderRowSelector":"thead tr"},{"id":"#Samples","type":"SelectorText","parentSelectors":["batch_link"],"selector":"div.columns:nth-of-type(2) div.column:nth-of-type(2) h3","multiple":false,"regex":"","delay":0},{"id":"Names","type":"SelectorGroup","parentSelectors":["Batch_Details"],"selector":"li.input.stringish label.label","delay":0,"extractAttribute":""},{"id":"Value","type":"SelectorHTML","parentSelectors":["Batch_Details"],"selector":"li#batch_number_input.number","multiple":false,"regex":"[0-9]+","delay":"1"}]}
[Side Question]
*Desired Datum Structure:
Element n / Type / First Day/ Last Day / Array of Data
The array is a Table associated with every link "a" with layout
{Day, Measurement1, Measurement2, Measurement3} for up to 20 days.
I understand this may make up to twenty unique data entries but I need them all associated to one root. Does this need to be done in post-processing of the data?
Can you suggest any post-processing software?
Finally, this is a sample of one of the nth "nb" pages I need to scrape - found by the second link selector under the root in my sitemap. I have edited out much of the sample and the website is behind our company login so I can't exactly post that for clarity.
Im seeking value="315", value="400.0" value="2019-01-07" & value="2019-01-18">
<div class="column" style="width: 49.0%; margin-right: 2%;">
<fieldset class="inputs"><legend><span>Basics</span></legend><ol><li class="number input optional numeric stringish" id="batch_number_input"><label class="label" for="batch_number">Number</label><input id="batch_number" name="batch[number]" readonly="readonly" step="any" style="opacity:0.5;" type="number" value="315" class="">
</li><li class="number input optional numeric stringish" id="batch_brew_size_input"><label class="label" for="batch_brew_size">Brew size</label><input id="batch_brew_size" name="batch[brew_size]" step="any" type="number" value="400.0" class="">
</li><li class="date_time_picker input optional stringish" id="batch_start_date_input"><label class="label">Start date</label><input class="date-time-picker" data-datepicker-options="{}" id="batch_start_date" maxlength="19" name="batch[start_date]" size="19" type="text" value="2019-01-07">
</li><li class="date_time_picker input optional stringish" id="batch_keg_date_input"><label class="label">Keg date</label><input class="date-time-picker" data-datepicker-options="{}" id="batch_keg_date" maxlength="19" name="batch[keg_date]" size="19" type="text" value="2019-01-18">
</li>
</ol></fieldset> </div>
If you can help with this you are amazing! I'm sure it is a novice question and in the linked forum topic up >>there<< someone suggested this can't be done. Halp?