Hello everyone! Would really appreciate some help in fixing a scraper (first time building one scraper-n00b!)
I am using an element wrapper to select the rows, checking "multiple" and then I'm using text selectors to extract data from the child elements inside the rows. I tried element_scroll, tried a table, but it continues to pick up only the visible rows of the table. There has been one iteration where it seemed to have picked up the first row of several pages, but not until the end (not been able to replicate). When trying to scroll, I tested 50, 500, 1000 and 2000 ms wait times, same result.
Would really appreciate your help!
Url: https://techportal.epri.com/entities?researchImperative=Next-Generation%20Renewables
Sitemap:
{"_id":"epri_next_gen_renewables","startUrl":["https://techportal.epri.com/entities?researchImperative=Next-Generation%20Renewables"],"selectors":[{"id":"companies","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.Entities-table-row","multiple":true,"delay":"1000"},{"id":"company","type":"SelectorElement","parentSelectors":["companies"],"selector":"parent","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["company"],"selector":".Entities-table-link div","multiple":false,"regex":"","delay":0},{"id":"focus_area","type":"SelectorText","parentSelectors":["company"],"selector":"div:nth-of-type(3) div","multiple":false,"regex":"","delay":0},{"id":"tech_area","type":"SelectorText","parentSelectors":["company"],"selector":"div:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"descr","type":"SelectorText","parentSelectors":["company"],"selector":".Entities-table-cell-4 div","multiple":false,"regex":"","delay":0},{"id":"trl","type":"SelectorText","parentSelectors":["company"],"selector":"div.Entities-table-cell-1","multiple":false,"regex":"","delay":0}]}