Overlay preventing element selection

How to?
#1

Hi everyone - the following website has sone sort of overlay that preventing me from selecting any elements.

The page itself seems to load everything from js

I can’t see the elements in devtools but can’t select any of them, even manually.

https://review.gerrithub.io/admin/groups/db3f6c4b11b81f1dda8e947511a5b2096d491f06,members

#2

Is this a shadow-root sort of thing? @KristapsWS

#3

@bretfeig Hi. Yes, this website contains multiple shadow-roots, therefore to access certain data points you will have to edit the code manually.

shadow-root
shadow-root1795×1018 188 KB

For example to access e-mails - #app:shadow-root #app-element:shadow-root #mainHeader + main > gr-admin-view:shadow-root gr-group-members:shadow-root table tr:contains("@")

Sitemap example - {"_id":"shadow-root","startUrl":["https://review.gerrithub.io/admin/groups/db3f6c4b11b81f1dda8e947511a5b2096d491f06,members"],"selectors":[{"id":"e-mails","type":"SelectorText","parentSelectors":["_root"],"selector":"#app:shadow-root #app-element:shadow-root #mainHeader + main > gr-admin-view:shadow-root gr-group-members:shadow-root table tr:contains(\"@\")","multiple":true,"regex":"","delay":0}]}

More information: CSS selector | Web Scraper Documentation

Hope this helps!

#4

Thank you that's awesome. Trying to translate that into the previous page that list all the groups. I would like to select every row and then click into each group, down into the members part and extract all names & e-mails.

Did you just follow the dom all the way down selecting each shadow-root ID or was/is there an easier way to getting the right selectors and building the map?

#5

@bretfeig You have to start with the very 1st and follow the path till you reach the desired data.

#6

What i do wrong when I`m trying to scrape count of security vendors flagged this IP address as malicious?

image
image914×1324 73.2 KB

vt-ui-shell:shadow-root #view-container > ip-address-view:shadow-root #report:shadow-root > vt-ui-detections-widget:shadow-root > div.positives

#7

@vd40 Hi, it happens because the shadow root of vt-ui-shell is not a direct parent of ip-address-view.

Here's a working example:

{"_id":"virustotal-com","startUrl":["https://www.virustotal.com/gui/ip-address/1.1.1.1/detection"],"selectors":[{"id":"count","multiple":false,"parentSelectors":["_root"],"regex":"","selector":"ip-address-view:shadow-root vt-ui-main-generic-report:shadow-root vt-ui-detections-widget:shadow-root .positives","type":"SelectorText"}]}

1 Like