Help with scraping Level.fyi

How to?
#1

Describe the problem.

Url: Open Compensation Data | Levels.fyi

A little assistance here, racking my brains to sort this out

If you see each row has a extension that drops down upon clicking on the element. I can't find a good element selector that encompasses both the main row and the drop down.

Ideally I need to account for pagination and have each row + it's corresponding drop-down treated as one row; which is why I'd normally do with Element selector.

If I tie Element Selector just to the row, when I go to the child selectors, the drop down can be triggered but is out bounds from the parent row selector..

Maybe it's a table selector I need?

image
image1200×627 49.9 KB

#2

I think pd.read_html does a pretty good job. I also parse the page source and add delimiters (eg '|' ) after the header of each cell by selecting the class name and appending a delimiter, so that it parses properly.

#3

@neel @bretfeig Hi, at first you should expand all of the table rows, and afterward, add the expanded element as a sub-wrapper.

{"_id":"levels-fyi","startUrl":["https://www.levels.fyi/t/software-engineer?country=254"],"selectors":[{"clickElementSelector":".undefined ","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"expand-wrapper","multiple":true,"parentSelectors":["_root"],"selector":"tr.undefined","type":"SelectorElementClick"},{"id":"main-wrapper","multiple":true,"parentSelectors":["_root"],"selector":"tr.undefined","type":"SelectorElement"},{"id":"company","multiple":false,"parentSelectors":["main-wrapper"],"regex":"","selector":"td:nth-of-type(1)","type":"SelectorText"},{"id":"level-name","multiple":false,"parentSelectors":["main-wrapper"],"regex":"","selector":"td:nth-of-type(2)","type":"SelectorText"},{"id":"years-of-experience","multiple":false,"parentSelectors":["main-wrapper"],"regex":"","selector":"td:nth-of-type(3)","type":"SelectorText"},{"id":"total-compensation","multiple":false,"parentSelectors":["main-wrapper"],"regex":"","selector":"td:nth-of-type(4)","type":"SelectorText"},{"id":"sub-wrapper","multiple":true,"parentSelectors":["main-wrapper"],"selector":"+ tr","type":"SelectorElement"},{"id":"details","multiple":false,"parentSelectors":["sub-wrapper"],"regex":"","selector":"div.MuiGrid-container","type":"SelectorText"}]}

1 Like