How to click one option in a drop-down menu

How to?
#1

I am trying to scrape data from a page that only loads certain data based on which options are selected in the drop-down menus at the top. I have tried using element click, but I only seem to able to select different validation states (like select.ng-touched or select.ng-dirty). I want to be able to change the selections in the drop down box, then scrape the data that appears.

Here is an example page: https://stats.nba.com/player/203518/matchups/?Season=2018-19&SeasonType=Regular%20Season

Sitemap: {"_id":"matchup-scraper","startUrl":["https://www.nba.com/#/"],"selectors":[{"id":"stats","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.nba-nav__container--center-menu-item:nth-of-type(7) a","multiple":false,"delay":0},{"id":"players","type":"SelectorLink","parentSelectors":["stats"],"selector":"div.nav-inner__menu-item:nth-of-type(3) a","multiple":false,"delay":0},{"id":"player-index","type":"SelectorLink","parentSelectors":["players"],"selector":"a[title='View Player Index']","multiple":false,"delay":0},{"id":"select-player","type":"SelectorLink","parentSelectors":["player-index"],"selector":".players-list__name a","multiple":true,"delay":0},{"id":"matchups","type":"SelectorLink","parentSelectors":["select-player"],"selector":"a[data-type-beta='matchups']","multiple":false,"delay":0},{"id":"table","type":"SelectorText","parentSelectors":["matchups"],"selector":"td:nth-of-type(n+4)","multiple":true,"regex":"","delay":0},{"id":"touched","type":"SelectorElementClick","parentSelectors":["matchups"],"selector":"select.ng-touched","multiple":false,"delay":0,"clickElementSelector":"select.ng-touched","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"}]}

#2

Hi, this should help get you started. It will click on the Career option, wait a few secs, then scrape the Career Regular Season Stats table. I used Page load delay of 5000 for my test scrape:

{"_id":"nba_dropdown","startUrl":["https://stats.nba.com/player/203518/matchups/?Season=2018-19&SeasonType=Regular%20Season"],"selectors":[{"id":"click-career","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"[data-type-beta='career'] span","multiple":false,"delay":"4500","clickElementSelector":"[data-type-beta='career'] span","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"career_regular_season_table","type":"SelectorTable","parentSelectors":["_root"],"selector":"[rows='datasets[0].rows'] .nba-stat-table__overflow table","multiple":true,"columns":[{"header":"Season","name":"Season","extract":true},{"header":"TEAM","name":"TEAM","extract":true},{"header":"AGE","name":"AGE","extract":true},{"header":"GP","name":"GPvalue","extract":true},{"header":"GS","name":"GSvalue","extract":true},{"header":"MIN","name":"MIN","extract":true},{"header":"PTS","name":"PTS","extract":true},{"header":"FGM","name":"FGM","extract":true},{"header":"FGA","name":"FGA","extract":true},{"header":"FG%","name":"FG%","extract":true},{"header":"3PM","name":"3PM","extract":true},{"header":"3PA","name":"3PA","extract":true},{"header":"3P%","name":"3P%","extract":true},{"header":"FTM","name":"FTM","extract":true},{"header":"FTA","name":"FTA","extract":true},{"header":"FT%","name":"FT%","extract":true},{"header":"OREB","name":"OREB","extract":true},{"header":"DREB","name":"DREB","extract":true},{"header":"REB","name":"REB","extract":true},{"header":"AST","name":"AST","extract":true},{"header":"STL","name":"STL","extract":true},{"header":"BLK","name":"BLK","extract":true},{"header":"TOV","name":"TOV","extract":true},{"header":"PF","name":"PFvalue","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr, tfoot tr","tableHeaderRowSelector":"thead tr"},{"id":"get_table_title","type":"SelectorText","parentSelectors":["_root"],"selector":"[rows='datasets[0].rows'] div.nba-stat-table__caption","multiple":false,"regex":"","delay":0}]}

#3

Hi leemeng, thank you so much for your reply. Using the code you provided gets data from the "Profile" page, but I want to get data from the "Matchups" page. Here is the full JSON I have so far:

{"_id":"matchup-scraper","startUrl":["https://www.nba.com/#/"],"selectors":[{"id":"stats","type":"SelectorLink","parentSelectors":["_root"],"selector":"li.nba-nav__container--center-menu-item:nth-of-type(7) a","multiple":false,"delay":0},{"id":"players","type":"SelectorLink","parentSelectors":["stats"],"selector":"div.nav-inner__menu-item:nth-of-type(3) a","multiple":false,"delay":0},{"id":"player-index","type":"SelectorLink","parentSelectors":["players"],"selector":"a[title='View Player Index']","multiple":false,"delay":0},{"id":"select-player","type":"SelectorLink","parentSelectors":["player-index"],"selector":".players-list__name a","multiple":true,"delay":0},{"id":"matchups","type":"SelectorLink","parentSelectors":["select-player"],"selector":"a[data-type-beta='matchups']","multiple":false,"delay":0},{"id":"click-defense","type":"SelectorElementClick","parentSelectors":["matchups"],"selector":"[data-type-beta='defense'] 1 span","multiple":false,"delay":"2000","clickElementSelector":"[data-type-beta='defense'] span","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"defense_table","type":"SelectorTable","parentSelectors":["click-defense"],"selector":"[rows='datasets[0].rows'] .nba-stat-table__overflow table","multiple":true,"columns":[{"header":"MATCHUP","name":"MATCHUP","extract":true},{"header":"TEAM","name":"TEAM","extract":true},{"header":"GAMES","name":"GAMES","extract":true},{"header":"POSS","name":"POSS","extract":true},{"header":"PLAYER PTS","name":"PLAYER PTS","extract":true},{"header":"PLAYER PTS DIFF","name":"PLAYER PTS DIFF","extract":true},{"header":"TEAM PTS","name":"TEAM PTS","extract":true},{"header":"TEAM PTS DIFF","name":"TEAM PTS DIFF","extract":true},{"header":"AST","name":"AST","extract":true},{"header":"TOV","name":"TOV","extract":true},{"header":"BLK","name":"BLK","extract":true},{"header":"HELP BLK","name":"HELP BLK","extract":true},{"header":"HELP BLK REC","name":"HELP BLK REC","extract":true},{"header":"FGM","name":"FGM","extract":true},{"header":"FGA","name":"FGA","extract":true},{"header":"DET FACTOR","name":"DET FACTOR","extract":true},{"header":"FG%","name":"FG%","extract":true},{"header":"3PM","name":"3PM","extract":true},{"header":"3PA","name":"3PA","extract":true},{"header":"3P%","name":"3P%","extract":true},{"header":"FTM","name":"FTM","extract":true},{"header":"SFL","name":"SFL","extract":true},{"header":"DEF FLS","name":"DEF FLS","extract":true},{"header":"OFF FLS","name":"OFF FLS","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr, tfoot tr","tableHeaderRowSelector":"thead tr"},{"id":"get_table_title","type":"SelectorText","parentSelectors":["_root"],"selector":"[rows='datasets[0].rows'] div.nba-stat-table__caption","multiple":false,"regex":"","delay":0}]}

So, starting from the nba.com homepage, I have the scraper navigating to each player's "Matchups" page. Where my code breaks down in is getting the scraper to select "Defense" in the drop-down menu, and then to scrape the data in the tables. Can you provide any additional assistance? Thank you so much for your expertise!

#4

No prob, glad to help. You can try this scrape which starts off at Will Barton's matchups page, chooses Defence, scrolls to the bottom, then grabs the table.

{"_id":"nba_test_will_barton","startUrl":["https://stats.nba.com/player/203115/matchups/"],"selectors":[{"id":"click_defence","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"select.ng-touched","multiple":false,"delay":"5000","clickElementSelector":"main > div.stats-container__inner > div > div > div.row > div > div > div > div.row.row5.collapse.stats-filters-top > div:nth-child(4) > div > div > label > select > option:nth-child(2)","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTMLText"},{"id":"scroll_Down","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".stats-table-pagination__inner--bottom div","multiple":false,"delay":"2000"},{"id":"get_table","type":"SelectorTable","parentSelectors":["_root"],"selector":".nba-stat-table__overflow table","multiple":true,"columns":[{"header":"MATCHUP","name":"MATCHUP","extract":true},{"header":"TEAM","name":"TEAM","extract":true},{"header":"GP","name":"GPgames played","extract":true},{"header":"POSS","name":"POSS","extract":true},{"header":"PLAYERPTS","name":"PLAYERPTS","extract":true},{"header":"PLAYERPTSDIFF","name":"PLAYERPTSDIFF","extract":true},{"header":"TEAMPTS","name":"TEAMPTS","extract":true},{"header":"TEAMPTSDIFF","name":"TEAMPTSDIFF","extract":true},{"header":"AST","name":"AST","extract":true},{"header":"TOV","name":"TOV","extract":true},{"header":"BLK","name":"BLK","extract":true},{"header":"HELPBLK","name":"HELPBLK","extract":true},{"header":"HELPBLKREC","name":"HELPBLKREC","extract":true},{"header":"FGM","name":"FGM","extract":true},{"header":"FGA","name":"FGA","extract":true},{"header":"DETFACTOR","name":"DETFACTOR","extract":true},{"header":"FG%","name":"FG%","extract":true},{"header":"3PM","name":"3PM","extract":true},{"header":"3PA","name":"3PA","extract":true},{"header":"3P%","name":"3P%","extract":true},{"header":"FTM","name":"FTM","extract":true},{"header":"SFL","name":"SFL","extract":true},{"header":"DEFFLS","name":"DEFFLS","extract":true},{"header":"OFFFLS","name":"OFFFLS","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr","tableHeaderRowSelector":"thead tr"}]}

BTW most of the top row dropdown options will change the URL, so that is another way to access them instead of trying to click downdowns. e.g. Will's "Defense" page can be reached directly at https://stats.nba.com/player/203115/matchups/?Season=2018-19&SeasonType=Playoffs&Matchup=Defense&sort=FG3A&dir=1

If you change the Season and/or Season Type, you'll notice how the URL changes too. URLs for the other players' will have that same pattern; only the six-digit player number will change, i.e. /player/xxxxxx/