Wont recognize certain links

andrewdcf · May 9, 2022, 11:43pm

A bit new so sorry if this is basic.

I have no problem navigating from the root all the way to the "fights" tab of the "events" page. But then when it lists the fighters it won't click through to the individual "fighter" page even though it recognizes the link when I click "data preview."

Url: Fight Odds | MMA Betting Odds, News and Analysis

Sitemap:
{"_id":"ufc_camp_crawler","startUrl":["Fight Odds | MMA Betting Odds, News and Analysis a","type":"SelectorLink"},{"columns":[{"extract":false,"header":"Name","name":"Name"},{"extract":true,"header":"Charles Oliveira","name":"name"},{"extract":false,"header":"Height","name":"Height"},{"extract":true,"header":"5'11"","name":"height"}],"delay":0,"id":"fighter_name","multiple":false,"parentSelectors":["fighter_link"],"selector":"#full-width-tabpanel-\/fighters\/charles-oliveira-1239 div:nth-of-type(1) table","tableDataRowSelector":"tr:nth-of-type(1)","tableHeaderRowSelector":"tr:nth-of-type(1)","type":"SelectorTable"},{"columns":[{"extract":false,"header":"Birth Date","name":"Birth Date"},{"extract":false,"header":"1990-02-02","name":"1990-02-02"},{"extract":false,"header":"Reach","name":"Reach"},{"extract":true,"header":"73.0"","name":"arm_reach"}],"delay":0,"id":"fighter_arm_reach","multiple":false,"parentSelectors":["fighter_link"],"selector":"#full-width-tabpanel-\/fighters\/charles-oliveira-1239 div:nth-of-type(1) table","tableDataRowSelector":"tr:nth-of-type(3)","tableHeaderRowSelector":"tr:nth-of-type(3)","type":"SelectorTable"},{"columns":[{"extract":false,"header":"Fighting Style","name":"Fighting Style"},{"extract":true,"header":"BJJ Grappler / Striker","name":"Style"},{"extract":false,"header":"Stance","name":"Stance"},{"extract":true,"header":"Southpaw","name":"stance"}],"delay":0,"id":"fighter_style","multiple":false,"parentSelectors":["fighter_link"],"selector":"div.MuiPaper-root:nth-of-type(2) table","tableDataRowSelector":"tr:nth-of-type(1)","tableHeaderRowSelector":"tr:nth-of-type(1)","type":"SelectorTable"},{"columns":[{"extract":false,"header":"Ranks","name":"Ranks"},{"extract":true,"header":"Black BeltBJJ","name":"rank"},{"extract":false,"header":"Camp","name":"Camp"},{"extract":true,"header":"Bronx's Gold Team USAChute Boxe Diego LimaGold TeamGold Team Texas","name":"camp"}],"delay":0,"id":"fighter_camp","multiple":false,"parentSelectors":["fighter_link"],"selector":"div.MuiPaper-root:nth-of-type(2) table","tableDataRowSelector":"tr:nth-of-type(2)","tableHeaderRowSelector":"tr:nth-of-type(2)","type":"SelectorTable"}]}

ViestursWS · May 10, 2022, 2:26pm

@andrewdcf Hi, could you, please, re-paste your sitemap(by applying the 'Preformatted text' option) as the JSON of it seem to be invalid?

andrewdcf · May 10, 2022, 2:43pm

oops... here's a better version

{"_id":"ufc_camp_crawler","startUrl":["https://fightodds.io/recent-mma-events/ufc"],"selectors":[{"id":"event_link","parentSelectors":["_root","event_link"],"type":"SelectorLink","selector":"a.MuiLink-underlineNone.MuiTypography-colorPrimary","multiple":true,"delay":0},{"id":"fights_link","parentSelectors":["event_link","fights_link"],"type":"SelectorLink","selector":"a.MuiTab-root:nth-of-type(3)","multiple":false,"delay":0},{"id":"fighter_link","parentSelectors":["fights_link","fighter_link"],"type":"SelectorLink","selector":".jss6446 a","multiple":true,"delay":0},{"id":"fighter_name","parentSelectors":["fighter_link"],"type":"SelectorTable","delay":0,"multiple":false,"selector":"#full-width-tabpanel-\\/fighters\\/charles-oliveira-1239 div:nth-of-type(1) table","tableDataRowSelector":"tr:nth-of-type(1)","tableHeaderRowSelector":"tr:nth-of-type(1)","columns":[{"extract":false,"header":"Name","name":"Name"},{"extract":true,"header":"Charles Oliveira","name":"name"},{"extract":false,"header":"Height","name":"Height"},{"extract":true,"header":"5'11\"","name":"height"}]},{"id":"fighter_arm_reach","parentSelectors":["fighter_link"],"type":"SelectorTable","delay":0,"multiple":false,"selector":"#full-width-tabpanel-\\/fighters\\/charles-oliveira-1239 div:nth-of-type(1) table","tableDataRowSelector":"tr:nth-of-type(3)","tableHeaderRowSelector":"tr:nth-of-type(3)","columns":[{"extract":false,"header":"Birth Date","name":"Birth Date"},{"extract":false,"header":"1990-02-02","name":"1990-02-02"},{"extract":false,"header":"Reach","name":"Reach"},{"extract":true,"header":"73.0\"","name":"arm_reach"}]},{"id":"fighter_style","parentSelectors":["fighter_link"],"type":"SelectorTable","delay":0,"multiple":false,"selector":"div.MuiPaper-root:nth-of-type(2) table","tableDataRowSelector":"tr:nth-of-type(1)","tableHeaderRowSelector":"tr:nth-of-type(1)","columns":[{"extract":false,"header":"Fighting Style","name":"Fighting Style"},{"extract":true,"header":"BJJ Grappler / Striker","name":"Style"},{"extract":false,"header":"Stance","name":"Stance"},{"extract":true,"header":"Southpaw","name":"stance"}]},{"id":"fighter_camp","parentSelectors":["fighter_link"],"type":"SelectorTable","delay":0,"multiple":false,"selector":"div.MuiPaper-root:nth-of-type(2) table","tableDataRowSelector":"tr:nth-of-type(2)","tableHeaderRowSelector":"tr:nth-of-type(2)","columns":[{"extract":false,"header":"Ranks","name":"Ranks"},{"extract":true,"header":"Black BeltBJJ","name":"rank"},{"extract":false,"header":"Camp","name":"Camp"},{"extract":true,"header":"Bronx's Gold Team USAChute Boxe Diego LimaGold TeamGold Team Texas","name":"camp"}]}]}

ViestursWS · May 10, 2022, 2:57pm

@andrewdcf It happens mainly because the selectors are based on 'auto-generated' classes(they are re-generated on hourly/daily basis & serves as an anti-scraping mechanism). Most of the selectors can be specified using the jQuery selector - :contains().

Learn more:

https://api.jquery.com/category/selectors/

Example:

{"_id":"ufc_camp_crawler","startUrl":["https://fightodds.io/recent-mma-events/ufc"],"selectors":[{"delay":0,"id":"event_link","multiple":true,"parentSelectors":["_root"],"selector":"a.MuiLink-underlineNone.MuiTypography-colorPrimary","type":"SelectorLink"},{"delay":0,"id":"fights_link","multiple":false,"parentSelectors":["event_link"],"selector":"a.MuiTab-root:contains(\"Fights\")","type":"SelectorLink"},{"delay":0,"id":"fighter_link","multiple":true,"parentSelectors":["fights_link"],"selector":"a[href*=\"/fighters\"]","type":"SelectorLink"},{"delay":0,"id":"fighter_name","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Name\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_arm_reach","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Arm Reach\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_style","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Fighting Style\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_camp","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Camp\"):nth(0) + td","type":"SelectorText"}]}

andrewdcf · May 10, 2022, 9:35pm

so this works great but im having the same problem now that ive built in a scrolling function. not sure why it wont recognize the link that it recognized at the top. when i do the data or element preview it populates no problem. Any ideas?

{"_id":"camp_crawler","startUrl":["https://fightodds.io/recent-mma-events/ufc"],"selectors":[{"delay":0,"id":"event_link","multiple":true,"parentSelectors":["_root","event_link","events_scroll"],"selector":"a.MuiLink-underlineNone.MuiTypography-colorPrimary","type":"SelectorLink"},{"delay":0,"id":"fights_link","multiple":false,"parentSelectors":["event_link"],"selector":"a.MuiTab-root:contains(\"Fights\")","type":"SelectorLink"},{"delay":0,"id":"fighter_link","multiple":true,"parentSelectors":["fights_link"],"selector":"a[href*=\"/fighters\"]","type":"SelectorLink"},{"delay":0,"id":"fighter_name","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Name\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_arm_reach","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Arm Reach\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_style","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Fighting Style\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_camp","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Camp\"):nth(0) + td","type":"SelectorText"},{"delay":500,"id":"events_scroll","multiple":true,"parentSelectors":["_root"],"selector":"a.MuiLink-underlineNone.MuiTypography-colorPrimary","type":"SelectorElementScroll"},{"delay":0,"id":"leg_reach","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"tr:nth-of-type(4) td:nth-of-type(4)","type":"SelectorText"},{"delay":0,"id":"rank","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"div.MuiPaper-root:nth-of-type(2) tr:nth-of-type(2) td:nth-of-type(2)","type":"SelectorText"}]}

ViestursWS · May 13, 2022, 10:48am

@andrewdcf It happens because the current 'event_link' & 'events_scroll' child-parent relation is not valid.

You can test that by clicking on the 'Element preview' for the 'event_link' selector in the 2nd navigation level.

You can fix that by updating any of the aforementioned selectors.

Example:

{"_id":"camp_crawler","startUrl":["https://fightodds.io/recent-mma-events/ufc"],"selectors":[{"delay":0,"id":"event_link","multiple":true,"parentSelectors":["events_scroll"],"selector":"a.MuiLink-underlineNone.MuiTypography-colorPrimary","type":"SelectorLink"},{"delay":0,"id":"fights_link","multiple":false,"parentSelectors":["event_link"],"selector":"a.MuiTab-root:contains(\"Fights\")","type":"SelectorLink"},{"delay":0,"id":"fighter_link","multiple":true,"parentSelectors":["fights_link"],"selector":"a[href*=\"/fighters\"]","type":"SelectorLink"},{"delay":0,"id":"fighter_name","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Name\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_arm_reach","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Arm Reach\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_style","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Fighting Style\"):nth(0) + td","type":"SelectorText"},{"delay":0,"id":"fighter_camp","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"td:contains(\"Camp\"):nth(0) + td","type":"SelectorText"},{"delay":500,"id":"events_scroll","multiple":true,"parentSelectors":["_root"],"selector":"div.MuiCardHeader-content","type":"SelectorElementScroll"},{"delay":0,"id":"leg_reach","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"tr:nth-of-type(4) td:nth-of-type(4)","type":"SelectorText"},{"delay":0,"id":"rank","multiple":false,"parentSelectors":["fighter_link"],"regex":"","selector":"div.MuiPaper-root:nth-of-type(2) tr:nth-of-type(2) td:nth-of-type(2)","type":"SelectorText"}]}