Hi,
I have been trying to scrape YouTube Comments off YouTube pages. YouTube dynamically loads pages which isn't a problem since I am using the Element scroll that allows the web scrape to scroll to the end of the comment threads. For videos, with less than 2500 comments or so, the scraper works well.
The problem happens for pages that have more comments. The scraper will scroll down all the way until the end of the comment thread before returning 2k+ comments with the remaining comments missing. I have tried playing around with the page load delays but the same problem happens. Is there something wrong with the way I am setting up the sitemap? Or does this problem have something to do with the way YouTube pages are structured?
Any help would be greatly appreciated! Thank You!
Sitemap:
{"_id":"youtubecomments","startUrl":["Mixing Every Nail Polish in My Brand Together (What Colour Will It Be?👀) - YouTube a","type":"SelectorLink"},{"delay":5000,"id":"Comment Section","multiple":true,"parentSelectors":["CommentBlock"],"selector":"ytd-comment-thread-renderer","type":"SelectorElementScroll"},{"delay":0,"id":"CommentUser","multiple":false,"parentSelectors":["Comment Section"],"selector":"a.ytd-author-comment-badge-renderer","type":"SelectorLink"},{"delay":0,"id":"CommentText","multiple":false,"parentSelectors":["Comment Section"],"regex":"","selector":"yt-formatted-string#content-text","type":"SelectorText"},{"delay":0,"id":"NumberofLikes","multiple":false,"parentSelectors":["Comment Section"],"regex":"","selector":"span#vote-count-middle","type":"SelectorText"},{"delay":0,"id":"CommentBlock","multiple":false,"parentSelectors":["_root"],"selector":".ytd-comments div#contents.ytd-item-section-renderer","type":"SelectorElement"},{"clickElementSelector":".more-button yt-icon","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":5000,"discardInitialElements":"do-not-discard","id":"ReplySection","multiple":true,"parentSelectors":["Comment Section"],"selector":"div.ytd-comment-thread-renderer","type":"SelectorElementClick"},{"delay":0,"id":"ReplyName","multiple":false,"parentSelectors":["Showmore"],"selector":"a.ytd-comment-renderer","type":"SelectorLink"},{"delay":0,"id":"ReplyText","multiple":false,"parentSelectors":["Showmore"],"regex":"","selector":"yt-formatted-string#content-text","type":"SelectorText"},{"delay":0,"id":"ReplyLikes","multiple":false,"parentSelectors":["Showmore"],"regex":"","selector":"span#vote-count-middle","type":"SelectorText"},{"delay":0,"id":"CommentDate","multiple":false,"parentSelectors":["Comment Section"],"regex":"","selector":".above-comment a","type":"SelectorText"},{"delay":0,"id":"ReplyDate","multiple":false,"parentSelectors":["Showmore"],"regex":"","selector":".above-comment a","type":"SelectorText"},{"clickElementSelector":".ytd-continuation-item-renderer tp-yt-paper-button","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":5000,"discardInitialElements":"do-not-discard","id":"Showmore","multiple":true,"parentSelectors":["ReplySection"],"selector":".ytd-continuation-item-renderer tp-yt-paper-button","type":"SelectorElementClick"}]}