How to scrape these comments

Hi all,

I have some experience in webscraping. I have scraped a lot of fora and comments. But at the moment I have a website where it is not possible to scrape the comments of these articles.

Example:

Does anyone know how it is possible to scrape all the comments below an article? I tried everything with element click but the problem is I can't select those boxes. It will automatically select a different part of the website, which I do not need. So when I scrape those comments I want to have 1 comment = 1 element.

Hopefully someone can help me! Thnx :slight_smile:

@Wouter_Rikkink Hi, you probably can't do it because the highlighted comments are divided from regular comments and it appears that they are in iframe (new document/unique webpage)

Therefore you can just use the link which contains these comments.

Sitemap example:

{"_id":"talk","startUrl":["https://talk.nu.nl/embed/stream?asset_url=https%3A%2F%2Fwww.nu.nl%2Fartikel%2F6033418%2Fredirect.html&initialWidth=601&childId=coral_talk_wrapper&parentTitle=Minister%20Koolmees%20onderzoekt%20aanpassing%20rekenregels%20voor%20pensioenen%20%7C%20NU%20-%20Het%20laatste%20nieuws%20het%20eerst%20op%20NU.nl&parentUrl=https%3A%2F%2Fwww.nu.nl%2Fpolitiek%2F6033418%2Fminister-koolmees-onderzoekt-aanpassing-rekenregels-voor-pensioenen.html%3FcommentId%3D%23coral_talk_wrapper"],"selectors":[{"id":"comments","type":"SelectorText","parentSelectors":["_root"],"selector":"[data-slot-name=\"commentContent\"]","multiple":true,"regex":"","delay":0}]}

Hope it helps! :smiley: