Hi there,
I am new to using WebScraper but am trying to pull the name, handle, date and content of pre-selected news related tweets from twitter (which I can successfully do), and then pull the comments from these tweets.
As mentioned, I can successfully pull all the tweet information from the list of tweets in my 'Likes' but cannot figure out how to create a sitemap to pull the comments as well. Does anyone have any advice on this? Is this even possible?
Below is my sitemap (I am sure my second part is incorrect):
{"_id":"twitterscrape","startUrl":["https://twitter.com/i/likes"],"selectors":[{"id":"Project","type":"SelectorLink","parentSelectors":["_root"],"selector":"div.tweet.has-cards","multiple":true,"delay":0},{"id":"Tweets","type":"SelectorElement","parentSelectors":["Project"],"selector":"div.tweet.has-cards","multiple":true,"delay":0},{"id":"Comments","type":"SelectorElement","parentSelectors":["Project"],"selector":"li.ThreadedConversation--loneTweet:nth-of-type(n+2) div.tweet","multiple":true,"delay":0},{"id":"News outlet","type":"SelectorText","parentSelectors":["Tweets"],"selector":"strong.fullname","multiple":false,"regex":"","delay":0},{"id":"Handle","type":"SelectorText","parentSelectors":["Tweets"],"selector":"a.account-group span.username","multiple":false,"regex":"","delay":0},{"id":"Tweet date","type":"SelectorText","parentSelectors":["Tweets"],"selector":"span._timestamp","multiple":false,"regex":"","delay":0},{"id":"Headline","type":"SelectorText","parentSelectors":["Tweets"],"selector":"p.TweetTextSize","multiple":false,"regex":"","delay":0},{"id":"Commenter name","type":"SelectorText","parentSelectors":["Comments"],"selector":"strong.fullname","multiple":false,"regex":"","delay":0},{"id":"Commenter handle","type":"SelectorText","parentSelectors":["Comments"],"selector":"a.account-group span.username","multiple":false,"regex":"","delay":0},{"id":"Comment date","type":"SelectorText","parentSelectors":["Comments"],"selector":"span._timestamp","multiple":false,"regex":"","delay":0},{"id":"Comment","type":"SelectorText","parentSelectors":["Comments"],"selector":"p.TweetTextSize","multiple":false,"regex":"","delay":0}]}
Thanks for your help and advice.
Kind regards,
K