Scraping your own twitter account "with replies" section

Describe the problem.


[EDIT - see post below]

Basically, I am trying to get twitter to scrape my own "with replies" section.

Long story short twitter's export feature doesn't work for me so I gotta export my 40 000 tweets "the hard way"

I have managed to approach a solution using tweetdeck ( ) configured to have one column displaying my own timeline, but have hit some major roadblocks and can't proceed further (see post below for more details and attempted sitemaps)

Okay, I have had some very intermittent success with Tweetdeck

There are two versions of sitemap I've done using undocumented scroll feature from here Scroller does not work on certain websites - #2 by martins

Version one, using "general" selector


version two


Neither works quite right, and both exhibit following problems:

  1. while scrolling works, part of the tweets don't get "grabbed" by the scraper, and process seems very hit-or-miss.
    This seems to be related to the fact that both twitter web and twitter tweetdeck unload material as you scroll, so tweets "out of sight" are lost

  2. very few tweets are generally captured
    likely related to previous problem

  3. the scraping process terminates spontaneously 5-15 seconds in. Not sure why that happens, but scrolling through all 40 000 tweets seems solidly out of reach

After this I am quite lost and would appreciate any help

Tweetdeck is configured to single column, which is configured to show my own timeline (add column -> user->pick your own account)