Scripting News: A decentralized Twitter? (Scripting News)
Blaine Cook
· 1 year ago
Blaine from Twitter here. FWIW, we have an XMPP PubSub service that allows you to subscribe to public timeline updates. It's currently in "beta", and we're rolling it out as people request it. As far as social network federation goes, it's something that I've talked publicly about since early last year; see my talk from XTech '07 for a general direction, and some of the discussions that came out of the Mediamatic gathering in December for more details. Ralph Meijer (formerly with Jaiku) and I have been discussing the possibility of federating micro-blogging networks since before we met at XTech last year, and really the barrier to doing it is partially a business discussion, and partially a lack of implementors.
There are a whole array of problems with doing the sort of micro-blogging / lifestreaming that Twitter does via RSS / Atom that I'm happy to discuss, but currently I'm a bit too busy dealing with the capacity problems we're facing to do in depth writing on the subject. I will be speaking at a number of events in the next few months, but probably the most relevant (and closest time-wise) is FoWA Miami, where I'll be giving a workshop on building web services using Jabber (XMPP).
dominik
· 1 year ago
That's pretty big (and cool) news!
andrewbaron
· 1 year ago
Dave, thank you so much for posting about this. In looking at everyone's comments here, it's amazing to see the brain trust of everyone coming together to crack the nut. Twitter could never in a million years buy this kind of value for their product.
I guess what makes Twitter special is the community we all have there so of course it will requires the community to understand and solve our own problems.
dave
· 1 year ago
When we were breaking through on podcasting our slogan was:
Users and developers party together.
That's the magic combo. You need both to make a community boot up.
Dan York
· 1 year ago
Dave - if we create a "decentralized Twitter" don't we really just wind up with..... IRC?
(Mind you, I very definitely *do* agree that Twitter needs to be more scalable - and that a decentralized system could do that.)
dave
· 1 year ago
Maybe so. But probably not.
The key idea to Twitter is the same one that's key to RSS, the idea that my flow of messages is the result of who I've subscribed to. That's an opt-in.
If I don't want to listen to someone on IRC, my options are to get off the channel, basically unsub from everyone, or find a way to block one person (if there is a way). But even so, people will assume everyone can hear everyone else.
You might say that's good and democratic and all, but the problem is that trolls end up ruling the space. Just like mail lists. In Twitter this doesn't happen, as it doesn't with blogs and RSS because there is no assumption that everyone can hear everyone else. And if I don't like what someone is saying, I can unsub from them, quietly, without ceremony.
We haven't seen Twitter devolve into troll paranoia, I think this is why. It has an immunity that IRC does not have.
Jan
· 1 year ago
I basically agree with dave here.
> If I don't want to listen to someone on IRC, my options are to get off the channel, basically unsub from everyone, or find a way to block one person (if there is a way).
There's /ignore <nick>
>But even so, people will assume everyone can hear everyone else.
It's kinda like that in Twitter, too. Or do you have the distinct notion of you talking to an audience that chose you?
dave
· 1 year ago
I don't get your last paragraph. The only people who can hear me on Twitter are the people who have chosen to "follow" me. It's not a big chatroom, every person's experience on Twitter is different, by their design.
I can only hear those who I have chosen to follow. Very different from IRC where I listen to the people who joined the chatroom. It's their choice on IRC, my choice on Twitter.
Jan
· 1 year ago
Technically, yes. But I (and I heard that from friends and followers, too), that it feels like a huge chat room, only that my real-life student friends don't see what I say to the indie mac devs and vice versa. But from my POV, it's all the same.
Jan
· 1 year ago
Seems to be easy to do with CouchDB and its replication system
greywulf
· 1 year ago
I reckon all that's needed really is an rss-only blog engine. I wrote a simple one from scratch a while ago, so it's not that difficult to implement; just bolt a secure front-end and you're done. From there, it's just aggregation and dispersal, surely.
Kevin Newman
· 1 year ago
I wonder if Jabber/XMMS would be a good starting place for a decentralized Twitter-like system. I don't know that much about it, but it is open, XML-based, decentralized, and already in wide use.
Indeed it would. And there is Publish-Subscribe which resembles most of the functionality required for a twitter-like service. See http://www.xmpp.org/extensions/xep-0060.html for details.
jeremy
· 1 year ago
Sounds like there could be more opportunities to expand the business model if you were to decentralize. I would pay $20 - $50 / yr to be able to host my own tweets on my server. I would expect to download something similar to WP, and install it in a matter of clicks. I am sure most hosts would offer a 1-click install of apps like these.
Steve
· 1 year ago
I think hosting and all that isnt an issue, what is the issue is getting a SMS Short Code so you can text in tweets on your own. At around 5k a year (last I checked) it kind of puts it out of most price ranges...That is what is good about twitter, you can piggyback on their SMS Short Code, and build apps around it..
silencematters
· 1 year ago
That is where Twitter would come in handy - they would support the SMS features,.. and the email prefs, etc.
Jay Goodman Tamboli
· 1 year ago
If you host your own, don't you lose the interactivity? I could respond to you on your server, but all my other followers would miss that exchange.
dave
· 1 year ago
Not if it's done right.
Think about it this way.
I subscribe to about 500 feeds.
Most of them are hosted on different servers.
Yet I see all their updates.
Same idea only on a larger scale. (More feeds.)
John H
· 1 year ago
Well, yeah, but you only get updates (eg) every hour, which'd makes conversations rather slow. Ramping that up to polling every couple of minutes would generate an awful lot of HTTP requests, probably more than Joe Normal's {hosted space || DSL line } could handle.
dave
· 1 year ago
I thought of that, and don't have an answer. Thinking about it.
Michael Langford
· 1 year ago
Couldn't you make it so you push messages out? You know where all your followers are, and you do the equivalent of an HTTP post on them?
--Michael
jonknee
· 1 year ago
What happens when you have 4,000 subscribers? Kind of hard on your server to POST out to 4,000 hosts (some of which might be quite slow). Update 20x daily and you're looking at a nightmare.
silencematters
· 1 year ago
Now wouldnt it be nice if we could do the same things with Open ID?
alexsolleiro
· 1 year ago
Hi,
Are you aware of the DiSo project? We're looking into building blocks for distributed social networking, and here are some examples of the interactions we're looking into: - a profile page + friends list and permissioning, managed in your chosen "home" (for the moment all hacking is done on WordPress, so your "blog" would be your home but it's not limited to that) - trackbacking and messaging using different discovery techniques (Microformats to identify content is one) for pinging and distributing content to other people's 'homes' - rosters (XMPP) to distribute content from "home" to, say, IM, email, etc.
These blocks effectively allow for a decentralized Twitter.
Dave, you're in great position to create this. At one level it's really nothing more than OPML + a cut down subset of RSS[1] + some client software. If you can get a core group of a 100 people using this, it might well spread and turn into an ecosystem of alternate clients and aggregation services.
What's interesting is that we've already got this problem because there's Twitter, Facebook, Jaiku, Pownce, etc etc. We need a bit of software to automatically link them all together and provide a quick and easy way to post to one of them.
[1] No item.title, limited length item.description, channel.image
dave
· 1 year ago
It may be that all I need to do is write a blog post, there's so little software needed.
In any case, I'll work on this software after I get the podcatcher out, which should be pretty soon. It'll be very easy to get 100 people with the software installed because we've already got a few hundred people running FlickrFan, and I think they may be the kind of people who are interested in this stuff. They all have enough CPU bandwidth and disk storage to do it, it's considerably smaller app than FlickrFan.
I also like the idea of using RSS as the basis because the payloads stuff that I've been wanting so badly just falls out -- it's the enclosure element.
I also find myself wondering if Amazon's SimpleDB can play a role here. I have all the basics already implemented and am itching to find a nice application for it. :-)
Anyway thanks for the encouraging words Julian.
cshotton
· 1 year ago
The trick is to realize that "hosting your own Twitter" really means not running Twitter for use by others, just yourself. And if your Twitter knows who your buddies are (and what shared communities you have with others), then Twitter becomes a simple set of exchanges of messages between peers and not really the current client/server model at all. It'd never be like IRC. If anything, it's more like NNTP insofar as machines have to negotiate with each other about subscribers and messages that need to be exchanged. Dave rightly points out that the biggest issue is having a rendezvous service, Piggy-back it on email. That's been done and works well. Reconnecting a network by emailing around an "address book" is easy.
Larry Borsato
· 1 year ago
You would still need a way to get texts from your phone into your feed. Perhaps Twitter could still handle the incoming SMS messages and route them, while your decentralized idea handles everything else.
Offbeatmammal
· 1 year ago
One of the things I like about Twitter at the moment is the domainless names. So I'm offbeatmammal there, not offbeatmammal@twitter.scripting.com or whatever. It makes it neat and easy to communicate. If the distributed Twitter-sphere had a name server equivalent as well as a p2p network of feeds (so it doesn't matter which server I write my "@davewiner has another great idea" tweet you get to see it and I get your reply.... in a timely, aggregated way. The moment we go down the IM or mobile phone roaming tax world of split systems I'll go back to email and blogging and face-time ;)
hallson
· 1 year ago
I too am a non-techie, so this question might seem silly.. Email is decentralized. SMS is decentralized. Why can't a twitter-type service be as well?
Sol Young
· 1 year ago
So what happens to TwitterVision and the public timeline in a decentralized Twitter? Sure, we can all subscribe to everyone elses feeds... And everyone already does that for blogs/news... But we lose two of the special ingredients in a centralized Twitter: 1. Public Timeline. 2. Identity (single username, similar to AIM vs. IRC).
If those two items are solved I'd be on board. I don't expect they will.
Would this be an issue if Twitter were solid? My money is on "No."
dave
· 1 year ago
What happens to TwitterVision and the public timeline when Twitter goes down?
Your last two sentences were right on the money.
The only time I think about creating a colony of clones is when Twitter is down. Other times I'm too busy. :-)
Matt Terenzio
· 1 year ago
@solyoung @davewiner The identity part is not an issue. We have distributed identity in email, jabber, or openid. Public Timeline is somewhat tougher, since in a distributed system based on RSS (in the fashion of Usenet), it would take time to propagate a post throughout all host nodes. I've been working on this idea for some time (an RSS usenet). Actually, it's been since Microsoft announced SSE which I though would help a distributed discussion system sync itself. I think it can be achieved though. In fact, I think a little known part of RSS 2.0 can play a part. The Cloud element. Look it up, if you are not familiar with it. Any PHP folks who want to work on an open system with me., I have 90% of the Twitter API coded already. Let's do it.
Sol Young
· 1 year ago
90% of the Twitter API is 5% of what Twitter is about. It's the centralized public timeline, speed (when not congested) and identity that matter. It's the pure simplicity and fact that there are users (content) on Twitter. It took years for RSS to spin up. Twitter is the coral reef that is now.
I trust they'll resolve their stability before a decentralized competitor could compete.
And SSE for the volume and spread of Twitter updates? That's like choosing Ruby and a single database for.....
Matt Terenzio
· 1 year ago
My 90% number was a real calculation. Your 5% number is opinion. I think the API is much more important than that. Who sits around watching the public_timeline anyway?
dave
· 1 year ago
>>Who sits around watching the public_timeline anyway?
as far as I know only reporters trying to prove how banal twitter is. :-)
Offbeatmammal
· 1 year ago
but then again so many (mainstream) reporters (a) don't grok what it's about or (b) have the braincells to work out how to dive in and use it ;)
Matt Terenzio
· 1 year ago
Let me clarify the identity thought I had. Like DNS , a username maps to to a domain specific name. If you are @davewiner then that maps to davewiner@twitter.com (for now) But in the future, @opmlmaster could actually map to another domain opmlmaster@scripting.com Everyone that wanted to be in the cooperative would have to register a username. Or we could use inames. ; ) check out inames.net Then openid could be used for sign in.
Perhaps this will enable some alternative possibilities?
Sol Young
· 1 year ago
This distributes their load and gains developer and community support. I love 'em a little more for it.
Nick Johnson
· 1 year ago
Funnily enough, I've been thinking about exactly this sort of thing.
Why is discovery a problem, though? We have DNS, and that works just fine for discovery for both email and the web. A decentralised twitter need be no more than some microblogging software run by anyone anywhere, with a loose federation system akin to email or XMPP.
Aaron Strout
· 1 year ago
Dave - how about instead of decentralizing Twitter, we get some investors to step in and fund it so they have more than one server. Or better yet, let's have someone like Yahoo! buy it and let it stand alone like Flickr (I know, this is heresy to many.) Personally, I would be willing to live with a banner ad or two if we could keep Twitter stable.
@astrout
Shannon Whitley
· 1 year ago
For what it's worth, I started working on the Open Micro-blogging (OpenMB) project a couple of months ago. It's a Twitter clone, based on RSS. I'm about ready to introduce the code. I thought to myself, if micro-blogging is similar to blogging, and I run my own WordPress instance, why shouldn't I do the same for my micro-blog. OpenMB is written in PHP and is similar in structure to WordPress. It can also utilize WordPress themes. More coming soon.
jasonw22
· 1 year ago
XMPP should be the protocol. Torrent uses trackers for discovery. I'm fuzzy on how tracker-less discovery could work.
Robb Montgomery
· 1 year ago
Google needs to buy twitter.
Offbeatmammal
· 1 year ago
um, why? they bought Jaiku and that's just about dead now. the longer Twitter stay independent the better. Actually, I'd love to see MS buy them to show the Live guys what you can do if you understand the audience well enough
Curt Monash
· 1 year ago
Maybe I'm missing something, but I just don't see the need to radically decentralized Twitter. Complex event/stream processing (CEP) could easily handle it at a central site, and then the rest is standard redundancy. What StreamBase does for Wall Street or the spooks is 1-2 orders of magnitude more demanding than what Twitter needs today. (Coral8 can do the same things.) That's a lot of headroom. http://www.dbms2.com/2008/01/16/twitter-could-e... has some details.
Curt Monash
dave
· 1 year ago
Hi Curt, I read the piece on DBMS2 yesterday, and it was interesting, even compelling. What's the next step? I have no idea how to operate a StreamBase server.
Curt Monash
· 1 year ago
Hi Dave,
Coral8 is more aggressive than StreamBase about making their stuff downloadable to play with, but I'm sure I can hook you up with either company. I'm guessing you'd have to hack your own adapter in either case, but we can ask. As far as persisting, I forget which DBMS each is friendliest with, but dumping stuff into MySQL or, better, EnterpriseDB/MySQL shouldn't be at all hard.
Ping me directly if you like -- I'm just CurtMonash on AIM and Twitter, and of course you have my email address.
The Coral8 Server can indeed process vast volumes and data in real time, and should be a good match for your requirements. If you have any questions, and I suspect you will, please do not hesitate to send an email to support@coral8.com. We'll be very happy to help you.
Best,
Mark Tsimelzon CTO / Coral8
Bill
· 1 year ago
Would a decentralized twitter be pretty much the same as IRC?
rob friedman
· 1 year ago
Jabber knows all about being decentralized and federation. It's built in. Someone just needs to implement or clone a twitter like feature set as a jabber service. People could then choose to run their own jabber daemon and twitter clone, or source it all out as a service from an ASP.
rptony
· 1 year ago
I had been thinking about such a system, though not in thecontext of twitter; but rather as a comment system for blogs. This can be applied here aswell. Let me try;
1. Everyone has a tiny tweet server; its an xml-rpc server with pre-defined interface;
2. When you decide to follow someone who has a hosted tweet server, will as well provide a authentication key; and get one from you too. You don't have to provide it unless the person who desires to follow you has not verified his credintials (i.e you just "blocked" that person)
3. Assume you made a tweet post.; a scheduling algorithm on your tweet server picks one of your followers server as the initial seed to federate the tweet. So your server makes an xml-rpc call to your "follower's" server; So that particular follower gets your tweet.
4. Now the follower who just received the tweet calls back your server and find the next follower from the list and pass on the tweet to them.
5. This process continues until all of your followers has received the tweets.
The idea is to share the load among the ring of tweet servers.
Hope I was clear.
rptony
· 1 year ago
Oops this won't work as such. There should be a scheme to authenticate your followers on your behalf to cross post the tweet that originated from you.
jstanforth
· 1 year ago
Thought about this a bit this afternoon (Norton apparently takes HOURS to virus-scan 1.7m files... argh), and my ideas are similar but still not completely decentralized. Seems like a system of root-servers (a la DNS) could manage the @username repository and auth mechanisms (to avoid users impersonating other @usernames on the distributed servers), and then a series of widely-distributed servers could mesh via XMPP using the root-server auth to get subscriber lists, etc. The benefit here is that those with thousands of followers would be the ones upgrading their own servers, and others who either don't post frequently or don't care about the lag can just remain on the main (current) Twitter site. So, Engadget sets up their own server, and during MacWorld, all their followers get tweets best-effort pushed directly from their distributed server to all their followers. If they want to support more followers without lag, they can upgrade their own systems--- the followers don't need to upgrade anything at that point, which keeps the upgrade burden on the right side of that equation (i.e., if it's slow, complain to Engadget, not to Twitter or whomever is running the top-level root servers). It also means that Engadget followers don't overwhelm the overall system during high-volume conference weeks, because their added traffic is mostly segregated to their own sub-networks.
This approach essentially builds a caching layer for the current Twitter architecture, so while the many distributed servers use a best-effort push directly to the other specified followers, every tweet would ALSO be pushed up to the root servers for general users, long-term archive, etc., the way that Twitter currently works. If this were built outside of Twitter, presumably the Twitter API would enable pushes directly to Twitter when the system is working normally, and allow tweets to be queued and re-delivered asap if Twitter is unstable/unavailable at that moment. So, yes, there's a lag in this latter case, but in the current system, there's no posting at all in that case.
Then build the root-server level with S3, EC2, SimpleDB, etc, and store the tweet archives as RSS and the follower lists as OPML, and you can then serve them directly from S3 and --- I think? --- support almost every idea I've heard from Dave so far, including enclosures, etc.
Can you think of reasons this wouldn't work? Or even better, better ideas to improve upon this?
All Comments
There are a whole array of problems with doing the sort of micro-blogging / lifestreaming that Twitter does via RSS / Atom that I'm happy to discuss, but currently I'm a bit too busy dealing with the capacity problems we're facing to do in depth writing on the subject. I will be speaking at a number of events in the next few months, but probably the most relevant (and closest time-wise) is FoWA Miami, where I'll be giving a workshop on building web services using Jabber (XMPP).
I guess what makes Twitter special is the community we all have there so of course it will requires the community to understand and solve our own problems.
Users and developers party together.
That's the magic combo. You need both to make a community boot up.
(Mind you, I very definitely *do* agree that Twitter needs to be more scalable - and that a decentralized system could do that.)
The key idea to Twitter is the same one that's key to RSS, the idea that my flow of messages is the result of who I've subscribed to. That's an opt-in.
If I don't want to listen to someone on IRC, my options are to get off the channel, basically unsub from everyone, or find a way to block one person (if there is a way). But even so, people will assume everyone can hear everyone else.
You might say that's good and democratic and all, but the problem is that trolls end up ruling the space. Just like mail lists. In Twitter this doesn't happen, as it doesn't with blogs and RSS because there is no assumption that everyone can hear everyone else. And if I don't like what someone is saying, I can unsub from them, quietly, without ceremony.
We haven't seen Twitter devolve into troll paranoia, I think this is why. It has an immunity that IRC does not have.
> If I don't want to listen to someone on IRC, my options are to get off the channel, basically unsub from everyone, or find a way to block one person (if there is a way).
There's /ignore <nick>
>But even so, people will assume everyone can hear everyone else.
It's kinda like that in Twitter, too. Or do you have the distinct notion of you talking to an audience that chose you?
I can only hear those who I have chosen to follow. Very different from IRC where I listen to the people who joined the chatroom. It's their choice on IRC, my choice on Twitter.
http://www.jabber.org/about/overview.shtml
Think about it this way.
I subscribe to about 500 feeds.
Most of them are hosted on different servers.
Yet I see all their updates.
Same idea only on a larger scale. (More feeds.)
--Michael
Are you aware of the DiSo project? We're looking into building blocks for distributed social networking, and here are some examples of the interactions we're looking into:
- a profile page + friends list and permissioning, managed in your chosen "home" (for the moment all hacking is done on WordPress, so your "blog" would be your home but it's not limited to that)
- trackbacking and messaging using different discovery techniques (Microformats to identify content is one) for pinging and distributing content to other people's 'homes'
- rosters (XMPP) to distribute content from "home" to, say, IM, email, etc.
These blocks effectively allow for a decentralized Twitter.
Do take a look: http://diso-project.org , then head to the GoogleGroup and the Wiki.
It's dataportability.org and VRM-aware and -friendly.
What's interesting is that we've already got this problem because there's Twitter, Facebook, Jaiku, Pownce, etc etc. We need a bit of software to automatically link them all together and provide a quick and easy way to post to one of them.
[1] No item.title, limited length item.description, channel.image
In any case, I'll work on this software after I get the podcatcher out, which should be pretty soon. It'll be very easy to get 100 people with the software installed because we've already got a few hundred people running FlickrFan, and I think they may be the kind of people who are interested in this stuff. They all have enough CPU bandwidth and disk storage to do it, it's considerably smaller app than FlickrFan.
I also like the idea of using RSS as the basis because the payloads stuff that I've been wanting so badly just falls out -- it's the enclosure element.
I also find myself wondering if Amazon's SimpleDB can play a role here. I have all the basics already implemented and am itching to find a nice application for it. :-)
Anyway thanks for the encouraging words Julian.
If the distributed Twitter-sphere had a name server equivalent as well as a p2p network of feeds (so it doesn't matter which server I write my "@davewiner has another great idea" tweet you get to see it and I get your reply.... in a timely, aggregated way.
The moment we go down the IM or mobile phone roaming tax world of split systems I'll go back to email and blogging and face-time ;)
1. Public Timeline.
2. Identity (single username, similar to AIM vs. IRC).
If those two items are solved I'd be on board. I don't expect they will.
Would this be an issue if Twitter were solid? My money is on "No."
Your last two sentences were right on the money.
The only time I think about creating a colony of clones is when Twitter is down. Other times I'm too busy. :-)
Public Timeline is somewhat tougher, since in a distributed system based on RSS (in the fashion of Usenet), it would take time to propagate a post throughout all host nodes.
I've been working on this idea for some time (an RSS usenet). Actually, it's been since Microsoft announced SSE which I though would help a distributed discussion system sync itself.
I think it can be achieved though. In fact, I think a little known part of RSS 2.0 can play a part. The Cloud element. Look it up, if you are not familiar with it.
Any PHP folks who want to work on an open system with me., I have 90% of the Twitter API coded already. Let's do it.
I trust they'll resolve their stability before a decentralized competitor could compete.
And SSE for the volume and spread of Twitter updates? That's like choosing Ruby and a single database for.....
I think the API is much more important than that.
Who sits around watching the public_timeline anyway?
as far as I know only reporters trying to prove how banal twitter is. :-)
Like DNS , a username maps to to a domain specific name.
If you are @davewiner then that maps to davewiner@twitter.com (for now)
But in the future, @opmlmaster could actually map to another domain opmlmaster@scripting.com
Everyone that wanted to be in the cooperative would have to register a username.
Or we could use inames. ; ) check out inames.net
Then openid could be used for sign in.
Perhaps this will enable some alternative possibilities?
Why is discovery a problem, though? We have DNS, and that works just fine for discovery for both email and the web. A decentralised twitter need be no more than some microblogging software run by anyone anywhere, with a loose federation system akin to email or XMPP.
@astrout
the longer Twitter stay independent the better.
Actually, I'd love to see MS buy them to show the Live guys what you can do if you understand the audience well enough
Curt Monash
Coral8 is more aggressive than StreamBase about making their stuff downloadable to play with, but I'm sure I can hook you up with either company. I'm guessing you'd have to hack your own adapter in either case, but we can ask. As far as persisting, I forget which DBMS each is friendliest with, but dumping stuff into MySQL or, better, EnterpriseDB/MySQL shouldn't be at all hard.
Ping me directly if you like -- I'm just CurtMonash on AIM and Twitter, and of course you have my email address.
CAM
Downloading the Coral8 server is very easy at http://www.coral8.com/developers/download.html. There is also plenty of documentation at http://www.coral8.com/developers/documentation...., starting with a very simple tutorial.
The Coral8 Server can indeed process vast volumes and data in real time, and should be a good match for your requirements. If you have any questions, and I suspect you will, please do not hesitate to send an email to support@coral8.com. We'll be very happy to help you.
Best,
Mark Tsimelzon
CTO / Coral8
1. Everyone has a tiny tweet server; its an xml-rpc server with pre-defined interface;
2. When you decide to follow someone who has a hosted tweet server, will as well provide a authentication key; and get one from you too. You don't have to provide it unless the person who desires to follow you has not verified his credintials (i.e you just "blocked" that person)
3. Assume you made a tweet post.; a scheduling algorithm on your tweet server picks one of your followers server as the initial seed to federate the tweet. So your server makes an xml-rpc call to your "follower's" server; So that particular follower gets your tweet.
4. Now the follower who just received the tweet calls back your server and find the next follower from the list and pass on the tweet to them.
5. This process continues until all of your followers has received the tweets.
The idea is to share the load among the ring of tweet servers.
Hope I was clear.
This approach essentially builds a caching layer for the current Twitter architecture, so while the many distributed servers use a best-effort push directly to the other specified followers, every tweet would ALSO be pushed up to the root servers for general users, long-term archive, etc., the way that Twitter currently works. If this were built outside of Twitter, presumably the Twitter API would enable pushes directly to Twitter when the system is working normally, and allow tweets to be queued and re-delivered asap if Twitter is unstable/unavailable at that moment. So, yes, there's a lag in this latter case, but in the current system, there's no posting at all in that case.
Then build the root-server level with S3, EC2, SimpleDB, etc, and store the tweet archives as RSS and the follower lists as OPML, and you can then serve them directly from S3 and --- I think? --- support almost every idea I've heard from Dave so far, including enclosures, etc.
Can you think of reasons this wouldn't work? Or even better, better ideas to improve upon this?