-
Website
http://www.scripting.com/ -
Original page
http://www.scripting.com/stories/2008/01/18/faqIsDecentralizedTwitterJ.html -
Subscribe
All Comments -
Community
-
-
Top Commenters
-
Popular Threads
All Comments
RSS doesn't scale to real-time data exchange. Even Google measures their RSS aggregation in the minutes or hours. We (at Twitter) measure our delivery times in milliseconds. Even with the concepts that have been discussed for federation of Twitter-like infrastructures, the delivery time is between 10 to 100 milliseconds. RSS also lacks real concepts of privacy - it's all or nothing; no amount of social overlays will fix that, really. OAuth might help, but it's a hard problem.
XMPP really does do all of these things, but essentially no-one has implemented them. To my knowledge, Twitter currently has the most built-out XMPP infrastructure of any social web service, and we're really aggressively moving towards implementing and launching more XMPP features (e.g., PubSub).
You're absolutely correct about the *use* of Twitter, though --- it's entirely up to the user to decide what it is. Some people use it for jokes, for poetry, for recipes, all sorts of writing. Others use it for diaries, questions, answers, all sorts of community endeavors. Some people use it for system-level logging! I'm not sure that we're blind, just that a simple answer (or question) isn't going to pin it down.
And it's pretty clear that Twitter, as currently implemented, doesn't scale either. :-)
Something needs to change.
Twitter as implemented scales just fine --- we're running up capacity constraints, which we're working hard to address. It's probably worth noting that during the macworld outage, which affected our databases and web front-ends, our messaging infrastructure ran basically uninterrupted. For what it's worth, the majority of our traffic is inefficient polling-based API traffic (which uses our JSON, XML, Atom, and RSS endpoints).
[ @dshaw Runs off to try to figure out what XMPP PubSub is. :) ]
I still don't see how the decentralized RSS approach would encourage community like Twitter does, though. I get some discovery out of shared feeds in Google Reader, but not community. I like your idea, but I think we'd see the greatest benefit from some rearchitecting of the Twitter message queue system.
I follow you, but you don't follow me. I think that one way to approach this is in a p2p manner instead of a centralized server. Each person would own their node, chapel@mysite.com which then I could send updates and people could opt to follow me. By opting to follow me they are subscribing basically to an RSS feed of what I say. I can do the same for them or others. I think to really make it work best though would be in the use of xml-rpc for pinging everyone. Each persons node would track who is subscribed and ping them when there is an update, then their node will know to update. So each node is responsible for those that follow, but not those that they are following. The only problem I see is for a public list of updates. The only way that would work is if there was a centralized server that each node pinged with updates. Somewhat like technorati does.
I am interested in where this is going, because I like sites like Twitter and Pownce and discussion is key.
Where Twitter falls down and a distributed version could succeed is in allowing multiple overlapping private communities of interest to form. Twitter is more or less public broadcasting on one channel. It entices people to communicate bidirectionally but by its very nature makes it quite hard to do. Twitter with multiple channels (e.g., my public broadcast channel, my discussion of politics with friends channel, my software product support channel, etc.) would be far more useful to me than just replicating Twitter's functionality in a distributed architecture.
Son of Twitter needs communities the same way we have seen them emerge in other social nets.
The main difference I see is that Twitter makes it dead obvious who's following who. Not so with RSS.
http://dembot.com/post/24117475
I don't anything like think this is going to replace Twitter (or Google Reader shared feeds for that matter) any time soon.
Is there an architecture for s2s built into twitter? (s2s is server to server communication amongst xmpp servers). This would seem to be a good thing to have for corporate twitters and for the capacity issue.
If all my server ever does is give you locally originated messages, we have a N-squared network traffic problem because I will be forced to talk to EVERY subscriber to my feed. So the ability to forward new messages beyond your immediate peers is required. Which means the concept of unique IDs is mandatory. It can be retrofitted by convention into any one of several RSS entities, but that's not optimal.
Here are a couple thoughts:
1. If Twitter were ever to support such a federation, it sounds as if XMPP might help the movement gain some traction with them.
2. Trying to work out an RSS based solution does have the advantage of a ton of developers familiar with it (more than XMPP, I'd think), but any RSS-based solution would have to be more complex than
current RSS development, anyway.
3. Can XMPP and RSS co-exist using RSS cloud element? I can imagine an "RSS Server" that gets notified via XML-RPC from the XMPP Federation, allowing for RSS to poll it for the latest results.
Such a system wouldn't alienate the RSS-only developers.
To me it sound good, even if I'll have to learn more XMPP, but I'm interested in objections out there.
So if we were to set up a federating server, and configure a subset of twitter users to register there, it seems we could take load of twitter.
I think outages are partially a case of capacity tuning for average and not maximum usage. Maximum usage would have two many fallow servers. Its perhaps possible to set up an Amazon ec2 image with ejabberd which can be turned on when requited and pay for peak usage/bandwidth as one goes. The ejabberd's can be clustered, so scaling would be a matter of adding another image...modulo how exactly amazon allocates virtual machines and bandwidth.
Twitter dosent have, sensibly( for spam reasons), account creation API's. So it wouldnt be possible to create accounts on a separate server and mirrir to twitter. If Twitter would provide a way to hook into the registration process as a redirect from another site with the site's info filled in federation would be easy. The other server could intercept twitter traffic between members on it so as to not load twitter but send non-members messages over to twitter.
It would remain to implement the twitter API on the other server which would be a big task in itself but not insurmountable if we reduce the scope of the transports we support and some features.
I'd be willing to install it if someone wants to install a peer and experiment with some API features. Never used Erlang but I don't think that matters.
Gtalk is very stable, so we could build apps on top of it. In other words, you don't need to run out and get a jabber server up and running to get in on the fun.
This could be a nice indirect way to use google's scalability instead of running ones own jabber servers.
They did hire atleast one IRC guru, you know.