Scripting News - Latest Comments in A decentralized Twitter? (Scripting News)

Re: A decentralized Twitter? (Scripting News)

m1k3y — Tue, 01 Jul 2008 03:27:33 -0000

Re: A decentralized Twitter? (Scripting News)

Mark Tsimelzon — Fri, 18 Jan 2008 21:48:25 -0000

Hi Dave,

Downloading the Coral8 server is very easy at http://www.coral8.com/devel.... There is also plenty of documentation at http://www.coral8.com/devel..., starting with a very simple tutorial.

The Coral8 Server can indeed process vast volumes and data in real time, and should be a good match for your requirements. If you have any questions, and I suspect you will, please do not hesitate to send an email to support@coral8.com. We'll be very happy to help you.

Best,

Mark Tsimelzon
CTO / Coral8

Re: A decentralized Twitter? (Scripting News)

Curt Monash — Fri, 18 Jan 2008 19:10:19 -0000

Hi Dave,

Coral8 is more aggressive than StreamBase about making their stuff downloadable to play with, but I'm sure I can hook you up with either company. I'm guessing you'd have to hack your own adapter in either case, but we can ask. As far as persisting, I forget which DBMS each is friendliest with, but dumping stuff into MySQL or, better, EnterpriseDB/MySQL shouldn't be at all hard.

Ping me directly if you like -- I'm just CurtMonash on AIM and Twitter, and of course you have my email address.

CAM

Re: A decentralized Twitter? (Scripting News)

dominik — Fri, 18 Jan 2008 09:26:19 -0000

That's pretty big (and cool) news!

Re: A decentralized Twitter? (Scripting News)

rob friedman — Thu, 17 Jan 2008 23:23:20 -0000

Jabber knows all about being decentralized and federation. It's built in. Someone just needs to implement or clone a twitter like feature set as a jabber service. People could then choose to run their own jabber daemon and twitter clone, or source it all out as a service from an ASP.

Re: A decentralized Twitter? (Scripting News)

rptony — Thu, 17 Jan 2008 22:26:02 -0000

Oops this won't work as such. There should be a scheme to authenticate your followers on your behalf to cross post the tweet that originated from you.

Re: A decentralized Twitter? (Scripting News)

jstanforth — Thu, 17 Jan 2008 21:56:08 -0000

Thought about this a bit this afternoon (Norton apparently takes HOURS to virus-scan 1.7m files... argh), and my ideas are similar but still not completely decentralized. Seems like a system of root-servers (a la DNS) could manage the @username repository and auth mechanisms (to avoid users impersonating other @usernames on the distributed servers), and then a series of widely-distributed servers could mesh via XMPP using the root-server auth to get subscriber lists, etc. The benefit here is that those with thousands of followers would be the ones upgrading their own servers, and others who either don't post frequently or don't care about the lag can just remain on the main (current) Twitter site. So, Engadget sets up their own server, and during MacWorld, all their followers get tweets best-effort pushed directly from their distributed server to all their followers. If they want to support more followers without lag, they can upgrade their own systems--- the followers don't need to upgrade anything at that point, which keeps the upgrade burden on the right side of that equation (i.e., if it's slow, complain to Engadget, not to Twitter or whomever is running the top-level root servers). It also means that Engadget followers don't overwhelm the overall system during high-volume conference weeks, because their added traffic is mostly segregated to their own sub-networks.

This approach essentially builds a caching layer for the current Twitter architecture, so while the many distributed servers use a best-effort push directly to the other specified followers, every tweet would ALSO be pushed up to the root servers for general users, long-term archive, etc., the way that Twitter currently works. If this were built outside of Twitter, presumably the Twitter API would enable pushes directly to Twitter when the system is working normally, and allow tweets to be queued and re-delivered asap if Twitter is unstable/unavailable at that moment. So, yes, there's a lag in this latter case, but in the current system, there's no posting at all in that case.

Then build the root-server level with S3, EC2, SimpleDB, etc, and store the tweet archives as RSS and the follower lists as OPML, and you can then serve them directly from S3 and --- I think? --- support almost every idea I've heard from Dave so far, including enclosures, etc.

Can you think of reasons this wouldn't work? Or even better, better ideas to improve upon this?

Re: A decentralized Twitter? (Scripting News)

jon — Thu, 17 Jan 2008 20:18:03 -0000

What happens when you have 4,000 subscribers? Kind of hard on your server to POST out to 4,000 hosts (some of which might be quite slow). Update 20x daily and you're looking at a nightmare.

Re: A decentralized Twitter? (Scripting News)

rptony — Thu, 17 Jan 2008 18:17:43 -0000

I had been thinking about such a system, though not in thecontext of twitter; but rather as a comment system for blogs. This can be applied here aswell. Let me try;

1. Everyone has a tiny tweet server; its an xml-rpc server with pre-defined interface;

2. When you decide to follow someone who has a hosted tweet server, will as well provide a authentication key; and get one from you too. You don't have to provide it unless the person who desires to follow you has not verified his credintials (i.e you just "blocked" that person)

3. Assume you made a tweet post.; a scheduling algorithm on your tweet server picks one of your followers server as the initial seed to federate the tweet. So your server makes an xml-rpc call to your "follower's" server; So that particular follower gets your tweet.

4. Now the follower who just received the tweet calls back your server and find the next follower from the list and pass on the tweet to them.

5. This process continues until all of your followers has received the tweets.

The idea is to share the load among the ring of tweet servers.

Hope I was clear.

Re: A decentralized Twitter? (Scripting News)

Bill — Thu, 17 Jan 2008 17:44:35 -0000

Would a decentralized twitter be pretty much the same as IRC?

Re: A decentralized Twitter? (Scripting News)

Offbeatmammal — Thu, 17 Jan 2008 17:10:40 -0000

but then again so many (mainstream) reporters (a) don't grok what it's about or (b) have the braincells to work out how to dive in and use it ;)

Re: A decentralized Twitter? (Scripting News)

Offbeatmammal — Thu, 17 Jan 2008 17:07:53 -0000

um, why? they bought Jaiku and that's just about dead now.
the longer Twitter stay independent the better.
Actually, I'd love to see MS buy them to show the Live guys what you can do if you understand the audience well enough

Re: A decentralized Twitter? (Scripting News)

dave — Thu, 17 Jan 2008 16:44:57 -0000

Hi Curt, I read the piece on DBMS2 yesterday, and it was interesting, even compelling. What's the next step? I have no idea how to operate a StreamBase server.

Re: A decentralized Twitter? (Scripting News)

Curt Monash — Thu, 17 Jan 2008 16:42:48 -0000

Maybe I'm missing something, but I just don't see the need to radically decentralized Twitter. Complex event/stream processing (CEP) could easily handle it at a central site, and then the rest is standard redundancy. What StreamBase does for Wall Street or the spooks is 1-2 orders of magnitude more demanding than what Twitter needs today. (Coral8 can do the same things.) That's a lot of headroom. http://www.dbms2.com/2008/0... has some details.

Curt Monash

Re: A decentralized Twitter? (Scripting News)

Robb Montgomery — Thu, 17 Jan 2008 16:41:19 -0000

Google needs to buy twitter.

Re: A decentralized Twitter? (Scripting News)

Blaine Cook — Thu, 17 Jan 2008 15:25:24 -0000

Blaine from Twitter here. FWIW, we have an XMPP PubSub service that allows you to subscribe to public timeline updates. It's currently in "beta", and we're rolling it out as people request it. As far as social network federation goes, it's something that I've talked publicly about since early last year; see my talk from XTech '07 for a general direction, and some of the discussions that came out of the Mediamatic gathering in December for more details. Ralph Meijer (formerly with Jaiku) and I have been discussing the possibility of federating micro-blogging networks since before we met at XTech last year, and really the barrier to doing it is partially a business discussion, and partially a lack of implementors.

There are a whole array of problems with doing the sort of micro-blogging / lifestreaming that Twitter does via RSS / Atom that I'm happy to discuss, but currently I'm a bit too busy dealing with the capacity problems we're facing to do in depth writing on the subject. I will be speaking at a number of events in the next few months, but probably the most relevant (and closest time-wise) is FoWA Miami, where I'll be giving a workshop on building web services using Jabber (XMPP).

Re: A decentralized Twitter? (Scripting News)

jasonw22 — Thu, 17 Jan 2008 11:45:48 -0000

XMPP should be the protocol. Torrent uses trackers for discovery. I'm fuzzy on how tracker-less discovery could work.

Re: A decentralized Twitter? (Scripting News)

Shannon Whitley — Thu, 17 Jan 2008 10:34:30 -0000

For what it's worth, I started working on the Open Micro-blogging (OpenMB) project a couple of months ago. It's a Twitter clone, based on RSS. I'm about ready to introduce the code. I thought to myself, if micro-blogging is similar to blogging, and I run my own WordPress instance, why shouldn't I do the same for my micro-blog. OpenMB is written in PHP and is similar in structure to WordPress. It can also utilize WordPress themes. More coming soon.

Re: A decentralized Twitter? (Scripting News)

Aaron Strout — Thu, 17 Jan 2008 09:10:40 -0000

Dave - how about instead of decentralizing Twitter, we get some investors to step in and fund it so they have more than one server. Or better yet, let's have someone like Yahoo! buy it and let it stand alone like Flickr (I know, this is heresy to many.) Personally, I would be willing to live with a banner ad or two if we could keep Twitter stable.

@astrout

Re: A decentralized Twitter? (Scripting News)

dave — Thu, 17 Jan 2008 07:13:07 -0000

>>Who sits around watching the public_timeline anyway?

as far as I know only reporters trying to prove how banal twitter is. :-)

Re: A decentralized Twitter? (Scripting News)

Nick Johnson — Thu, 17 Jan 2008 07:02:09 -0000

Funnily enough, I've been thinking about exactly this sort of thing.

Why is discovery a problem, though? We have DNS, and that works just fine for discovery for both email and the web. A decentralised twitter need be no more than some microblogging software run by anyone anywhere, with a loose federation system akin to email or XMPP.

Re: A decentralized Twitter? (Scripting News)

zeank — Thu, 17 Jan 2008 04:36:51 -0000

Indeed it would. And there is Publish-Subscribe which resembles most of the functionality required for a twitter-like service. See http://www.xmpp.org/extensi... for details.

Re: A decentralized Twitter? (Scripting News)

Matt Terenzio — Thu, 17 Jan 2008 03:39:52 -0000

My 90% number was a real calculation. Your 5% number is opinion.
I think the API is much more important than that.
Who sits around watching the public_timeline anyway?

Re: A decentralized Twitter? (Scripting News)

Michael Langford — Wed, 16 Jan 2008 23:39:16 -0000

Couldn't you make it so you push messages out? You know where all your followers are, and you do the equivalent of an HTTP post on them?

--Michael

Re: A decentralized Twitter? (Scripting News)

Sol Young — Wed, 16 Jan 2008 22:21:48 -0000

This distributes their load and gains developer and community support. I love 'em a little more for it.