Scripting News: Why does Twitter go down? (Scripting News)
Drew Olanoff
· 1 year ago
Nobody really knows what goes on behind the scenes at Twitter. We can take their word at face value or not. But hey, at least they're eating their own dogfood, and using Twitter and their blog to keep us updated. That's more than a lot of companies do.
Honestly, being passionate about a service is awesome. I'm pretty passionate about communication, the web, and naturally twitter, which brings both together.
But at the end of the day, they're a company. Google doesn't open its doors to discuss infrastructure with just anyone, and time will tell whether Twitter can get geared up for the masses.
The way I see it, early adopters have obviously been extremely valuable to Twitter, but with early adoption comes growing pains (see iPhone). Many web companies have closed their doors because of lack of passion. Twitter has our passion, and while I'm sure they appreciate people talking about Twitter, they are a company of human beings who say they are working as hard as they can.
They're hiring too, so all of us who know talented developers and programmers should send them their way!
Matt Gifford
· 1 year ago
I firmly believe that the problem is with their host, Joyent, rather than with Twitter. I recently had to move my hosting away from Joyent because the service went down *at least* once a day.
Matthew Gifford
· 1 year ago
Never mind what I said about the problem being Joyent. Twitter left them last night.
ericabiz
· 1 year ago
I know why because I've talked to the people who run their infrastructure. To really oversimplify matters, someone had no clue what they were doing on the hosting side of things, and they never corrected that problem by hiring someone who knew how to properly scale infrastructure.
Twitter doesn't really make any money, and everyone who works there basically wants to be a developer. Bad news all around when it comes to uptime and scalability.
ViscountHaldane
· 1 year ago
It's just generally difficult to scale Ruby on Rails deployments. This is one of those dirty little secrets that doesn't get talked about enough.
Anthony
· 1 year ago
Their problems are more fundamental and go way beyond RoR.
Yan Pritzker
· 1 year ago
Thanks for this intelligent bit of FUD. Why is RoR scalability different than any other web platform?
segfault
· 1 year ago
I built Web Apps with RoR, and this is not FUD. RoR scalability is harder because there isn't any established, production-grade (eg Apache) web server that Rails can work efficiently with.
RoR community basically gave up on Apache (FCGI/CGI/whatever), Lighttpd has its own issues and Mongrel is just not there yet (some sites set a cronjob to restart it every hour)
Rails is neat, Ruby is beautiful, but without a good web server its indeed hard to scale. As FUD goes, the hyposphere is definitely on RoR's side anyway.
Yan Pritzker
· 1 year ago
If people are restarting mongrel it's typically because of their own leaky code. Nginx is a production grade webserver and the recommended solution. See engineyard.com for an example of a solid Rails hosting stack.
evan
· 1 year ago
do you think that Twitter will begin to charge, or go with ads, to help pay for their server and infrastructure?
macfixer
· 1 year ago
Like a Twittercon 2008? I'm totally liveblogging it ;-)
I agree, we should be able to possibly help them out or point them into the right direction
jhaggard
· 1 year ago
I'm all with you on this. The level of frustration is directly proportional to how successful Twitter has become. No other service can get away with this level of service and expect to survive, but I'm sure they will.
Count me in on this discussion. I'm from the security aspect of the tech. field, but also well aware of what is happening with new trends. etc.
I've been wondering if the twitter agents feeding real time feeds to users is the major problem. People, me included, are using Twitter agents to give me the illusion of a push technology rather than the pull technology it (html) is.
What ever you do, put me in on any "out of bounds" communication channel established. jhaggard1@leapmail.net
PS. thanks to others for insight.
Rick Mahn
· 1 year ago
Dave, you make a good case for a Twitter Infrastructure briefing. As an enterprise IT architect I'd be called on the carpet in impromptu meetings with upper management if my project had this much impact on users - and I'd have to have some really good explanation and plans to fix the problem asap.
On a side note, as much as I wish my fellow Republican's used Twitter like I do - I also doubt that the debate was a huge traffic generator on Twitter. Let's face it, what Steve Jobs has to say is deemed more important in the tech sphere than politicians stumping for votes.
Regards, Rick
Danielito
· 1 year ago
Hi Dave,
I can understand that there's some load on their servers. But at the end of the day, what they just have to do is to queue incoming messages and dispatch those messages in other queues. That doesn't sound like an "undoable task", even with some load.
I'd really like to know if that's related to bad design decisions, bad code or bad hosting / infra, in all cases thats' equally frustrating !
Stefan Constantinescu
· 1 year ago
Do you have any idea how much more efficient Twitter would be if they switched to XMPP based RSS instead of whatever the hell it is they have now. Twitter is basically instant messaging with even less volume than instant messaging, yet they can't figure out in 2008 what AOL, MSN, ICQ and Yahoo did way back when. What are they waiting for?
Kath
· 1 year ago
Agreed. You don't realize how much you depend on Twitter until you don't have it anymore.
Helge V. Keitel
· 1 year ago
Twitter wouldn't exist without its users...
That's right. My life and work doesn't depend on Twitter but I've noticed the problems. Twitter means an extension of my natural Northern European community. I can follow what happens on the dark / light side of the globe with Twitter. The popularity of Twitter means that the customers should be informed. It's a great tool for open innovation. Why not use the knowledge base of the users. People are willing to help if asked. Br Helge V. Keitel from Finland
David Geller
· 1 year ago
Twitter is not an open, user supported community tool. It's a VC backed commercial entity with goals of rapid growth and, eventually, revenue. It's true they appear to have problems I can guess are related to scaling, but I'm confident they're well aware of these issues and actively and aggressively working to combat them. These scaling problems, frankly, are what any fresh startup dreams of. Hang tight. Things will get better. Hopefully.
Dave Therio
· 1 year ago
When Twitter goes down, so do my spirits. Good Post
Gene_Poole
· 1 year ago
Twitter used to be hosted by a hosting company called Joyent. When they reached sugnificant size they elected to bring up their own infrastructure. They swtiched over recently when the service went off line over a weekend.
The latest clue is that Twitter expectes the Superbowl to really stress the system so they have contracted with Joyent again for extra capacity. This split infrastructure is probably the reason they needed to re-configure the systems... so some of the work stays local and some opverload gets dispatched to infrastructure at Joyent.
NOTE: All of this is based upon a recent Joyent blog post and it IS pure conjecture.
If twitter survives the superbowl and doesn't go down then they have truly scaled and the money invested in infrastructure was well spent. if not, then they will need to invest in more infrasatructure engineering talent to carck the code on the next wave of adoption.
Thinking that twitter can be replaced by protocols misses the essential problem. Large systems that experience network effects always require new levels of expertise that have not been tested by any but a small handful of talent.
Scale anything to a great degree and it will break or deliver service levels that are unacceptable.
During these times of crisis the last thing you want to do is add more "engineers" to the process... especially engineers that use the service and just want it fixed. So, wait and shop for alternative services.
Google out engineered Yahoo for search. Apple is out engineering MS for user experience on a laptop.
Can anyone out engineer twitter for this context sensitive chat environment? It will take a lot of hardware and talent to keep it running. And that means money. I'm amazed they keep scaling and haven't considered how to make money yet.
PS> This all applies to wordpress.com and they are amazing, IMHO.
dahowlett
· 1 year ago
What you're saying is not quite right (though close) regarding the way the Joyent relationship worked or the timings of events.
I can easily imagine the servers choking under 2GB leaking ruby processes...
Phil
· 1 year ago
What I find interesting is that when Twitter goes down, most people try the web interface (API via some app like Twitterrific or the actual page) get an error, and give up.
I've found that the Jabber service rarely goes down, and will even function when the web stuff is experiencing problems. Give it a try next time you can't get to Twitter via the web.
This speaks loudly to the fact that Jabber is much more scalable than the web stuff in Twitter's infrastructure.
Anthony
· 1 year ago
"Infrastructure overhaul?" They ought to do themselves a favor and stop saying that, b/c with all the overhauls and upgrades and maintenance they had over the last several months, you'd think they'd be on an upward path. Instead, they're spiraling down.
My take is, they have no experience with scalability/reliability/availability (e.g., they don't know how to "fail fast") and they're trying to learn the very hard way. They may get there, 2 years from now. Have you seen their queueing component they open sourced. YIKES! I'm sure it gets worse from there.
They claim to have recently hired an expert, and I could just picture it now: He starts putting in place all his pieces (processes, architecture, h/w, s/w, etc), and, well, it turns out he has no idea how to scale ***this type of service*** either. Not his fault. He never did it before, and they all naively believed "scalability is just scalability."
Twitter needs to bring in a real expert.
ifyoumakeit
· 1 year ago
Dave, I think that's a great idea and hope they take you up on the offer. Even if the put you under NDA and you can't share the contents of your meeting, it would be great at this point for them to assemble a "counsel of users" that can act as a PR shield by saying "we've heard their plans and we believe it, so just hold tight". I'll have to do that with my next gig.
dave
· 1 year ago
Eric, I didn't mean to imply that I should be one of the people they brief. They should pick people they trust, and you're right -- even if it's under NDA, it would still be helpful. As long as they're also people that the community trusts.
Nona Myous
· 1 year ago
re: interfaces not supported by scripting environment
If the problem with Google Talk's implementation of the XMPP protocol, or is the problem with XMPP in general?
If the problem is with Google Talk's implementation and not XMPP, then try creating an account at jabber.org or one of the other open services listed here: http://www.jabber.org/user/publicservers.shtml . Most XMPP client applications (Psi, Adium, Pidgen, ...) have the ability to create an account at an open service.
Jeremy Bowers
· 1 year ago
Assuming nobody has touched the Jabber driver for Frontier/Radio Userland since I wrote it, the problem is that Jabber/XMPP clients now tend to absolutely require upgrading to TLS connections, or starting the connection out with SSL, where it was merely an option at the time.
At the time that I wrote the driver, there were no SSL drivers for Frontier/Radio Userland, so the driver does not support it. A quick Google search shows what looks to be a for-pay driver, which looks an awful lot like it only works for HTTPS anyhow. (IIRC, last I knew that was the case, but this is from a long time ago.) Could be wrong, I'm just guessing from the docs.
Most of the public sites will have the same problem. You'd have to run your own server and allow plain-text connections. At that point, if you got the federation setup correct, you could talk to the full Jabber federated network.
If this was fixed, it would otherwise work. The parts of the protocol that have changed since then are mostly really high-level things like conference rooms; the rest of the support would still work.
dave
· 1 year ago
Jeremy, thanks for the update. As far as I know there have been no changes to the tcp.im interface for Jabber. You can check it out by downloading the FlickrFan release and jump to tcp.im. http://flickrfan.org/
99.99% is a max of 52 minutes 33 seconds of downtime per year (no "scheduled downtime" cheating). Not many Web sites, even the best like eBay, easily achieve it.
Advice: Work on 99%. It'll take you a year just to get there.
Ontario Emperor
· 1 year ago
I was going to post a reply to Biz Stone's blog post, suggesting that Twitter convene a select list of customers under NDA to advise them on both technical and monetization matters.
However, comments were not enabled for the Biz Stone post.
Perhaps I'm overreacting, but I find that interesting.
I absolutely agree with David Geller on this. Twitter is not an open, user supported, community tool. However, I have an idea about how we might develop one. The infrastructure is simple (in my opinion) and within the range of capabilities of the platform that Radio/OPML Editor/FlickrFan was written on.
I read this post and Biz Stone's response, and I'm thinking that a small group of outside advisors, under NDA, is the...um...obvious solution. Not just for technical issues, but also for the monetization issue that they keep on deferring.
Dave: yo might want to check out what I had to say at ZDN: http://blogs.zdnet.com/Howlett/?p=300 and then read between the lines. One thing: well written Ruby on Rails code is not an issue - at least not at these levels. It is the infrastructure but the whole story has not been told and may never be told.
Rex Pechler
· 1 year ago
you can script iChat with AppleScript to talk with Twitter's Jabber bot with your Google Talk account.
Gato
· 1 year ago
Dave, we will make either IgniteRealtime.org or another option. Let's connect about what you need.
Eddie Dickey
· 1 year ago
It'd be nice if they could display a message when they go down - but it seems internet connectivity is the current problem so they can't display a 'busy' page. It's a bit of a mystery when the site is down and why.
Edward Vielmetti
· 1 year ago
twitter is down for the same reason that blogger went down all the time. it's being designed to maximize interestingness, not to maximize reliability. going down from time to time makes for a better corporate story..
you don't think they are running a back office for an insurance company, do you?
systems under development should crash from the load from time to time just to give you a chance to reflect upon what you are doing and stress them past the point of breakage. It's rare that most developers get to see things like congestive collapse inside a system of their own design that they can fix.
don't like it? start an irc server on a machine in some monkey's basement and chat with your friends. that works fine since 10+ years ago.
timoni
· 1 year ago
Why do you suggest having a meeting with "a few people from the tech community", rather than just having more details from Twitter?
Allan Stewart
· 1 year ago
Hi,
Twitter is always problematic, goes up and down. Has done since its first conception. I should expect that they have a linux cluster with some kind of a load balancer. I suspect its something to do with too many concurrent connections, they could take this down on each machine and add some move servers to the cluster. Either this or their LB machine is too slow...
chuckwh
· 1 year ago
Weird. As soon as I bless Twiiter with my words, I get followers. Very unscientific, since I've only reached out once to the twitter community, but, somehow, even though I am pretty convinced that my words are gospel to any intelligent being, I suspect that if Twitter goes down my followers will survive.
After all, at the end of the day, I believe in the hardiness of people, twittlee dum, or not.
silpol
· 1 year ago
just go off that Twitter, switch to Jaiku and have a fun of un-interruptible service, with notifications gone both SMS (mobile) and XMPP (yes, GTalk too), and vibrant community... need inivitation? ;)
Robert MacEwan
· 1 year ago
Switch to Pownce or Jaiku. Neither has displayed problems at least not in my hands on use. Should anyone need an invite just give me a buzz.
All Comments
Honestly, being passionate about a service is awesome. I'm pretty passionate about communication, the web, and naturally twitter, which brings both together.
But at the end of the day, they're a company. Google doesn't open its doors to discuss infrastructure with just anyone, and time will tell whether Twitter can get geared up for the masses.
The way I see it, early adopters have obviously been extremely valuable to Twitter, but with early adoption comes growing pains (see iPhone). Many web companies have closed their doors because of lack of passion. Twitter has our passion, and while I'm sure they appreciate people talking about Twitter, they are a company of human beings who say they are working as hard as they can.
They're hiring too, so all of us who know talented developers and programmers should send them their way!
Twitter doesn't really make any money, and everyone who works there basically wants to be a developer. Bad news all around when it comes to uptime and scalability.
RoR community basically gave up on Apache (FCGI/CGI/whatever), Lighttpd has its own issues and Mongrel is just not there yet (some sites set a cronjob to restart it every hour)
Rails is neat, Ruby is beautiful, but without a good web server its indeed hard to scale. As FUD goes, the hyposphere is definitely on RoR's side anyway.
//k
Count me in on this discussion. I'm from the security aspect of the tech. field, but also well aware of what is happening with new trends. etc.
I've been wondering if the twitter agents feeding real time feeds to users is the major problem. People, me included, are using Twitter agents to give me the illusion of a push technology rather than the pull technology it (html) is.
What ever you do, put me in on any "out of bounds" communication channel established. jhaggard1@leapmail.net
PS. thanks to others for insight.
On a side note, as much as I wish my fellow Republican's used Twitter like I do - I also doubt that the debate was a huge traffic generator on Twitter. Let's face it, what Steve Jobs has to say is deemed more important in the tech sphere than politicians stumping for votes.
Regards,
Rick
I can understand that there's some load on their servers. But at the end of the day, what they just have to do is to queue incoming messages and dispatch those messages in other queues. That doesn't sound like an "undoable task", even with some load.
I'd really like to know if that's related to bad design decisions, bad code or bad hosting / infra, in all cases thats' equally frustrating !
That's right. My life and work doesn't depend on Twitter but I've noticed the problems. Twitter means an extension of my natural Northern European community. I can follow what happens on the dark / light side of the globe with Twitter. The popularity of Twitter means that the customers should be informed. It's a great tool for open innovation. Why not use the knowledge base of the users. People are willing to help if asked. Br Helge V. Keitel from Finland
Good Post
The latest clue is that Twitter expectes the Superbowl to really stress the system so they have contracted with Joyent again for extra capacity. This split infrastructure is probably the reason they needed to re-configure the systems... so some of the work stays local and some opverload gets dispatched to infrastructure at Joyent.
NOTE: All of this is based upon a recent Joyent blog post and it IS pure conjecture.
If twitter survives the superbowl and doesn't go down then they have truly scaled and the money invested in infrastructure was well spent. if not, then they will need to invest in more infrasatructure engineering talent to carck the code on the next wave of adoption.
Thinking that twitter can be replaced by protocols misses the essential problem. Large systems that experience network effects always require new levels of expertise that have not been tested by any but a small handful of talent.
Scale anything to a great degree and it will break or deliver service levels that are unacceptable.
During these times of crisis the last thing you want to do is add more "engineers" to the process... especially engineers that use the service and just want it fixed. So, wait and shop for alternative services.
Google out engineered Yahoo for search. Apple is out engineering MS for user experience on a laptop.
Can anyone out engineer twitter for this context sensitive chat environment? It will take a lot of hardware and talent to keep it running. And that means money. I'm amazed they keep scaling and haven't considered how to make money yet.
PS> This all applies to wordpress.com and they are amazing, IMHO.
Check this post I wrote for ZDN that got as close as I could disclose: http://blogs.zdnet.com/Howlett/?p=300
I've found that the Jabber service rarely goes down, and will even function when the web stuff is experiencing problems. Give it a try next time you can't get to Twitter via the web.
This speaks loudly to the fact that Jabber is much more scalable than the web stuff in Twitter's infrastructure.
My take is, they have no experience with scalability/reliability/availability (e.g., they don't know how to "fail fast") and they're trying to learn the very hard way. They may get there, 2 years from now. Have you seen their queueing component they open sourced. YIKES! I'm sure it gets worse from there.
They claim to have recently hired an expert, and I could just picture it now: He starts putting in place all his pieces (processes, architecture, h/w, s/w, etc), and, well, it turns out he has no idea how to scale ***this type of service*** either. Not his fault. He never did it before, and they all naively believed "scalability is just scalability."
Twitter needs to bring in a real expert.
If the problem with Google Talk's implementation of the XMPP protocol, or is the problem with XMPP in general?
If the problem is with Google Talk's implementation and not XMPP, then try creating an account at jabber.org or one of the other open services listed here: http://www.jabber.org/user/publicservers.shtml . Most XMPP client applications (Psi, Adium, Pidgen, ...) have the ability to create an account at an open service.
At the time that I wrote the driver, there were no SSL drivers for Frontier/Radio Userland, so the driver does not support it. A quick Google search shows what looks to be a for-pay driver, which looks an awful lot like it only works for HTTPS anyhow. (IIRC, last I knew that was the case, but this is from a long time ago.) Could be wrong, I'm just guessing from the docs.
Most of the public sites will have the same problem. You'd have to run your own server and allow plain-text connections. At that point, if you got the federation setup correct, you could talk to the full Jabber federated network.
If this was fixed, it would otherwise work. The parts of the protocol that have changed since then are mostly really high-level things like conference rooms; the rest of the support would still work.
99.99%?! Man, we'll just take 99%.
99.99% is a max of 52 minutes 33 seconds of downtime per year (no "scheduled downtime" cheating). Not many Web sites, even the best like eBay, easily achieve it.
Advice: Work on 99%. It'll take you a year just to get there.
However, comments were not enabled for the Biz Stone post.
Perhaps I'm overreacting, but I find that interesting.
I absolutely agree with David Geller on this. Twitter is not an open, user supported, community tool. However, I have an idea about how we might develop one. The infrastructure is simple (in my opinion) and within the range of capabilities of the platform that Radio/OPML Editor/FlickrFan was written on.
Take a look at this: http://distributedtwitter.pbwiki.com/
What do you think?
you don't think they are running a back office for an insurance company, do you?
systems under development should crash from the load from time to time just to give you a chance to reflect upon what you are doing and stress them past the point of breakage. It's rare that most developers get to see things like congestive collapse inside a system of their own design that they can fix.
don't like it? start an irc server on a machine in some monkey's basement and chat with your friends. that works fine since 10+ years ago.
Twitter is always problematic, goes up and down. Has done since its first conception. I should expect that they have a linux cluster with some kind of a load balancer. I suspect its something to do with too many concurrent connections, they could take this down on each machine and add some move servers to the cluster. Either this or their LB machine is too slow...
After all, at the end of the day, I believe in the hardiness of people, twittlee dum, or not.