DISQUS

Andrew Fielden · 7 months ago

I suspect that it is technical and that it is one of a batch of things that they are doing to lower server load. 2 weeks ago I was in a group that had a discussion with twitter about the API. There were queries over the availability of tweets for searching (21 days at the time) and the taking away of the firehose stream of tweets. Neither was given a date for alteration. Furthermore it was readily apparant that they were suffering from a lack of capacity to handle the load.

So I started to think about the volumes of traffic that twitter has to cope with. A standard API call to get the latest updates for a user can be made up to 100 times per hour and returns the last 200 relevant tweets by default. A search can be made more times via the API but I not sure how many. In any case with the increase of client apps to interact with twitter and the increase of the number of users the number of API calls will be increasing dramatically. It is no wonder that they are looking for ways to cut down the work that the server is undertaking.

On the other hand this could just be disembling the real reason behind it all. Maybe they are looking for features that power users/communities will pay for and this maybe one.

MarinaMartin · 7 months ago

I'm a novice programmer, so maybe I'm really missing something, but could someone provide an example where the now-default "only see @ replies if you're following that person" would be *less* process-intensive than "see all messages sent by users I'm following"? Seems like you'd have to filter each and every tweet in the former situation, whereas the latter situation requires less work. I'm partially interested because I don't believe Twitter and partially interested because I'd like to expand my programming horizons.

dacort · 7 months ago

As I understand their architecture (which is from various blog posts/presentations so reader beware), here are the two work flows.

- New setting
1) a reply is posted
2) retrieve original poster's followers
3) see if that @reply user is followed by those users
4) post the message to the necessary timelines

**Depending on their structure (!!) that can be a pretty easy query. i.e. if they can do some simple "set" queries to find friends of friends.

- Old setting
1) a reply is posted
2) retrieve original poster's followers
3) for each of those followers, see if they have the special setting enabled
3a) If so, populate the timeline
3b) If not, check if the @reply user is followed by that user

3a/3b are the killer - if the data isn't structured right, it becomes a much more complex problem to find the intersection of two settings than simply the friends of friends. This may not have been too much of a problem when the most popular person had 100,000 followers. But now that those numbers are nearing 2M, that's a lot of operations to perform for one tweet.

MarinaMartin · 7 months ago

A-ha! That makes sense. I was looking at it from the perspective of accessing tweets via the API (which I'm aware Twitter itself doesn't do) and distributing tweets the opposite way, where it pulls my timeline from the relevant sources instead of pushing each update into the right timelines.

Heaven's forbid they have explained it THAT way instead of saying they took away the option because it confused people.

mckoss · 7 months ago

Still confused. New (3) is the SAME as Old (3b) - whu is it "pretty easy" now, and "killer" then? The real difference is Old(3).

It seems to me, you just send the tweet to the cluster for each of the followers to be added to the timeline. That cluster can then check the receiving user's settings to determine if the tweet should be filtered out. Simple!

dacort · 7 months ago

Sorry, I should have said that the old (3) as a whole is the killer. I glossed over the "see if they have the special setting enabled", which is really where the extra order of work comes (if(!!) I understand their speculated(!!) architecture correctly).

Because now, instead of performing the intersection of several in-memory arrays of numeric id's, you have to perform a round trip to the server for each of a user's followers to see if they have this setting enabled. Again - this is based completely on speculation of how the architecture is - that friends/followers are cached in memory and per-user settings are not. Even if the per-user settings are, I wouldn't think it'd be too easy to do a simple "join" as they're likely using a simple key/value memcache store. If they have fractured users into clusters, those clusters still have to do the work of checking each incoming tweet's followers to see if the "special setting" is enabled in order to determine whether to send the tweet or not. My guess(!!) is that the social graph data is heavily cached and even if the user settings were, it would still require iterating over a large number of users to get the setting.

joelfox · 7 months ago

twitter is down for maintenance.

Foomandoonian · 7 months ago

That was really nicely summed up!

joec0914 · 7 months ago

Totally agree, especially with the last paragraph about centralization. It's folly to think that the whole world will let two or three guys tell them how to communicate. More unilateral changes like this that fundamentally alter people's ability to use and enjoy the service are certainly down the road, whether Twitter gets sold or stays in the hands of the founders. Communication mechanism always have tended to standardize and decentralize from the telegraph, to the telephone to email. Twitter cannot be any different, methinks.

anildash · 7 months ago

I think the second half of this post could be extracted into some nice general principles for almost *any* web service that real people use. It's the same philosophy I see in the work that community managers do on the sites I enjoy using, where they address each of these areas (often very subtly) whenever they announce a change.

howardweaver · 7 months ago

Thanks, Dave. I appreciate getting a deeply informed view of things like this. I don't know enough about the technology, for example, to evaluate on my own why something may or may not have happened. I trust your judgment here and thus have a more informed impression myself.

Ari Herzog · 7 months ago

What's your thought, Dave, whether future Twitter changes (or any social networking site change for this matter) be put to the users/customers to provide feedback BEFORE anything is enacted, in case the what-if protocol needs amendment? This is what Obama proposed during his campaign for non-emergency legislation and what Facebook recently enacted for its terms of service, etc. Essentially allowing either Digg-like votes or a central repository to send feedback, e.g. blog comments.

Or... should companies make changes and then post explanations why?

dsims · 7 months ago

This is also shows why twitter is stuck and is unable to add any real new functionality. So many have no idea how twitter replies work, or that "how" they tweet can effect who can see it, or that a reply can be linked to a status (if your app supports it, and if you use the reply button correctly). All these details are hidden from the average user, which is a big reason why twitter has seen so much success so far. But, since the twitter-verse can't seem to handle even this simple concept, I don't plan on there ever being additional metadata added to tweets.

sliu · 7 months ago

GPL micro-blogging software such as http://laconi.ca might solve some of the problems you mentioned here.

Bart · 7 months ago

I don't understand what the big deal is if this is how people's accounts have been set for a year and they haven't complained all this time. It seems like this wouldn't have blown up the way it has if they simply said "you've had your accounts like this for more than a year, 95% of our users have kept that setting for the past year, we are no longer able to support a setting only 5% of our users have been turning on".

A3PixitesMunier · 7 months ago

spot on! who'll build twatter, twetter, twotter, twutter, twytter?

joelfox · 7 months ago

this is what you should write about:

http://entertainment.timesonline.co.uk/tol/arts...

vk77de · 7 months ago

Yes, the centralization. And the fact that it is propriatory.

Why couldn't the micro-blogging information flow through a distributed social network of nodes? Mobile open source P2P?