Scripting News - Latest Comments in Innovating outside the 140 (Scripting News)

Re: Innovating outside the 140 (Scripting News)

sull — Thu, 22 Oct 2009 00:37:05 -0000

i'd steer this into long file name territory.
file names can commonly be up to 255 characters.
the file name becomes the message/tweet.
each user add a base "resources url" in their settings.
a publishing tool uploads a package to the users server in dedicated resources directory.
the package would be a zip/jar format.
metadata can be added in an rdf xml file inside this package.
see some prior art such as MAFF - http://maf.mozdev.org/maff-...

the API should describe whether or not this message is a file reference.

in essence, you have file sharing with long descriptive filenames ;)

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 17:43:56 -0000

In regards to A, I may not be right, and I'll happily acknowledge that possibility. However, as you say, "Twitter dies because of the intertwined relational queries that they have to spew at their database engine to generate pages and API responses." I don't know what Twitter's SQL (or noSQL) schema looks like, but based on my experience with relational databases, every new user-defined type of data you attach to a Tweet represents either a new JOIN their servers have to perform or a new column in a big table.

In regards to B, I push the idea for a client-side enclosure-like-link system because it's something no one has to wait for Twitter to enable. I could be wrong, but to me, the idea that Twitter is a coral reef means all the cool features are what get stuck atop what exists, not what come from within.

Re: Innovating outside the 140 (Scripting News)

bowerbird — Wed, 21 Oct 2009 17:43:32 -0000

always did love that resedit icon. :+)

-bowerbird

Re: Innovating outside the 140 (Scripting News)

Chuck Shotton — Wed, 21 Oct 2009 17:28:05 -0000

A) I am not sure you are right.
B) I have no idea why you are arguing with me. Neither you nor I am going to build this for Twitter. I was simply trying to point out that it is an incrementally small change to their existing APIs to accept a multipart MIME message instead of a dumb text payload.

WIth regards to A, I seriously doubt that serving static data in response to a HTTP request is going to send Twitter crashing to its knees. They seem to do that quite well with 140 character messages as it is, so it isn't because of the data size. Serving static data is all a simple HTTP server does and honestly, we figured that technology out a long time ago. I explicitly point out below that Twitter has no need to index or otherwise perform any CPU-intensive functions on the extended data. Twitter dies because of the intertwined relational queries that they have to spew at their database engine to generate pages and API responses. Incrementally adding a bit of additional data payload to their API responses only increases their bandwidth and storage costs and only for the small percentage of tweets that might have this meta data. It doesn't place any additional load on their query engines.

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 16:21:46 -0000

But it'll also tax Twitter beyond imagination. A streaming system, which is essentially what Twitter is, can't be a file server as well without some trade-offs. In the case of storing bigger chunks of data, your trade-off is speed.

This is the same problem Second Life has had for years trying to host images, audio, text, XML, etc. The solution, since 2004, has been obvious: let people host their files on their own service. If their service goes down, that's their problem and not Second Life's. Second Life's problem is running a 3D world, not hosting media.

Everything you list, from GPS coordinates, relations to other tweets, etc. can be packaged in an XML file that the client fetches and parses out. Everything stored in that XML file can be slurped up later by a service, including Twitter, for caching and redundancy. However, if you make Twitter the sole repository for any kind of extra data, then they become the sole repository for that data...including when they go down.

Re: Innovating outside the 140 (Scripting News)

Nestor Silk — Wed, 21 Oct 2009 15:56:14 -0000

One of the main questions is which part of a twitter message one takes to get to external resources. normally you would anticipate that you work with a unique id/url/tinyurl to reference additional metadata/metafunctions. but hashtags together with a simple ontology can do wonders, too. that's exactly what we did with combining twitterfeeds with ongoing sportsevents, combined together with a betting interface. Have a look there: https://labs.bwin.com/rtb/ and you will understand what i mean. So what i propose is that the whole twitter message with its semantic context compared to any ontology, whatsoever could exactly result in tweets with an interesting additional value.

For sure it's less about storing and sending references and metadata together with a tweet. As we are moving from a web of fixed places, aka URLs, to a web of flow, it will be much more about intelligent aggregation of tweet and data on the fly and context sensitive.

Re: Innovating outside the 140 (Scripting News)

Chuck Shotton — Wed, 21 Oct 2009 15:46:47 -0000

For the simplistic application of attaching some sort of media to a tweet, yes, you can take up some of the scarce payload with a URL. But what about applications where the extra data is keywords? geographic coordinates? relations to other tweets? annotations? a comment thread? multiple different language translations of the tweet?

Dave's zeroed in on a nice piece of relatively easy-to-add functionality for twitter that provides unbounded opportunities on the client side beyond just using Twitter as a near real-time picture sharing tool.

Think beyond the simplistic applications and you'll see why URLs are not sufficient. (They also make the data much more fragile as compared to keeping it intact with the original tweet -- you become subject to not only Twitter's failures, but those of the other server(s) hosting the additional data.)

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 14:36:12 -0000

The point of this post and the bulk of the comments seems to be: gosh, it'd be nice if we could include pictures and other media in out Tweets. You suggest sending a blob (http://en.wikipedia.org/wik... -- linked in case we're thinking of two different kinds of blobs) with the post and cshotton suggests a multipart MIME post. What I'm seeing here is ways to push data UP to Twitter. However, if "shoot it up to Twitter as part of a PICT resource" means something other than push data to Twitter for storage, then I apologize for misunderstanding.

I suggest the XML-RPC envelope and enclosure-like-links idea (neither of which are original ideas) because it seems like putting the burden of grabbing, storing, and rendering media is something better left to the client-side of things than the Twitter service.

Re: Innovating outside the 140 (Scripting News)

dave — Wed, 21 Oct 2009 13:38:04 -0000

You keep missing the point. People send URLs around in Twitter all the time.
You might want to assume everyone else here knows about it.

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 13:08:50 -0000

Sure, it's another form of enclosures, but that's kind of my underlying point in the post above and my XML-RPC post last night: it's VERY SIMPLE for Twitter to turn themselves into Pownce, but why would they want to copy a failed experiment?

Re: Innovating outside the 140 (Scripting News)

dave — Wed, 21 Oct 2009 12:54:58 -0000

<sarcasm>Brilliant insight. Wonder why no one thought of that
before?</sarcasm>

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 12:40:24 -0000

A plain URL in a Tweet could serve the same purpose. Have Twitter, or the client, see that the post is an address, go fetch the address, and then embed the media into the page. (Cache as needed to prevent link rot.) Facebook already does this to some degree with their update system, and ANY Twitter client could do this without changing a single thing on the service end of things.

Re: Innovating outside the 140 (Scripting News)

concordedly — Wed, 21 Oct 2009 12:01:51 -0000

I like it. Sounds like open source twittering

Re: Innovating outside the 140 (Scripting News)

dave — Wed, 21 Oct 2009 11:14:07 -0000

It just falls out of rssCloud. Enclosures allow you to attach any kind of
data to an item.

Re: Innovating outside the 140 (Scripting News)

fahrni — Wed, 21 Oct 2009 10:56:13 -0000

Agreed.

Dave, how would/could, rssCloud <http: rsscloud.org=""> married to Torndao <http: tornadoweb.org=""/>, work together to do some, if not all, of this? Tornado could supply the rich metadata, rssCloud could supply the network?

Just spitballin'.

Re: Innovating outside the 140 (Scripting News)

dave — Wed, 21 Oct 2009 10:48:08 -0000

Yes, but the URLs take up space in the 140 characters, and because they are
communicated via shortened URLs there's another domain that can fail to
cause linkrot.

Re: Innovating outside the 140 (Scripting News)

fahrni — Wed, 21 Oct 2009 10:44:20 -0000

Twitter may not be able to display these items, yet, but some clients currently do a great job of expanding short URL's, and also display pictures linked at yfrog and twitpic, etc. So you can kind of do some of that today, by convention. I know that's a bit broken, metadata to describe the media type would be super nice.

It's surprising what technologies win races isn't it? FriendFeed was awesome because of this type of support, I don't know how good their API's are, but the web based UI is far better than Twitter's in a lot of ways. Too bad is dying.

Re: Innovating outside the 140 (Scripting News)

Justin Hart — Wed, 21 Oct 2009 10:08:04 -0000

Twitter can't keep their site up passing around 140 chars at a time, I have a feeling deep payloads are out of their grasp at this point. I think if you're going to try to plant your flag on a twitter-based 'fork' solution, there is a lot of opportunity to make more seamless solution - really, Twitgoo, Twitpic, even the url shortners, etc, are partway there - giving a unique identifier to something that could be that resource fork. Its just a matter of filling out those APIs and getting apps on board to connect the pieces. Currently, the experience even degrades well for apps that dont fully hook up the pieces, its just a link. Its more like the POSIX toolchain instead of the Windows registry.

Re: Innovating outside the 140 (Scripting News)

Chuck Shotton — Wed, 21 Oct 2009 06:10:53 -0000

Twitter already has a Web-based API that takes various payloads for content. Of course, they're all bounded by the arbitrary 140 char limit, but the key is that the transaction happens via HTTP and HTTP already has a well-defined mechanism for sending multiple kinds of content in a single transaction. The "multipart" MIME types are all designed to allow inclusion of several content type or representations, each with its own sub-headers that can be used to provide "hints" about what to do with the parts (multipart/alternative can support multiple mark-up styles or languages for example, while multipart/mixed might come closer to what Dave is suggesting with mixed media types). Since the sub-parts of a multipart message each have their own MIME type as well, you can pretty much make up your own content types, just like the old Mac resource fork.

The challenge, obviously, is to get Twitter to expand its API to allow multipart object bodies and then parse them and do something sensible with the parts. If all they did was use the first text/plain item as the canonical 140 character message and then store the remainder of the multipart blob as an attribute off the message, all sorts of clients could be concocted to retrieve the associated blob and render the remaining content. The blob data doesn't have to be searchable, indexed, or otherwise incur any processing costs beyond additional physical data storage. It'd be a pretty low-impact extension to their APIs, IMO, with a HUGE upside.

An alternative would be to figure out how to sidecar an external service onto Twitter (perhaps tying message IDs together) that allowed cooperating clients to post 140 char messages to Twitter and then post the rich content to this external service. At that point, you've pretty much co-opted Twitter because it's a heck of a lot easier to add 140 character text data to a rich media repository and replicate their functionality than it probably is for Twitter to add rich media storage and APIs to their 140 characters

See http://en.wikipedia.org/wik... for some background on multipart MIME payloads.

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 00:12:24 -0000

3. It's not a centralized system run by one company.

Re: Innovating outside the 140 (Scripting News)

dave — Wed, 21 Oct 2009 00:02:09 -0000

Yes but -- it doesn't have: 1. A storage system. 2. The coral reef effect
that Twitter has.

Re: Innovating outside the 140 (Scripting News)

AndrewBurton — Wed, 21 Oct 2009 00:00:27 -0000

XML-RPC already does this with Base64-encoding.

Re: Innovating outside the 140 (Scripting News)

dave — Tue, 20 Oct 2009 19:55:29 -0000

That's why they say I write great specs Dan.

Re: Innovating outside the 140 (Scripting News)

mterenzio — Tue, 20 Oct 2009 19:55:16 -0000

Explain that a little more Chuck, if you would be so kind.

Re: Innovating outside the 140 (Scripting News)

Daniel Berlinger — Tue, 20 Oct 2009 19:47:38 -0000

That's an awesome analogy Dave. Besides, I love a spec that includes "Or whatever the fuck." That's my kinda "open".