Scripting News: Innovating outside the 140 (Scripting News)
cshotton
· 2 months ago
Ding! Well heck, a multipart MIME message could do what you want if they'd bother to parse it for you. They could render the text/plain part, and your code could interpret the blob/dave part for example.
mterenzio
· 2 months ago
Explain that a little more Chuck, if you would be so kind.
cshotton
· 2 months ago
Twitter already has a Web-based API that takes various payloads for content. Of course, they're all bounded by the arbitrary 140 char limit, but the key is that the transaction happens via HTTP and HTTP already has a well-defined mechanism for sending multiple kinds of content in a single transaction. The "multipart" MIME types are all designed to allow inclusion of several content type or representations, each with its own sub-headers that can be used to provide "hints" about what to do with the parts (multipart/alternative can support multiple mark-up styles or languages for example, while multipart/mixed might come closer to what Dave is suggesting with mixed media types). Since the sub-parts of a multipart message each have their own MIME type as well, you can pretty much make up your own content types, just like the old Mac resource fork.
The challenge, obviously, is to get Twitter to expand its API to allow multipart object bodies and then parse them and do something sensible with the parts. If all they did was use the first text/plain item as the canonical 140 character message and then store the remainder of the multipart blob as an attribute off the message, all sorts of clients could be concocted to retrieve the associated blob and render the remaining content. The blob data doesn't have to be searchable, indexed, or otherwise incur any processing costs beyond additional physical data storage. It'd be a pretty low-impact extension to their APIs, IMO, with a HUGE upside.
An alternative would be to figure out how to sidecar an external service onto Twitter (perhaps tying message IDs together) that allowed cooperating clients to post 140 char messages to Twitter and then post the rich content to this external service. At that point, you've pretty much co-opted Twitter because it's a heck of a lot easier to add 140 character text data to a rich media repository and replicate their functionality than it probably is for Twitter to add rich media storage and APIs to their 140 characters
A plain URL in a Tweet could serve the same purpose. Have Twitter, or the client, see that the post is an address, go fetch the address, and then embed the media into the page. (Cache as needed to prevent link rot.) Facebook already does this to some degree with their update system, and ANY Twitter client could do this without changing a single thing on the service end of things.
dave
· 2 months ago
<sarcasm>Brilliant insight. Wonder why no one thought of that before?</sarcasm>
AndrewBurton
· 2 months ago
Sure, it's another form of enclosures, but that's kind of my underlying point in the post above and my XML-RPC post last night: it's VERY SIMPLE for Twitter to turn themselves into Pownce, but why would they want to copy a failed experiment?
dave
· 2 months ago
You keep missing the point. People send URLs around in Twitter all the time. You might want to assume everyone else here knows about it.
AndrewBurton
· 2 months ago
The point of this post and the bulk of the comments seems to be: gosh, it'd be nice if we could include pictures and other media in out Tweets. You suggest sending a blob (http://en.wikipedia.org/wiki/Binary_large_object -- linked in case we're thinking of two different kinds of blobs) with the post and cshotton suggests a multipart MIME post. What I'm seeing here is ways to push data UP to Twitter. However, if "shoot it up to Twitter as part of a PICT resource" means something other than push data to Twitter for storage, then I apologize for misunderstanding.
I suggest the XML-RPC envelope and enclosure-like-links idea (neither of which are original ideas) because it seems like putting the burden of grabbing, storing, and rendering media is something better left to the client-side of things than the Twitter service.
cshotton
· 2 months ago
For the simplistic application of attaching some sort of media to a tweet, yes, you can take up some of the scarce payload with a URL. But what about applications where the extra data is keywords? geographic coordinates? relations to other tweets? annotations? a comment thread? multiple different language translations of the tweet?
Dave's zeroed in on a nice piece of relatively easy-to-add functionality for twitter that provides unbounded opportunities on the client side beyond just using Twitter as a near real-time picture sharing tool.
Think beyond the simplistic applications and you'll see why URLs are not sufficient. (They also make the data much more fragile as compared to keeping it intact with the original tweet -- you become subject to not only Twitter's failures, but those of the other server(s) hosting the additional data.)
AndrewBurton
· 2 months ago
But it'll also tax Twitter beyond imagination. A streaming system, which is essentially what Twitter is, can't be a file server as well without some trade-offs. In the case of storing bigger chunks of data, your trade-off is speed.
This is the same problem Second Life has had for years trying to host images, audio, text, XML, etc. The solution, since 2004, has been obvious: let people host their files on their own service. If their service goes down, that's their problem and not Second Life's. Second Life's problem is running a 3D world, not hosting media.
Everything you list, from GPS coordinates, relations to other tweets, etc. can be packaged in an XML file that the client fetches and parses out. Everything stored in that XML file can be slurped up later by a service, including Twitter, for caching and redundancy. However, if you make Twitter the sole repository for any kind of extra data, then they become the sole repository for that data...including when they go down.
cshotton
· 2 months ago
A) I am not sure you are right. B) I have no idea why you are arguing with me. Neither you nor I am going to build this for Twitter. I was simply trying to point out that it is an incrementally small change to their existing APIs to accept a multipart MIME message instead of a dumb text payload.
WIth regards to A, I seriously doubt that serving static data in response to a HTTP request is going to send Twitter crashing to its knees. They seem to do that quite well with 140 character messages as it is, so it isn't because of the data size. Serving static data is all a simple HTTP server does and honestly, we figured that technology out a long time ago. I explicitly point out below that Twitter has no need to index or otherwise perform any CPU-intensive functions on the extended data. Twitter dies because of the intertwined relational queries that they have to spew at their database engine to generate pages and API responses. Incrementally adding a bit of additional data payload to their API responses only increases their bandwidth and storage costs and only for the small percentage of tweets that might have this meta data. It doesn't place any additional load on their query engines.
AndrewBurton
· 2 months ago
In regards to A, I may not be right, and I'll happily acknowledge that possibility. However, as you say, "Twitter dies because of the intertwined relational queries that they have to spew at their database engine to generate pages and API responses." I don't know what Twitter's SQL (or noSQL) schema looks like, but based on my experience with relational databases, every new user-defined type of data you attach to a Tweet represents either a new JOIN their servers have to perform or a new column in a big table.
In regards to B, I push the idea for a client-side enclosure-like-link system because it's something no one has to wait for Twitter to enable. I could be wrong, but to me, the idea that Twitter is a coral reef means all the cool features are what get stuck atop what exists, not what come from within.
directeur
· 2 months ago
Won't feeds do that, Dave? With MediaRSS and enclosures? —Talking about twitter, have you seen socnode? http://socnode.org an attempt with a proof of concept of realtime distributed social networks?
AndrewBurton
· 2 months ago
XML-RPC already does this with Base64-encoding.
dave
· 2 months ago
Yes but -- it doesn't have: 1. A storage system. 2. The coral reef effect that Twitter has.
AndrewBurton
· 2 months ago
3. It's not a centralized system run by one company.
Daniel Berlinger
· 2 months ago
That's an awesome analogy Dave. Besides, I love a spec that includes "Or whatever the fuck." That's my kinda "open".
dave
· 2 months ago
That's why they say I write great specs Dan.
Justin Hart
· 2 months ago
Twitter can't keep their site up passing around 140 chars at a time, I have a feeling deep payloads are out of their grasp at this point. I think if you're going to try to plant your flag on a twitter-based 'fork' solution, there is a lot of opportunity to make more seamless solution - really, Twitgoo, Twitpic, even the url shortners, etc, are partway there - giving a unique identifier to something that could be that resource fork. Its just a matter of filling out those APIs and getting apps on board to connect the pieces. Currently, the experience even degrades well for apps that dont fully hook up the pieces, its just a link. Its more like the POSIX toolchain instead of the Windows registry.
fahrni
· 2 months ago
Twitter may not be able to display these items, yet, but some clients currently do a great job of expanding short URL's, and also display pictures linked at yfrog and twitpic, etc. So you can kind of do some of that today, by convention. I know that's a bit broken, metadata to describe the media type would be super nice.
It's surprising what technologies win races isn't it? FriendFeed was awesome because of this type of support, I don't know how good their API's are, but the web based UI is far better than Twitter's in a lot of ways. Too bad is dying.
dave
· 2 months ago
Yes, but the URLs take up space in the 140 characters, and because they are communicated via shortened URLs there's another domain that can fail to cause linkrot.
fahrni
· 2 months ago
Agreed.
Dave, how would/could, rssCloud <http://rsscloud.org> married to Torndao <http://tornadoweb.org/>, work together to do some, if not all, of this? Tornado could supply the rich metadata, rssCloud could supply the network?
Just spitballin'.
dave
· 2 months ago
It just falls out of rssCloud. Enclosures allow you to attach any kind of data to an item.
concordedly
· 2 months ago
I like it. Sounds like open source twittering
Nestor Silk
· 2 months ago
One of the main questions is which part of a twitter message one takes to get to external resources. normally you would anticipate that you work with a unique id/url/tinyurl to reference additional metadata/metafunctions. but hashtags together with a simple ontology can do wonders, too. that's exactly what we did with combining twitterfeeds with ongoing sportsevents, combined together with a betting interface. Have a look there: https://labs.bwin.com/rtb/ and you will understand what i mean. So what i propose is that the whole twitter message with its semantic context compared to any ontology, whatsoever could exactly result in tweets with an interesting additional value.
For sure it's less about storing and sending references and metadata together with a tweet. As we are moving from a web of fixed places, aka URLs, to a web of flow, it will be much more about intelligent aggregation of tweet and data on the fly and context sensitive.
bowerbird
· 2 months ago
always did love that resedit icon. :+)
-bowerbird
sull
· 2 months ago
i'd steer this into long file name territory. file names can commonly be up to 255 characters. the file name becomes the message/tweet. each user add a base "resources url" in their settings. a publishing tool uploads a package to the users server in dedicated resources directory. the package would be a zip/jar format. metadata can be added in an rdf xml file inside this package. see some prior art such as MAFF - http://maf.mozdev.org/maff-file-format.html
the API should describe whether or not this message is a file reference.
in essence, you have file sharing with long descriptive filenames ;)
All Comments
The challenge, obviously, is to get Twitter to expand its API to allow multipart object bodies and then parse them and do something sensible with the parts. If all they did was use the first text/plain item as the canonical 140 character message and then store the remainder of the multipart blob as an attribute off the message, all sorts of clients could be concocted to retrieve the associated blob and render the remaining content. The blob data doesn't have to be searchable, indexed, or otherwise incur any processing costs beyond additional physical data storage. It'd be a pretty low-impact extension to their APIs, IMO, with a HUGE upside.
An alternative would be to figure out how to sidecar an external service onto Twitter (perhaps tying message IDs together) that allowed cooperating clients to post 140 char messages to Twitter and then post the rich content to this external service. At that point, you've pretty much co-opted Twitter because it's a heck of a lot easier to add 140 character text data to a rich media repository and replicate their functionality than it probably is for Twitter to add rich media storage and APIs to their 140 characters
See http://en.wikipedia.org/wiki/MIME#Multipart_mes... for some background on multipart MIME payloads.
before?</sarcasm>
You might want to assume everyone else here knows about it.
I suggest the XML-RPC envelope and enclosure-like-links idea (neither of which are original ideas) because it seems like putting the burden of grabbing, storing, and rendering media is something better left to the client-side of things than the Twitter service.
Dave's zeroed in on a nice piece of relatively easy-to-add functionality for twitter that provides unbounded opportunities on the client side beyond just using Twitter as a near real-time picture sharing tool.
Think beyond the simplistic applications and you'll see why URLs are not sufficient. (They also make the data much more fragile as compared to keeping it intact with the original tweet -- you become subject to not only Twitter's failures, but those of the other server(s) hosting the additional data.)
This is the same problem Second Life has had for years trying to host images, audio, text, XML, etc. The solution, since 2004, has been obvious: let people host their files on their own service. If their service goes down, that's their problem and not Second Life's. Second Life's problem is running a 3D world, not hosting media.
Everything you list, from GPS coordinates, relations to other tweets, etc. can be packaged in an XML file that the client fetches and parses out. Everything stored in that XML file can be slurped up later by a service, including Twitter, for caching and redundancy. However, if you make Twitter the sole repository for any kind of extra data, then they become the sole repository for that data...including when they go down.
B) I have no idea why you are arguing with me. Neither you nor I am going to build this for Twitter. I was simply trying to point out that it is an incrementally small change to their existing APIs to accept a multipart MIME message instead of a dumb text payload.
WIth regards to A, I seriously doubt that serving static data in response to a HTTP request is going to send Twitter crashing to its knees. They seem to do that quite well with 140 character messages as it is, so it isn't because of the data size. Serving static data is all a simple HTTP server does and honestly, we figured that technology out a long time ago. I explicitly point out below that Twitter has no need to index or otherwise perform any CPU-intensive functions on the extended data. Twitter dies because of the intertwined relational queries that they have to spew at their database engine to generate pages and API responses. Incrementally adding a bit of additional data payload to their API responses only increases their bandwidth and storage costs and only for the small percentage of tweets that might have this meta data. It doesn't place any additional load on their query engines.
In regards to B, I push the idea for a client-side enclosure-like-link system because it's something no one has to wait for Twitter to enable. I could be wrong, but to me, the idea that Twitter is a coral reef means all the cool features are what get stuck atop what exists, not what come from within.
that Twitter has.
It's surprising what technologies win races isn't it? FriendFeed was awesome because of this type of support, I don't know how good their API's are, but the web based UI is far better than Twitter's in a lot of ways. Too bad is dying.
communicated via shortened URLs there's another domain that can fail to
cause linkrot.
Dave, how would/could, rssCloud <http://rsscloud.org> married to Torndao <http://tornadoweb.org/>, work together to do some, if not all, of this? Tornado could supply the rich metadata, rssCloud could supply the network?
Just spitballin'.
data to an item.
For sure it's less about storing and sending references and metadata together with a tweet. As we are moving from a web of fixed places, aka URLs, to a web of flow, it will be much more about intelligent aggregation of tweet and data on the fly and context sensitive.
-bowerbird
file names can commonly be up to 255 characters.
the file name becomes the message/tweet.
each user add a base "resources url" in their settings.
a publishing tool uploads a package to the users server in dedicated resources directory.
the package would be a zip/jar format.
metadata can be added in an rdf xml file inside this package.
see some prior art such as MAFF - http://maf.mozdev.org/maff-file-format.html
the API should describe whether or not this message is a file reference.
in essence, you have file sharing with long descriptive filenames ;)