Scripting News: How to fix URL-shorteners (Scripting News)
vrypan
· 4 months ago
I don't think that the short domain should be controlled by the creator of short URLs, but instead by the owner of the (destination) long domain.
That's because the person that's mostly interested in keeping a short URL alive and functional is the owner of the domain it points to. This is how my solution http://urlborg.com/a/urlborg_xml/about/ works. And it's nice, because "regular" users don't have to bother with such things: when the user urlborg.com, the generated short URL will be using a custom short domain, if the owner of the destination URL has registered one.
my $.02
kf6nvr
· 4 months ago
This was the main problem I saw with this solution, actually. But here's the catch:
Regular Joe user wants to create a short link. Site YYY.com, seeing the light of the short URL future, has shortlinks for all of their stories via ZZ.com/xxx. They control YYY.com and ZZ.com, so can keep ZZ.com up as long as they want. This is great and something Company YYY is providing for all their users. I suspect this is actually the use case Dave envisions, but _every single site has to implement it_ -- that won't happen.
Tech savvy Jane user wants to create a short link. Site RRR.com is old fashioned and only has one set of URLs: extraordinarily long ones. Jane creates JJ.com (using the above method) and creates links to RRR.com. These are Jane's links and these links will work so long as Jane wants. This helps Jane as she can grow her own brand, track how her own links spread, etc, but hurts RRR.com when Jane closes down JJ.com. Jane doesn't care, though. This is not so great, overall, but Jane would prefer this.
Real world users are stuck with link rot at some point. But as Louis Gray pointed out in a recent Google Reader comment, very, very few old links are clicked on.
Andrew Grant
· 4 months ago
There are three problems that I can see:
1) This is switching out one point of failure (bit.ly) for another (S3). Actually it isn't even S3, it's the person or entity who controls that S3 bucket.
2) There's a finite amount of available domains that are short without being nonsensical.
3) Setting this up (registering + DNS + finding an S3 provider) is outside the scope of what 99% of twitters users either can or wish to deal with.
The only way to solve these three problems in a manner that's likely to gain traction is to provide them as part of a service, but then we're back to the situation where if that service fails then links are dead.
I think an alternative is to recognize bit.ly and peers for what they are - a DNS service that requires persistence. These services should be encouraged to provide static dumps of their mappings that can periodically be retrieved and used by third parties where resolution is required. E.g Google calculating pagerank, or even for handling a 404 error.
dave
· 4 months ago
To your #1 -- back up the data locally. If S3 goes away, move the data to your Apache server and remap the CNAME. The data moves easily.
Don't know about your #2 and #3 except come on -- getting an S3 account is as easy as getting an Amazon account, and that's not a huge barrier. If you're worried about the people who can't get on Amazon, okay -- got it.
And you just need a sub-domain.
Nonsense names? That has not been a problem so far in URL-shorteners. :-)
Dave
Andrew Grant
· 4 months ago
Your suggestion that setting this up is achievable by anyone who can create an Amazon account is a little ridiculous. The fact that you offered to write a "how to" is clear evidence of this, I don't think anyone has ever done that for creating an Amazon account :)
Any solution that requires people to perform local backups to preserve integrity of shortlinks is a non-starter. Any solution that requires people to do more than type in a url and hit a button to get started is a non-starter.
There are hundreds of thousands, if not millions, of people out there generating short urls every day. Most of them don't give a fig whether tr.im or bit.ly ceases to exist in six months, nor spend 15-30 minutes fooling with DNS/CNAME/S3. To have any hope of being worthwhile any solution that is developed needs to recognize this.
Ben Buckman
· 4 months ago
I don't quite understand. What kind of URL shortening infrastructure (database, software) does the bucket have? Or are the redirects all done with CNAMES, with no URL data stored in the bucket? (That would lead to thousands of CNAME records for a single domain, I wonder if there's a limit.) The DNS records here don't seem to include a CNAME record for each short URL: http://network-tools.com/default.asp?prog=dnsre...
Could you clarify? Thanks!
George
· 4 months ago
I looks like they are using html meta tags to do the redirection.
One idea behind dropping the URLs in your bucket is that you could now make the URLs portable between URL shorteners that provide tracking stats.
For example, if you shortened a link with tr.im and they implemented this technique, you could then move to Adjix, in the future and Adjix would pull your shortened URL out of the HTML meta-refresh page, stored in your bucket, track the click, and then perform a 301 redirect.
In other words, not only could your bucket perform the live redirect if you wanted, it could also be a data store for your links.
sull
· 4 months ago
I'm glad that the wiring under the board is increasingly being exposed. Despite many code examples and articles on how to actually build a short trackable url redirect over the years, it's definately a hot topic now that short urls are so pervasive. And I agree that it should not be this way.... at least the reliance on centralized commercial endeavors. You can of course have your own short domains for your own content. Wordpress has several plugins and also the wp.me service. That is all moot to the concerns of link rot that is at the core of this topic.
So, using Amazon s3 is a great idea. Using CNAME is certainly an option. CNAME is known to be a bit costly and can cause a little lag but no doubt a common and useful way to map (and mask) domains to each other. In the early days of videoblogging/podcasting, Feedburner got very popular (2004/2005ish). I know, Dave, you have mentioned Feedburner as a reference to this topic several times. I too was quite vocal about people using their own domains and not publishing the feedburner domains (for obvious reasons).
What should also be mentioned here is the use of A Records instead of CNAME so you are not reliant on creating subdomains. This is common for services that let you use your own domaiin to map to your url on their service. I have done this with Tumblr in the past.
Thanks for the update. And btw, I love your new ou.rs domain :)
Joe Moreno
· 4 months ago
We tried out the A record approach, but it didn't work. Perhaps we overlooked something. Amazon S3 static IPs change from minute to minute and you can't set up a CNAME for a domain's root name. :^(
sull
· 4 months ago
Yeah for your method with S3, A Records won't work. For other services that will let you use your own domain, an A Record is likely the way they will provide the service.
brianryer
· 4 months ago
The actual resolution is in the particular brand of cyber-crisco used, the CNAME ref is just to keep the shrtURL decoupled from the repository. If the URL shortening system in use wants to change repository (db) for whatever reason, they can and the URL's don't break. If the dev using the URL shortening service wants to change services (location of via CNAME) for whatever reason, they can and the URL's don't break.
As they say across the pond, "Brill"!
bertilhatt
· 4 months ago
Good idea — except for the long tail of technical ability. :)
The Web was fantastic because it was easy to edit; without a service like bit.ly to accomodate less savy users looking at less savvy writers, you are left with them being less quoted in Twitter. I completely agree that massive content platform should have that yesterday (if not because you have a creative friend, but because they want to control their own traffic).
Joe Moreno
· 4 months ago
You're absolutely right - I wish this was simpler to do. We tried to simplify how to share your S3 bucket here: http://adjix.com/tt3b
Sharing Amazon S3 buckets with third parties opens up a lot of possibilities. For example, you could have Twitter store every Tweet you've ever sent, as an HTML page, in your bucket. Or, Blogger could store every post you've blogged in your bucket. Etcetera, etcetera.
Drew Kime
· 4 months ago
Can I just take this opportunity to mention a growing peeve of mine? Why in the name of all that's holy would people use shortened URLs in blog posts?
I'm not talking about mini-posts that were multicasted to Twitter and Blogspot at the same time. I'm talking 500-1,000 word posts that have bit.ly URLs in the middle of them. What the hell for? All it does is obscure where you're going, which makes me *much* more likely to not click them.
Maybe they enable better tracking? I'm not sure. Because I already track all my outbound links. I just don't understand why someone would take the extra effort to go out and obfuscate their links, when length is clearly not an issue.
[pant ... pant ... ]
Okay, sorry. That's been building up for a while now.
kurtiss
· 4 months ago
I like the simplicity of this idea. However, doing things like modifying your tracking code (say switch from Google Analytics to another provider) or the way that you decide to refresh becomes a big task where it didn't used to be. Big, but not insurrmountable. One thing you might think of adding is a sort of shortened url microformat that would make it easy to build tools to automate the process of traversing and modifying all your static redirect files. I saw <link rel="canonical" ...> in there, which might be half of it. Maybe just add a <link rel="self" href="http://tmp.loose.ly/jmxe" /> and that'd be enough.
dave
· 4 months ago
Actually I think I got that part of it wrong.
You map the CNAME to the URL-shortener. Later, if they go away or you decide to leave them, you point the CNAME to the Amazon bucket.
Then you don't need to change the API. And I think we need to see some convergence there. Since I've built on the tr.im API, and since it's going open source, that seems the most logical to me. I imagine that bit.ly, the 800-pound gorilla won't see it that way.
Dave
Wc_WhoTheLuckyStiff
· 4 months ago
I understand what you say and I couldn't agree more to that we should eliminating centralization, as you've discussed about microblogging. As for URLshorteners, this idea can do great help to the current situation.But basically I think it's a more meaningful solution to loosen the limitation of character number on microblogging.
Joe Moreno
· 4 months ago
@Drew Kime, why people would want to shorten a link in a blog post is an excellent question.
One key reason that people want to shorten URLs in blog posts is so they can see how many clicks each link receives, where it came from, by whom, etc. This is easiest for the layperson to do when using a URL shortener that tracks stats, as you mentioned in your post.
It's not only the fact that the link is shortened but rather the fact that it's trackable and you can brand your link by using your own domain name.
If you shortened a link with Adjix and you have your CNAME point to Adjix, then Adjix will track click stats while saving your links in your S3 bucket. If Adjix goes away, you can then just point your CNAME back to your S3 bucket and the link lives on.
One idea behind dropping the URLs in your bucket is that you could now make the URLs portable between URL shorteners that provide tracking stats.
For example, if you shortened a link with tr.im and they implemented this technique, you could then move to Adjix, in the future and Adjix, would pull your shortened URL out of the HTML meta-refresh page, track the click, and then perform a 301 redirect.
In other words, not only could your bucket preform the live redirect if you wanted, it could also be a data store for your links.
fwoncn
· 4 months ago
Thanks @Joe Moreno 's coment. It remainds of another important function which makes people use URLshortener. I almost forget it. Acknowledging this, I'm for this idea.
Joe Moreno
· 4 months ago
For the technical audience, here are some examples of a link shortened with Adjix:
4. http://urlpuppy.com/ty35 Shortened link with registrar domain name forwarding. Traffic stats logged by S3 bucket logging. One extra step is required to get #4 to work since you can't set a domain name's root to a CNAME. You must have your registrar forward your domain name to http://s3.amazonaws.com/YourS3BucketName.
Don't forget that you'll need to share your S3 bucket with Adjix to get this to work: http://adjix.com/tt3b
On the YouTube video around 12:20 the presenter begins talking about how the site reputation does not transfer between domains. So having canonical tags pointing to domains anywhere other than the current domain or sub-domain causes Google to ignore it.
ex. of good use of canonical element and bad. a.blah.com/1137 -> blog.blah.com/2009/08/19/some-post-about-stuff ad.vu/1137 -> blog.blah.com/2009/08/19/some-post-about-stuff
sull
· 4 months ago
Though this is a nice approach, I'd first encourage the use of self-hosted solutions. I think we all know that the actual code involved to make a short trackable url is simple. Their are various methods but none are complicated. Some won't work as well as others.
But we dont need adjix or any 3rd party involved (besides your web host and domain registrar) unless you find value in that, and some will.
tr.im is releasing their code and data for this reason while also providing services/support.
The good thing is that their are so many options for those who truly need to care about this stuff (until we can move on from this dark time of cryptic url usage ;)
Joe Moreno
· 4 months ago
I would debate that we don't need any 3rd parties involved.
I believe that we need *all* 3rd party link shrinkers involved, and using the technique outlined here, in order to make your links portable for when link shrinkers like tr.im or zi.ma close up shop.
sull
· 4 months ago
right "unless you find value in that, and some will."
i'm all for all service providers to do everything they can to make sure that the data is open and accessible. whatever approach works, works.
i think it's easy to mesh the discussion because their are those who:
a) have their own website/domain and want short trackable urls for their own content (ie. blog posts). b) have their own website/domain and want short trackable urls for ANY content (any url on the web). c) just want to quickly share links on social services like twitter and dont care about tracking, link rot etc. d) run a service as a data store, stats collector and as a result, a traffic handler and do not want the trend of DIY to gain much momentum. e) community-owned services like rp.ly (who first conjured up the idea) and soon to be tr.im (unless they change their mind again ;)
and any other variations. it's easy to misstep with comments :)
playerx
· 4 months ago
How would I be able to move my current "http://sho.rtu.rl/ty35" link to your service?
From what I've seen playing around with your solution, you would need everyone to work around the same name space, base32,64, 1-12? characters, otherwise there would be overlap when moving your short link files between services when they "close up shop" and you will have to create more sub-domains each time. {a,b,c,d}.ho.st
Joe Moreno
· 4 months ago
The service will work for you, going forward, if you follow the steps outlined below.
Ideally, you'd want all URL shorteners to use the same name space to prevent a token collision. Otherwise, each URL shortener will need to check your bucket to see if you've already created a link with the same token before shortening your link. If so, then the URL shortener will need to choose a new token which isn't really a big deal - most provide that service, now, by letting you pick your own token: http://tinyurl.com/ReallyCoolLink
Here's what you'll need to do:
1. Create an S3 bucket named sho.rtu.rl.
2. You'll need to share your bucket with Adjix so it can save a copy of each link to your bucket. Details here: http://adjix.com/tt3b
3. Log into your Adjix account and click Edit Profile. At the bottom, just enter the name of your bucket.
That's it - each link that you now shorten with Adjix will be copied to your bucket.
If you want Adjix to track click stats then you'll need to point the CNAME for sho.rtu.rl to partner.adjix.com.
If Adjix goes down, for any reason, then you can simply change the CNAME for sho.rtu.rl to sho.rtu.rl.s3.amazon.com and the link will continue to work.
Please don't hesitate to let me know if this is confusing.
Thanks, Joe
playerx
· 4 months ago
"Ideally, you'd want all URL shorteners to use the same name space to prevent a token collision."
Using all the same token name space, while everyone has their own FQDN would severely handicap the ability for any one URL shortener to act alone in provisioning tokens without a clearing house. I sure as hell don't want to see an ICANN of shortURL.
It's too late for collision prevention, no one does it, and it's too time constraining. Also, to manually choose new tokens and then update all the accessible HTML elements for links already distributed sounds like a nightmare.
All the manual DNS changes can be a hassle should your provider not allow you to adjust SOA or individual TTL refresh, expiry entries.
Here's a tip, if we point our CNAME to your partner.adjix.com; you should be the one to update DNS records when you fail, not us.
mterenzio
· 4 months ago
Is this as search engine friendly as as 301 redirect? Yes, it is now your domain that's getting the traffic but your individual pages will now be split between the long url and short url in the eyes of an engine, no?
BTW, I'll probably use the method anyway.
One other thing. You will need a record of the shorturls. They'll never break, but when you switch providers, the next system that generates your shorturls will need to know whether the shorturl they are creating is in fact unique, right?
Joe Moreno
· 4 months ago
The idea is that a third party (i.e. Adjix, tr.im, tinyurl) would follow this format by pulling your link from your bucket, track the stats, and then issue a 301 redirect.
In other words, think of your bucket as the link backup data store , with the host CNAME point to, say partner.adjix.com. You'd only have to deal with the non-301 redirect if something happened to your link shrinking partner.
mterenzio
· 4 months ago
So if one switches providers, the second provider will generate a shorturl, make an HTTP request for the shorturl. If a 200 is returned, they must generate a new URL because they know that one was taken. If not they can use the newly generated URL.
That's what I mean about having a record of the shorturls. In case you do switch providers, they need to know that any new shorturl they create is a unique one.
Joe Moreno
· 4 months ago
You're thinking! Yes, the new provider would have to do a quick check to avoid a namespace collision of the token. The only way around this is if one authority handed out the tokens. Perhaps that's where 301Works.org will enter the game.
sull
· 4 months ago
it is true that meta refresh is not optimal for seo and load time (as samj describes).
a system i built takes advantage of static files (just txt files) and optional mysql data recording/backup. the static files, a unique one for each url, can be easily sent to s3 or any other server as a backup. they can even be served via rss feed so that anyone can subscribe and generate more backups ( i thought dave would like that idea ;). if your site crashes, your backup nameserver could direct people to your mirror server(s). wont get into redundancy here as their are various approaches... like just using Amazon EC2 or Linode etc. Point is, i like static file approach. though i dont create static html pages with the meta refresh like adjix, i still am able to track clicks, referrers and other data with no problem during the 301redirect.
sull
samj
· 4 months ago
Whoa, HTML redirects are far more expensive than HTTP and URL shorteners already add significant overhead... I'd be interested to see what kinds of delays this adds (I suspect that real world tests would show it's actually *worse* for performance than the status quo).
My strong preference is to stick with the same domain and compress the path and query string as much as possible. For example:
That way your user agents are saved at least one recursive DNS query (with CNAMEs it's two), a TCP 3-way handshake and a HTTP transaction and can get everything done in one go with keep-alive. In that case the performance impact would be negligible, and you can still do your tracking.
Whatever the mechanism we should definitely advertise the result using the rel=shortlink standard.
Sam
Joe Moreno
· 4 months ago
You're right about the fact that HTML redirects are more expensive. Unfortunately, that's the only way to save a link to an Amazon S3 bucket in a way that it will work. However, the idea of doing this is so that links can be portable between URL shorteners that track stats. For example, let's say that your link was created by tr.im and stored in a bucket called go.samj.net. If tr.im closed up shop, you could then adjust your CNAME to point to Adjix and Adjix would pull your link from your bucket, track the stats, and do a 301 redirect.
samj
· 4 months ago
We should strive to make URL shortening invisible. DNS queries (esp for CNAMEs), TCP handshakes and HTTP requests are all very expensive and also all very unnecessary. On the other hand, fixing WordPress, Drupal or whatever CMS it is you're using to generate its own links and host them at the root is surely going to be the better/faster/cheaper way to accomplish this goal. Sure if you want to be fancy you can get your own domain but it's going to hit your performance and cost you more in the process (that is, only really necessary for long domains).
I think more to the point, before we go about declaring "$solution has solved all the problems with URL-shorteners" we need to assess the proposal thoroughly (look how far rev=canonical got before being reeled back in).
Sam
Joe Moreno
· 4 months ago
Ahh, now you're talking! The ideal solution should be part of the Internet. Perhaps an RFC needs to be sent out? I'm sure there's a PhD student working on something like this.
samj
· 4 months ago
I'm not sure it needs a PhD student, rather some common sense and a bunch of eyeballs. But yes, it's a good idea. I was actually thinking of whipping up an I-D for rel=shortlink but there's not enough hours in the day.
Sam
Steve
· 4 months ago
I'm out. My dssertation is about something else. Entirely. See you in May.
Yaniv Golan
· 3 months ago
Good point. Using the same domain name will improve performance, and reduce use confusion. It will also fix the incentives - the folks with the most incentive to keep links from rotting are the content publishers. See http://yaniv.golan.name/blog/2009/08/18/of-brew....
We'll be losing the "tracking" aspect of the current 3rd party short URLs, but this tracking is already quite unreliable given the non-unique nature of the short URLs. Perhaps it would be in the interest of Twitter etc to provide this click tracking capability. It'd still be unreliable, but would be far less damaging to the net infrastructure.
samj
· 3 months ago
Domains are usually short enough to be used as is (e.g. http://dell.com/az93 or http://microsoft.com/win7) and the result is clearly a much higher quality link that tells the reader something about the content before they click.
Regarding 3rd party tracking, what is the use case really (beyond feeding the egos of people whos egos don't need feeding). I realise that there are marketing benefits for the few of us who are in that industry but the cost of creating millions of links to each resource (rather than one or two - the canonical and short links) are many and not always obvious. I found today that browsing a twitter search for new content was almost impossible for example because my browser's colouring of visited links was broken.
Incidentally Wordpress blog software will be including shortening services (perhaps using wp.me) in its stats module soon... these will presumably capture stats as well, thus solving the problem.
Sam
Yaniv Golan
· 3 months ago
Oh, I agree re existing domain names being short enough and the value to the reader - that was my point, I guess it wasn't clear enough.
Re 3rd party tracking, yes, the value is the ego boost, and also the ability to understand how something spreads across the net and who are the real influencers. For companies like http://tra.cx this is invaluable information. It could be provided by the conversational media (e.g. Twitter) or by the publishing infrastructure (e.g. Wordpress).
And of course, once we have multiple stat-providers, there will be companies that will create services that will aggregate all these stats in one place, creating a new industry segment, and so the cycle continues...
kellan
· 4 months ago
Dave, I know this is just an example you're using, but you do know there that photo already has a shorter URL, right? http://flic.kr/p/6LeeFL
dave
· 4 months ago
As you said it's an example.
dave
· 4 months ago
There's no need to avoid naming collisions betw the different URL shorteners. Here's why.
Suppose I'm using the shortener miniu.rl and one day they go out of business so I point the CNAME at my bucket on Amazon instead of on their host.
Then I go create an account on newmi.ni and all my new URLs go there.
Just like when I start a new blog. I leave the old one where it is. No need to merge the two (though people often get hung up on doing exactly that).
BTW, as I said in the post, worth repeating because people are saying that it's a flaw, just trading one weak spot for another. If Jeff Bezos changes his mind and decides to shut down S3, I was smart, I backed up my bucket on my laptop hard disk. Then I got an account at Rackspace and moved the folder there, and pointed the CNAME at my server over there. So it's not trading one weakness for another, it is giving me control of my data. Big diff.
Derik
· 4 months ago
Not trying to be a solution giver, I think the keys to the problem may indeed be something in DNS. My first exposure with DNS and the "bind" was when it was being hacked. hmm, I think I hacked it myself to make something happen. All the same, I just was a poe (plain old engineer) in those days (1973-2009 ... college to now) when we just did something useful. I think am still just a poe anyhow but not sure how to be a solution giver in this new world. I think the next step may indeed be a look at where in the stack this issue lives (I mean the architecture stack versus something living elsewhere). It seems that a url (or uri or iri) is a plain old name (pon). Our fathers (them DNS hackers) invented the domain name system. They even invented something called a mail exchange thing (so that people who ran away could alias them servers and themselves to somewhere else ... yah, I haven't really remembered everything about MX anyhow). I think urls are sometimes like peoples personalities. Sometimes, people want to drop the past and go with the future. So, losing some urls may indeed be a good thing. I know I need to find some I lost over time anyhow and then "clean" them out. So, where on earth does a name to something else actually live in the stack these days? I'd say in the name giver. Anyhow, I asked someone about this issue and he gave me this http://bit.ly/RFSVW ... so I eat the cooking too
Jud Valeski
· 4 months ago
using cnames as a level of indirection on top of shared data would be great. only problem is that the users who care enough about their shortened links comprise a small population of the folks (knowingly or not) doing the actual shortening (most do it blindly through some tool like twitter). as a result, they have no interest, sadly, in protecting their information w/ an owned vanity/personal name-space in dns land.
unclear what happens if I stop paying my domain registration fee (the one that has the cnames in it). feels like dead links will still be out there.
micahwittman
· 4 months ago
The shared S3 bucket with adjix solution is already working nicely! I wrote it up briefly on friendfeed http://go.wittman.org/3tjf . Needless to say, great post, Dave, and the ensuing discussion.
Henri Asseily
· 4 months ago
Hypothetically, technically the simplest, fastest and most effective redirection would be something straight in the DNS. Let's say there's a new httpr:// and httpsr:// (for http(s) redirection) URI. The browser would take that domain, do a NAPTR query that would return one (or multiple) new URLs that it would follow. It's just like sip, ENUM, etc... but takes a url and turns it into another url. So httpr://3tjf.me.com would get the browser to do a DNS NAPTR call to 3tjf.me.com which would return the new url, and it would follow that.
Anyway, this is all extremely hypothetical for many obvious reasons... :) (including the lack of tracking offsetting the gain in speed and reliability)
mckoss
· 4 months ago
I understand the motivation, but don't like the solution very much. The problem is not so much that a third party is providing the redirection service, as it is that the redirection data and URL name are not portable. This can be solved w/o the technical deficiencies of this S3 hack.
[Does anyone else find it ironic that we're using a 3RD PARTY COMMENTING SERVICE to discuss how to reduce reliance on 3rd party link shorteners?]
- URL Shorteners should provide either a feed or batch backup solution to export data to another link redirection service. - Accept CNAME redirections as google does with their ghs.google.com service. Users could then register domains that they control and own (and can point them elsewhere if the service expires).
It would be far better to standardize on an export format and data/domain portability, than it would be for individual users owning and managing these kludgey S3 buckets.
webwright
· 4 months ago
Other than SMS (where it makes sense to selectively shorten URLs) can you explain to me why we want URL shorteners at all? In other words-- if I don't use SMS (I don't), why should I _ever_ see a shortened URL? Who does it benefit?
Joe Moreno
· 4 months ago
It's not just the shortening, but also being able to track traffic stats. See earlier comment addressing this exact subject in this thread.
In brief - I'd say to primary motivation is tracking/analytics when you are sharing links to sites that are not your own. Another reason I didn't list there, are that you are adding "features" to your link:
I use URL shorteners for giving people links offline. It's especially handy with tr.im's ability to name the links yourself: writing (to give a fake example) "tr.im/joezittsbook" on a bar napkin is a lot easier and less prone to human error than writing 'www.josephzitt.com/foo/bar/baz/this-is-the-name-of-my-book-draft-3.300dpi.pdf"
Drew Kime
· 4 months ago
Now that's actually a good reason for them. That's not what I see more often, not by a long shot, but it probably should be.
Joe Moreno
· 4 months ago
Perhaps I'm misunderstanding what you wrote, but URL name is portable. You have full control of the bucket to copy, backup, etc. The next step is to make the URL data portable, which isn't hard - we'd just need to standardize it between URL shorteners.
mckoss
· 4 months ago
The CNAME part of this solution is fine - I was mainly complaining about the fact that users would have to create and manage an S3 account, and that the redirection method breaks down (no 301's).
So if one of the mainstream (or other) shorteners would allow me to use a private CNAME record when creating links, and would allow me to export my data - problem solved (in the "right" way).
johnmaher
· 4 months ago
"URL-shorteners centralize linking, and make it more fragile, and more controllable....It must be possible for the user to own and control the domain his or her URLs live at...I would be happy to write a howto that explains."
That would be a really good idea. Then each of us could have our own software that shortens our URLs and those we send along, reduce the unnecessary centralization, and provide smaller URLS that can fit within the Twitter parameters. There are URL shorteners out there--mostly in the internet marketing arena--and I think they have much broader use than they're currently getting. Of course, we ARE a lazy species...
bradconnell
· 4 months ago
Couldn't this whole issue be solved by using hyperlinks?
ryanh
· 4 months ago
Does twitter really need the 140 limit anymore? Everybody I know that uses twitter either uses twitter.com or a smartphone app, and so for these people the limit is mostly an inconvenience.
Kent Larsson
· 4 months ago
One problem with any of these solutions is that the short URL often becomes quite long. If I have a site withareallylongdomainname.com and have s.withareallylongdomainname.com/a6x then that's pretty long too. To solve this case I would have to register another, shorter, domain name and use it instead for my short URL's. The domain name doesn't even have to be that long for the user to want another one for short URL's, as long as the "short" URL is longer than what current URL-shortening services offer we'll have a disadvantage right there. If a lot of web citizens have to register another domain name and administer some extra technology for URL shortening then there will be more dependencies to make each link work, which might cause link reliability issues. On top of that I don't think it will take off if it's complex to the average user.
After you set up your own Andrew... Like Dave said.. now think how to keep it alive for a long time... ;)
Nice Sharing
· 2 months ago
Also check out http://www.nicesharing.com which allows you to combine several URLs into a single short URL. It’s really useful if you want to add background music to a Picasaweb or Flickr slideshow. Or just see 2 sites together with a single URL.
All Comments
That's because the person that's mostly interested in keeping a short URL alive and functional is the owner of the domain it points to. This is how my solution http://urlborg.com/a/urlborg_xml/about/ works. And it's nice, because "regular" users don't have to bother with such things: when the user urlborg.com, the generated short URL will be using a custom short domain, if the owner of the destination URL has registered one.
my $.02
Regular Joe user wants to create a short link. Site YYY.com, seeing the light of the short URL future, has shortlinks for all of their stories via ZZ.com/xxx. They control YYY.com and ZZ.com, so can keep ZZ.com up as long as they want. This is great and something Company YYY is providing for all their users. I suspect this is actually the use case Dave envisions, but _every single site has to implement it_ -- that won't happen.
Tech savvy Jane user wants to create a short link. Site RRR.com is old fashioned and only has one set of URLs: extraordinarily long ones. Jane creates JJ.com (using the above method) and creates links to RRR.com. These are Jane's links and these links will work so long as Jane wants. This helps Jane as she can grow her own brand, track how her own links spread, etc, but hurts RRR.com when Jane closes down JJ.com. Jane doesn't care, though. This is not so great, overall, but Jane would prefer this.
Real world users are stuck with link rot at some point. But as Louis Gray pointed out in a recent Google Reader comment, very, very few old links are clicked on.
1) This is switching out one point of failure (bit.ly) for another (S3). Actually it isn't even S3, it's the person or entity who controls that S3 bucket.
2) There's a finite amount of available domains that are short without being nonsensical.
3) Setting this up (registering + DNS + finding an S3 provider) is outside the scope of what 99% of twitters users either can or wish to deal with.
The only way to solve these three problems in a manner that's likely to gain traction is to provide them as part of a service, but then we're back to the situation where if that service fails then links are dead.
I think an alternative is to recognize bit.ly and peers for what they are - a DNS service that requires persistence. These services should be encouraged to provide static dumps of their mappings that can periodically be retrieved and used by third parties where resolution is required. E.g Google calculating pagerank, or even for handling a 404 error.
your Apache server and remap the CNAME. The data moves easily.
Don't know about your #2 and #3 except come on -- getting an S3 account is
as easy as getting an Amazon account, and that's not a huge barrier. If
you're worried about the people who can't get on Amazon, okay -- got it.
And you just need a sub-domain.
Nonsense names? That has not been a problem so far in URL-shorteners. :-)
Dave
Any solution that requires people to perform local backups to preserve integrity of shortlinks is a non-starter. Any solution that requires people to do more than type in a url and hit a button to get started is a non-starter.
There are hundreds of thousands, if not millions, of people out there generating short urls every day. Most of them don't give a fig whether tr.im or bit.ly ceases to exist in six months, nor spend 15-30 minutes fooling with DNS/CNAME/S3. To have any hope of being worthwhile any solution that is developed needs to recognize this.
The DNS records here don't seem to include a CNAME record for each short URL: http://network-tools.com/default.asp?prog=dnsre...
Could you clarify? Thanks!
Ex.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title></title>
<META HTTP-EQUIV="Refresh" CONTENT="0;URL=http://www.flickr.com/photos/scriptingnews/3782292158/">
<meta name="robots" content="noindex"/>
<link rel="canonical" href="http://www.flickr.com/photos/scriptingnews/3782292158/"/>
</head>
<body>
{Google analytics code}
</body>
</html>
The links are stored in your S3 bucket as a static HTML page with a meta-refresh. Winer wrote about it here:
http://www.scripting.com/stories/2009/04/27/adj...
One idea behind dropping the URLs in your bucket is that you could now make the URLs portable between URL shorteners that provide tracking stats.
For example, if you shortened a link with tr.im and they implemented this technique, you could then move to Adjix, in the future and Adjix would pull your shortened URL out of the HTML meta-refresh page, stored in your bucket, track the click, and then perform a 301 redirect.
In other words, not only could your bucket perform the live redirect if you wanted, it could also be a data store for your links.
Despite many code examples and articles on how to actually build a short trackable url redirect over the years, it's definately a hot topic now that short urls are so pervasive. And I agree that it should not be this way.... at least the reliance on centralized commercial endeavors. You can of course have your own short domains for your own content. Wordpress has several plugins and also the wp.me service. That is all moot to the concerns of link rot that is at the core of this topic.
So, using Amazon s3 is a great idea. Using CNAME is certainly an option. CNAME is known to be a bit costly and can cause a little lag but no doubt a common and useful way to map (and mask) domains to each other. In the early days of videoblogging/podcasting, Feedburner got very popular (2004/2005ish). I know, Dave, you have mentioned Feedburner as a reference to this topic several times. I too was quite vocal about people using their own domains and not publishing the feedburner domains (for obvious reasons).
What should also be mentioned here is the use of A Records instead of CNAME so you are not reliant on creating subdomains.
This is common for services that let you use your own domaiin to map to your url on their service. I have done this with Tumblr in the past.
Thanks for the update. And btw, I love your new ou.rs domain :)
Amazon S3 static IPs change from minute to minute and you can't set up a CNAME for a domain's root name. :^(
As they say across the pond, "Brill"!
The Web was fantastic because it was easy to edit; without a service like bit.ly to accomodate less savy users looking at less savvy writers, you are left with them being less quoted in Twitter. I completely agree that massive content platform should have that yesterday (if not because you have a creative friend, but because they want to control their own traffic).
http://adjix.com/tt3b
Sharing Amazon S3 buckets with third parties opens up a lot of possibilities. For example, you could have Twitter store every Tweet you've ever sent, as an HTML page, in your bucket. Or, Blogger could store every post you've blogged in your bucket. Etcetera, etcetera.
I'm not talking about mini-posts that were multicasted to Twitter and Blogspot at the same time. I'm talking 500-1,000 word posts that have bit.ly URLs in the middle of them. What the hell for? All it does is obscure where you're going, which makes me *much* more likely to not click them.
Maybe they enable better tracking? I'm not sure. Because I already track all my outbound links. I just don't understand why someone would take the extra effort to go out and obfuscate their links, when length is clearly not an issue.
[pant ... pant ... ]
Okay, sorry. That's been building up for a while now.
You map the CNAME to the URL-shortener. Later, if they go away or you decide
to leave them, you point the CNAME to the Amazon bucket.
Then you don't need to change the API. And I think we need to see some
convergence there. Since I've built on the tr.im API, and since it's going
open source, that seems the most logical to me. I imagine that bit.ly, the
800-pound gorilla won't see it that way.
Dave
As for URLshorteners, this idea can do great help to the current situation.But basically I think it's a more meaningful solution to loosen the limitation of character number on microblogging.
One key reason that people want to shorten URLs in blog posts is so they can see how many clicks each link receives, where it came from, by whom, etc. This is easiest for the layperson to do when using a URL shortener that tracks stats, as you mentioned in your post.
It's not only the fact that the link is shortened but rather the fact that it's trackable and you can brand your link by using your own domain name.
If you shortened a link with Adjix and you have your CNAME point to Adjix, then Adjix will track click stats while saving your links in your S3 bucket. If Adjix goes away, you can then just point your CNAME back to your S3 bucket and the link lives on.
http://www.scripting.com/stories/2009/04/27/adj...
One idea behind dropping the URLs in your bucket is that you could now make the URLs portable between URL shorteners that provide tracking stats.
For example, if you shortened a link with tr.im and they implemented this technique, you could then move to Adjix, in the future and Adjix, would pull your shortened URL out of the HTML meta-refresh page, track the click, and then perform a 301 redirect.
In other words, not only could your bucket preform the live redirect if you wanted, it could also be a data store for your links.
1. http://adjix.com/ty35
Basic, shortened, Adjix link.
Traffic stats collected by Adjix.
2. http://go.usna93.com/ty35
Shortened link with CNAME for go.joemoreno.com pointing to partner.adjix.com.
Traffic stats collected by Adjix.
3. http://links.joemoreno.com/ty35
Shortened link with CNAME for links.joemoreno.com pointing to S3 bucket: links.joemoreno.com.s3.amazonaws.com.
Traffic stats logged by S3 bucket logging.
4. http://urlpuppy.com/ty35
Shortened link with registrar domain name forwarding.
Traffic stats logged by S3 bucket logging.
One extra step is required to get #4 to work since you can't set a domain name's root to a CNAME. You must have your registrar forward your domain name to http://s3.amazonaws.com/YourS3BucketName.
Don't forget that you'll need to share your S3 bucket with Adjix to get this to work: http://adjix.com/tt3b
http://www.codinghorror.com/blog/archives/00127...
http://en.wikipedia.org/wiki/URL_shortening#Cri...
http://userscripts.org/scripts/show/52584
http://www.google.com/support/webmasters/bin/an...
On the YouTube video around 12:20 the presenter begins talking about how the site reputation does not transfer between domains.
So having canonical tags pointing to domains anywhere other than the current domain or sub-domain causes Google to ignore it.
ex. of good use of canonical element and bad.
a.blah.com/1137 -> blog.blah.com/2009/08/19/some-post-about-stuff
ad.vu/1137 -> blog.blah.com/2009/08/19/some-post-about-stuff
I think we all know that the actual code involved to make a short trackable url is simple.
Their are various methods but none are complicated. Some won't work as well as others.
But we dont need adjix or any 3rd party involved (besides your web host and domain registrar) unless you find value in that, and some will.
tr.im is releasing their code and data for this reason while also providing services/support.
Wordpress users can use Plugins listed here:
http://wordpress.org/extend/plugins/search.php?...
Here is a stand-alone script that comes with a WP plugin and an API:
http://yourls.org
Wordpress.com users can leverage wp.me - http://en.blog.wordpress.com/2009/08/14/shorten/
RR/W just posted about a few other solutions:
http://www.readwriteweb.com/archives/you_dont_n...
The good thing is that their are so many options for those who truly need to care about this stuff (until we can move on from this dark time of cryptic url usage ;)
I believe that we need *all* 3rd party link shrinkers involved, and using the technique outlined here, in order to make your links portable for when link shrinkers like tr.im or zi.ma close up shop.
i'm all for all service providers to do everything they can to make sure that the data is open and accessible. whatever approach works, works.
i think it's easy to mesh the discussion because their are those who:
a) have their own website/domain and want short trackable urls for their own content (ie. blog posts).
b) have their own website/domain and want short trackable urls for ANY content (any url on the web).
c) just want to quickly share links on social services like twitter and dont care about tracking, link rot etc.
d) run a service as a data store, stats collector and as a result, a traffic handler and do not want the trend of DIY to gain much momentum.
e) community-owned services like rp.ly (who first conjured up the idea) and soon to be tr.im (unless they change their mind again ;)
and any other variations. it's easy to misstep with comments :)
From what I've seen playing around with your solution, you would need everyone to work around the same name space, base32,64, 1-12? characters, otherwise there would be overlap when moving your short link files between services when they "close up shop" and you will have to create more sub-domains each time. {a,b,c,d}.ho.st
Ideally, you'd want all URL shorteners to use the same name space to prevent a token collision.
Otherwise, each URL shortener will need to check your bucket to see if you've already created a link with the same token before shortening your link. If so, then the URL shortener will need to choose a new token which isn't really a big deal - most provide that service, now, by letting you pick your own token: http://tinyurl.com/ReallyCoolLink
Here's what you'll need to do:
1. Create an S3 bucket named sho.rtu.rl.
2. You'll need to share your bucket with Adjix so it can save a copy of each link to your bucket. Details here: http://adjix.com/tt3b
3. Log into your Adjix account and click Edit Profile. At the bottom, just enter the name of your bucket.
That's it - each link that you now shorten with Adjix will be copied to your bucket.
If you want Adjix to track click stats then you'll need to point the CNAME for sho.rtu.rl to partner.adjix.com.
If Adjix goes down, for any reason, then you can simply change the CNAME for sho.rtu.rl to sho.rtu.rl.s3.amazon.com and the link will continue to work.
Please don't hesitate to let me know if this is confusing.
Thanks,
Joe
Using all the same token name space, while everyone has their own FQDN would severely handicap the ability for any one URL shortener to act alone in provisioning tokens without a clearing house. I sure as hell don't want to see an ICANN of shortURL.
It's too late for collision prevention, no one does it, and it's too time constraining. Also, to manually choose new tokens and then update all the accessible HTML elements for links already distributed sounds like a nightmare.
All the manual DNS changes can be a hassle should your provider not allow you to adjust SOA or individual TTL refresh, expiry entries.
Here's a tip, if we point our CNAME to your partner.adjix.com; you should be the one to update DNS records when you fail, not us.
BTW, I'll probably use the method anyway.
One other thing. You will need a record of the shorturls. They'll never break, but when you switch providers, the next system that generates your shorturls will need to know whether the shorturl they are creating is in fact unique, right?
In other words, think of your bucket as the link backup data store , with the host CNAME point to, say partner.adjix.com. You'd only have to deal with the non-301 redirect if something happened to your link shrinking partner.
That's what I mean about having a record of the shorturls. In case you do switch providers, they need to know that any new shorturl they create is a unique one.
Yes, the new provider would have to do a quick check to avoid a namespace collision of the token. The only way around this is if one authority handed out the tokens. Perhaps that's where 301Works.org will enter the game.
seo related:
http://www.seobook.com/archives/000297.shtml
a system i built takes advantage of static files (just txt files) and optional mysql data recording/backup. the static files, a unique one for each url, can be easily sent to s3 or any other server as a backup. they can even be served via rss feed so that anyone can subscribe and generate more backups ( i thought dave would like that idea ;). if your site crashes, your backup nameserver could direct people to your mirror server(s). wont get into redundancy here as their are various approaches... like just using Amazon EC2 or Linode etc. Point is, i like static file approach. though i dont create static html pages with the meta refresh like adjix, i still am able to track clicks, referrers and other data with no problem during the 301redirect.
sull
My strong preference is to stick with the same domain and compress the path and query string as much as possible. For example:
http://samj.net/2009/08/twitters-tweet-trademar... -> http://samj.net/1a
That way your user agents are saved at least one recursive DNS query (with CNAMEs it's two), a TCP 3-way handshake and a HTTP transaction and can get everything done in one go with keep-alive. In that case the performance impact would be negligible, and you can still do your tracking.
Whatever the mechanism we should definitely advertise the result using the rel=shortlink standard.
Sam
For example, let's say that your link was created by tr.im and stored in a bucket called go.samj.net. If tr.im closed up shop, you could then adjust your CNAME to point to Adjix and Adjix would pull your link from your bucket, track the stats, and do a 301 redirect.
I think more to the point, before we go about declaring "$solution has solved all the problems with URL-shorteners" we need to assess the proposal thoroughly (look how far rev=canonical got before being reeled back in).
Sam
The ideal solution should be part of the Internet. Perhaps an RFC needs to be sent out? I'm sure there's a PhD student working on something like this.
Sam
We'll be losing the "tracking" aspect of the current 3rd party short URLs, but this tracking is already quite unreliable given the non-unique nature of the short URLs. Perhaps it would be in the interest of Twitter etc to provide this click tracking capability. It'd still be unreliable, but would be far less damaging to the net infrastructure.
Regarding 3rd party tracking, what is the use case really (beyond feeding the egos of people whos egos don't need feeding). I realise that there are marketing benefits for the few of us who are in that industry but the cost of creating millions of links to each resource (rather than one or two - the canonical and short links) are many and not always obvious. I found today that browsing a twitter search for new content was almost impossible for example because my browser's colouring of visited links was broken.
Incidentally Wordpress blog software will be including shortening services (perhaps using wp.me) in its stats module soon... these will presumably capture stats as well, thus solving the problem.
Sam
Re 3rd party tracking, yes, the value is the ego boost, and also the ability to understand how something spreads across the net and who are the real influencers. For companies like http://tra.cx this is invaluable information. It could be provided by the conversational media (e.g. Twitter) or by the publishing infrastructure (e.g. Wordpress).
And of course, once we have multiple stat-providers, there will be companies that will create services that will aggregate all these stats in one place, creating a new industry segment, and so the cycle continues...
Suppose I'm using the shortener miniu.rl and one day they go out of business so I point the CNAME at my bucket on Amazon instead of on their host.
Then I go create an account on newmi.ni and all my new URLs go there.
Just like when I start a new blog. I leave the old one where it is. No need to merge the two (though people often get hung up on doing exactly that).
BTW, as I said in the post, worth repeating because people are saying that it's a flaw, just trading one weak spot for another. If Jeff Bezos changes his mind and decides to shut down S3, I was smart, I backed up my bucket on my laptop hard disk. Then I got an account at Rackspace and moved the folder there, and pointed the CNAME at my server over there. So it's not trading one weakness for another, it is giving me control of my data. Big diff.
unclear what happens if I stop paying my domain registration fee (the one that has the cnames in it). feels like dead links will still be out there.
Let's say there's a new httpr:// and httpsr:// (for http(s) redirection) URI. The browser would take that domain, do a NAPTR query that would return one (or multiple) new URLs that it would follow.
It's just like sip, ENUM, etc... but takes a url and turns it into another url.
So httpr://3tjf.me.com would get the browser to do a DNS NAPTR call to 3tjf.me.com which would return the new url, and it would follow that.
Anyway, this is all extremely hypothetical for many obvious reasons... :) (including the lack of tracking offsetting the gain in speed and reliability)
[Does anyone else find it ironic that we're using a 3RD PARTY COMMENTING SERVICE to discuss how to reduce reliance on 3rd party link shorteners?]
- URL Shorteners should provide either a feed or batch backup solution to export data to another link redirection service.
- Accept CNAME redirections as google does with their ghs.google.com service. Users could then register domains that they control and own (and can point them elsewhere if the service expires).
It would be far better to standardize on an export format and data/domain portability, than it would be for individual users owning and managing these kludgey S3 buckets.
See earlier comment addressing this exact subject in this thread.
In brief - I'd say to primary motivation is tracking/analytics when you are sharing links to sites that are not your own. Another reason I didn't list there, are that you are adding "features" to your link:
Adding Real-time Discussions - http://go2.me
Adding an element of fun and personalization to your link - http://www.quip-art.com
The next step is to make the URL data portable, which isn't hard - we'd just need to standardize it between URL shorteners.
So if one of the mainstream (or other) shorteners would allow me to use a private CNAME record when creating links, and would allow me to export my data - problem solved (in the "right" way).
That would be a really good idea. Then each of us could have our own software that shortens our URLs and those we send along, reduce the unnecessary centralization, and provide smaller URLS that can fit within the Twitter parameters. There are URL shorteners out there--mostly in the internet marketing arena--and I think they have much broader use than they're currently getting. Of course, we ARE a lazy species...