A very long time ago (read: ten years ago), we were in-between the so-called
First and Second Browser Wars. Internet Explorer had killed Netscape
Navigator by taking advantage of their desktop monopoly and Scrooge McDuck-like
financial reserves to install a free copy of Internet Explorer on every single
computer in the world (basically). Internet Explorer 6 was the dominant
browser, and Netscape as a company was over.
Netscape, before their demise, had embarked on a project to totally rewrite
their web browser. Their new code was open-sourced and given to the Mozilla
foundation. In hindsight, this was a stunningly successful move, with the
ever-awesome Mozilla Foundation going from strength to strength now, nearly ten
years after its foundation.
The second browser war was initially a festering cold war between the reborn
Netscape Navigator (now entitled Mozilla Firefox) and the dormant Internet
Explorer 6 (eventually updated to IE 7 after a 6 year development freeze).
Later, other parties like Google Chrome joined the party. Oh, and Safari and
Opera were kinda floating around in this war too, but honestly they’re not that
important to the story I’m trying to tell.
Anyway, long story still kinda long, as part of these two browser wars,
browsers felt the need to compete with each other on features. However, to use
these features you needed to get web developers to build web sites that used
them. The problem is that your new feature would only work on your browser.
This meant that, when some poor soul came along trying to view your
super-awesome ActiveX powered web page, and they had the misfortune to be using
Netscape Navigator, your website would at best look awful, and at worst explode
in several mysterious ways.
These people would then go away and tell their friends about your crappy
website that wouldn’t even render properly! And they’d say that their friends
should use your competitor’s website, even though your competitor can’t even
spell ActiveX! And you’d go out of business and your children would have to
go to a state school, and it would just be horrible.
So you needed some way to tell what features a browser had. There was a way to
do that, of course: Javascript. Unfortunately, some features couldn’t be easily
detected in Javascript, and writing Javascript was, well, weird, and Javascript
was slow, and so lots of websites didn’t want to do that (or didn’t know they
should). What would they do instead?
Well, RFC 1945 and RFC 2616 (the HTTP 1.0 and HTTP 1.1 specifications) stated
that all browsers, web crawlers and other tools that interacted with web
servers should identify themselves using a special header in the HTTP they
send: the User-Agent header. This header should be (as much as possible)
unique to a specific type of agent. This means that Internet Explorer should
send a User-Agent header that is different to all other browsers and to all
other versions of IE.
“Perfect!” cry the web developers. “Our servers can check for this string,
and use its value to determine how to render the page!”. And so begins the
the trouble.
The Trouble
You see, the problem with using the User-Agent string to check for features is
that the User-Agent string tells you nothing about what features a given
User-Agent has. After all, that’s not what it’s for! So you, naïve late-1990s
web programmer, might write your site when only Mozilla Firefox has support for
the hot new Twiddlor feature (note: not a real feature). So you only server
Twiddlor-enabled pages to people whose User-Agent strings identify them as
being a version of Firefox.
The problem is, six months later the guys in Redmond get around to adding
Twiddlor support to Internet Explorer. But all their users are still
complaining that none of their favourite websites will let them use Twiddlor,
instead claiming that the website is “Best used in Mozilla Firefox” or some
such nonsense.
How does Microsoft get you to show them the Twiddlor-enabled page? Simple:
they change their User-Agent string! Sadly, I’m not even joking: this is
actually what happened. To prove it, I’m going to show you a few modern browser
UA strings.
Here’s the UA string sent by Google Chrome version 27.0.1453.47 beta (yeah),
running on my Mac:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.47 Safari/537.36
“What is all that crap?”, I hear you ask, quite rightly. Why does it say it’s
Mozilla? It’s not Mozilla! You’re quite right. But enough people have tested
for Firefox by just checking that the word ‘Mozilla’ is in the UA string that
everyone puts it there. And I mean everyone. Check out Safari, also on my
Mac:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17
Notice that both Safari and Chrome claim to be versions of Safari. That’s
pretty damn weird.
What about Internet Explorer 10, on my Windows machine?
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)
At least it’s not claiming to be Safari! In fact, this is the best UA string
I’ve seen, being a fairly honest representation of the browser.
Finally, let’s check Firefox, also on my Windows box.
Mozilla/5.0 (Windows NT 6.2; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0
What Should A User-Agent String Look Like?
To see an example of how these were supposed to look when the standard was
originally proposed, we can see what
Requests sends.
python-requests/1.2.0 CPython/2.7.2 Darwin/12.2.0
Short and to the point. The ‘browser’ and its version, the ‘platform’ and its
version, and the OS (sort of) and its version.
Why Does This Matter?
In principle, the new Javascript-heavy world should have cured us of this
problem. People should write JS that tests for features and then uses them, and
serves a less interesting version of the web page if you don’t support it. And,
mostly, this is what happens! Libraries like JQuery have taken a lot of the
hard work out of doing this, so most websites you’ll encounter nowadays do the
right thing.
The problem is, sometimes they don’t. And when they don’t, you can encounter
strange and confusing bugs. These bugs then tie up developer time and generally
make everyone’s life worse. To provide an example, I’m going to briefly walk
you through a bug that appeared on the Requests GitHub page a few days ago.
An Example
A user reported that, when he accessed a specific web page by doing a simple
GET with no complicated stuff, he was getting a httplib.IncompleteRead
exception thrown into his face.
This was odd in itself. This exception is only ever thrown when either the user
or the remote server is using chunked encoding, but the user reported that he
didn’t think either party was doing so. He also kindly provided the URL, so
that I could reproduce the bug locally. (This is excellent practice, by the
way: I’m far more likely to help out if I can easily reproduce your bug on my
machine.)
When I made the same request, I also got the IncompleteRead exception thrown
in my face. Further investigation showed that the web server claimed to be
serving using
chunked encoding, but in fact
was just sending the page as normal. This is pretty bad, and there’s not much
Requests can do about this: the web server is simply doing the wrong thing.
First note for website developers: do NOT claim to be using chunked encoding
when you are not!
I was interested to see if we could get the page data anyway, so I patched my
local copy of the standard library to see what we got when I returned the data
instead of throwing an exception. What I saw was the second unpleasant thing
this web site had done. The HTML for this page was about 20 lines long. All it
did was embed, at full size, a
frame containing
another page, or a warning if your browser doesn’t support frames.
This is pretty obnoxious: why not just server the other page? Why require
frames? You aren’t even doing anything with them, you’re just using them for
the sake of using them! Second note for website developers: do not use frames
when you don’t need them! They are awkward for anything that isn’t a browser.
In an attempt to be helpful, I pulled the URL being framed out of the HTML and
suggested the user hit that instead. Out of sheer curiosity, I then did a
Requests GET on the URL.
Requests threw an exception again.
I was pretty surprised here, the page rendered fine in my browser. So I looked
at the exception. Connection Reset By Peer, read the socket error text. For
those who don’t know their network protocols, this indicates that the
TCP connection to
the web server was closed while we were expecting data on it.
This is very odd. Requests sent a totally compliant, basic HTTP GET request,
and the remote server was shutting the connection in response to it. Doing this
is totally against the HTTP specification. Any compliant server is required
to respond with an HTTP error code and a Connection: close header if it
wants to tear the connection down. Additionally, why did it work fine in Chrome
but fail in Requests?
There’s really only one obvious thing to do. I grabbed Chrome’s User-Agent
string and got Requests to send that instead of its own UA string. (For those
who want to spoof their UA string, Requests allows you to pass it as a header.
We only set one ourselves if you don’t provide one for us.)
Success! The page rendered and returned to us.
For those who want a summary, what was happening here is that the remote site
was sniffing the User-Agent header. Instead of checking for features, however,
what it was doing was using the header as a gatekeeper! If you don’t have the
right User-Agent, you don’t just get a less feature-filled site: you get
nothing. Not even an HTTP error page.
This is probably the worst example of User-Agent sniffing I’ve ever seen. This
was a website developer using a bad practice to violate the HTTP specification.
In addition to simply being rude, this is also a genuine cost for many
developers. And crap like this leads to stupid UA strings like the ones I
showed above.
This is also the third note for website developers: always send HTTP error
codes, don’t just close connections.
The Moral Of The Story
The most important lesson, however, is this.
Ignore the User-Agent string unless you absolutely have to.
Detecting browser features is not what the User Agent string is for, so please
don’t use it for that. And if you do (which I’m sure you will, because no-one
listens to me anyway), make sure that you don’t refuse service based on the
User-Agent. If you want to render a slightly different page, fine, I get that.
But don’t refuse to render it at all. It’s obnoxious, it’s brittle, and it’s
so 1990s. And besides, as I showed above, all modern User-Agents can lie
in their User-Agent string! You can set it in Firefox, and in Chrome, and
(probably) in IE, Safari and Opera as well. So not only are you mis-using it,
you’re not even getting accurate information!
And if you do abuse the User-Agent string, and I catch you, I will come through
the internet and get you. And then I will write an angry blog post, and
publicly shame you.
So,
St Kieran Catholic Church and School.
Sort your website out. It’s being obnoxious. And is that really the impression
you want to give?