I’m delighted to announce that today is the release date of version 1.0.0 of
my brand new project, Hyper-h2. Hyper-h2 is the first step in what I hope will
be a long journey improving the state of HTTP in Python, by providing a set of
composable, re-usable libraries that can act as tools for building bigger and
better HTTP projects. If you want to check it out, jump straight to
the docs. Otherwise, I’d like to talk a little
bit about how I got here, and about the brand-new Hyper Project that I’m going
to be working on for the foreseeable future.
The Past
It’s a little mind-boggling to realise that I first emailed
Mark Nottingham to ask about adding my original
Hyper project to the list of HTTP/2
draft implementations in February of 2014. That decision has led me down a bit
of an unexpected path, including a working relationship with the IETF HTTPBis
working group and several conference talks, including one at
The Big PyCon in Montreal.
Altogether, it’s been a bit of a ride.
For the past nine months or so, however, I’ve been dissatisfied with the shape
of the Hyper project. It suffered from some limitations that were the result of
design decisions I fundamentally believe are
backward-looking. It was also
fairly monolithic, and contained a lot of code that might be more generally
useful in the Python community. However, its greatest problem was that it was
alone.
Despite the fact that HTTP/2 has been being drafted for more than two years,
and that it’s been a full standard since May, Python has basically had only one
HTTP/2 implementation, and it was a synchronous client. Additionally, there
was really no effort afoot that I could see to write any other implementation
in Python. As far as I could tell, no-one cared.
This seems unacceptable to me. Python is potentially a fantastic language to
work with HTTP/2 in. Python is especially well-suited to investigate the
potential use of HTTP/2 as an RPC mechanism (where GRPC)
is an example of one possible approach.
However, I can’t write all the possible HTTP/2 implementations in Python. I’m
just one person. Nor should I: there are lots of great HTTP-using tools in
Python, and I’m not interested in trying to replace or obsolete them. Instead,
I’d like to improve them.
With this in mind, I set out to build the Hyper Project.
The Hyper Project
It seemed to me that the problem was that HTTP/2 is complex. It has a framing
layer, a compression layer, a protocol stack, a priority tree, and all other
kinds of weirdness that implementations would have to write from scratch. This
is hard, and lots of developers simply didn’t have the time or inclination to
do that work from nothing. What these developers need is a toolkit.
(As an aside, this problem applies in the rest of the OSS world as well. You
might be surprised to know that most OSS implementations of HTTP/2 are actually
built on top of one code base: nghttp2. For example,
curl and the Apache Web Server are both built on top of it.)
When you want to bring HTTP/2 to your platform or project, you don’t want to
have to implement all of that stuff. Some of it isn’t too hard but just a fair
lot of work (e.g. framing), while some of it is fiddly and prone to subtle
bugs. Regardless, it would make your life easier if you were able to pick up
one or more ready-made, off-the-shelf implementations that you can simply plumb
into your project however you see fit.
Enter the Hyper Project. The goal of this project is to provide a collection of
tools for building HTTP/1.1 and HTTP/2 implementations. Each of these tools
will be targetted at the broadest possible use-case, and will compose together
to allow you to build complete HTTP/2 implementations. They will also function
well on their own: this means that people who need something unusual or
different in one small aspect of their implementation can use the standard bits
every where else, and only hand-code the bits that they do differently to
everyone else.
There is really no reason to spread our work across a number of different
software projects, all reinventing the same wheel in subtly different ways. It
should be possible for most of the Python world to build on the same set of
common code, differing only where we need to in order to express our different
goals and opinions.
Excitingly, this isn’t just a pipe dream for me: the Hyper Project exists,
right now. You can find it at the project website,
which also hosts documentation for the inaugural set of sub-projects. At the
moment, these include a HTTP/2 framing layer
(hyperframe) and a pure-Python HPACK
implementation (hpack), necessary building
blocks of any HTTP/2 implementation. Both of these were ripped out of the
original Hyper code and made general enough to be used elsewhere, and installed
by themselves. It also contains a CFFI-based wrapper to the
Brotli compression algorithm
reference implementation
(brotlipy), as proof that the Hyper
Project is about HTTP in general, and not just HTTP/2.
Hyper-h2
The crowning jewel of the current Hyper Project, however, is the fact that it
contains a general, pure-Python HTTP/2 stack, called
Hyper-h2. This stack has a lofty aim: to be the
base layer for the vast majority of Python HTTP/2 implementations. To that end,
it has a number of unusual features that are worth explaining.
Firstly, and most notably, Hyper-h2 does absolutely no I/O. It exists
entirely in memory, reading to and writing from in-memory buffers. The reason
for this is that it becomes possible to use this same kernel of code in any
programming paradigm. If you like synchronous code, that’ll work. If you like
threads, that’ll work too. If you like gevent, that’s fine. Twisted? Check.
Tornado? All good. asyncio? You bet. All you need to do is write the bit around
the outside that does the boring stuff of reading from and writing to sockets.
Pass the data into hyper-h2, and it’ll parse it and turn it into something you
can actually work with.
That’s the next notable thing about Hyper-h2: it’s not a complete
implementation, like Apache or curl. Instead, it’s intended to be a core part
of your implementation. Hyper-h2 lets you decide what you want to do on the
connection, and tells you what the other side did, but it doesn’t know
everything there is to know about your HTTP/2 application. This means it’s not
a client, or a server: it’s a tool for writing clients and servers. Hyper-h2
enforces the HTTP/2 state machine, manages settings and compression,
serializing and deserializing, and stream management: but it doesn’t do
anything about requests and responses. That’s up to you: to decide what works
best for you.
This flexibility means that Hyper-h2 can be used as the base for any number
of projects. If you want HTTP/2 in aiohttp, Hyper-h2 could be used there.
Twisted? Same deal. And if you want to do something more specialised, embedding
HTTP/2 directly in your application, you can do that with Hyper-h2 as well.
Hyper-h2 aims to be general enough that the majority of projects could use it
without adjustment, but specific enough that it manages to be useful. Thus,
some use-cases are likely to remain out of scope for it. For example, it will
almost certainly confine itself to strictly enforcing the HTTP/2 state machine:
this means that it may not be a good choice for implementations that
occasionally need to violate that state machine.
Success
So, what does success look like for the Hyper project? The goal is for other
projects to save time by building on top of our work, and from that perspective
we’re already well on the way. hpack has already been packaged by
Debian (and so by
extension Ubuntu),
Arch, and
Kali. This
is because hpack has been adopted by
netlib, the networking library for the
awesome mitmproxy project. Hyperframe appears to be
on the way to being a part of netlib as well, which suggests it’ll be next on
the list.
From my perspective, this is already a success: we’ve saved a great project
some time and effort in their implementation. But we can do more.
In the next few months I plan to start pushing forward with Hyper-h2. I aim to
add HTTP/2 support to Twisted. I am also going to start accepting offers from
other projects that would like help adding HTTP/2 to their list of features by
using Hyper-h2, or by using hpack or hyperframe directly. I intend to add even
more projects to the Hyper umbrella: an interesting next target would be a
library that implements the fiendishly complex HTTP/2 priority scheme.
Additionally, I plan to rip the heart out of the old Hyper implementation and
replace it with Hyper-h2. When I do so, I’ll bring that library under the
umbrella of the Hyper Project as well, which will fix this slightly tricky
naming problem we have with two different things called “hyper”.
Most importantly though, it’s an exciting time to be working in HTTP in Python,
and I want you to be a part of it. If you have ideas for how to enhance any of
these projects, I want to hear from you. If you have ideas for a project that
you think would be a useful part of the Hyper umbrella, I want to hear from
you. If you want to get started in Open Source and want somewhere welcoming to
get started, I want to hear from you. If you have a HTTP project you don’t want
to maintain any more, I want to hear from you. And if you’re interested in
learning more about HTTP, I definitely want to hear from you: I can’t
maintain all of these libraries on my own!
I’m looking forward to the next few months working with all the great people
already involved with Python HTTP: come join me and let’s build awesome stuff
together.