Rust and Rest

Libraries for HTTP

To make an HTTP request you have a choice of libraries. In particular
there are two in Rust you can try: hyper and
rust-curl. I tried both and there are
some releases with the former but I settled in rust-curl in the end. The
reason for this is twofold. The first is that curl (despite some of the
oddities in how it does things) is very powerful and integrates really
well with the system SSL libraries. This means that when I compile the
executable I get the native TLS support right away. rust-curl also
(despite being not a pure rust library) compiles really well out of the
box on Windows, macOS and Linux. The second reason is that Hyper is
currently undergoing a major shift in how it’s structured and a bit in
flux. I did not want to bet on that too much. When I started it also did
not have proxy support which is not great.

For JSON parsing and serializing I went with serde. I suppose that serde will eventually
be the library of choice for all things serialization but right now it’s
not. It depends on compiler plugins and there are two ways to make it
work right now. One is to go with nightly Rust (which is what I did) the
other is to use the build script support in Rust. This is similar to what
you do in Go where some code generation happens as part of the build. It
definitely works but it’s not nearly as nice as using serde with nightly
Rust.

API Design

The next question is what a good API design for a Rust HTTP library is. I
struggeld with this quite a bit and it took multiple iterations to end up
with something that I think is a good pattern. What I ended up is a
collection of multiple types:

• Api: I have a basic client object which I call Api internally.
  it manages the curl handles (right now it just caches one) and also
  exposes convenience methods to perform certain types of HTTP requests.
  On top of that it provides high level methods that send the right HTTP
  requests and handle the responses.
• ApiRequest: basically your request object. It’s mostly a builder
  for making requests and has a method to send the request and get a
  response object.
• ApiResponse: contains the response from the HTTP request. This also
  provides various helpers to convert the response into different things.
• ApiResult<T>: this is a result object which is returned from most
  methods. The error is a special API error that converts from all the
  APIs we call into. This means it can hold curl errors, form errors,
  JSON errors, IO errors and more.

To give you an idea how this looks like I want to show you one of the
high level methods that use most of the API:

pub fn list_releases(&self, org: &str, project: &str)
    -> ApiResult<Vec<ReleaseInfo>>
{
    Ok(self.get(&format!("/projects/{}/{}/releases/",
                         PathArg(org), PathArg(project)))?.convert()?)
}

(Note that I’m using the new question mark syntax ? instead of the
more familiar try! macro here)

So what is happening here?

1. This is a method on the Api struct. We use the get()
   shorthand method to make an HTTP GET request. It takes one argument
   which is the URL to make the request to. We use standard string
   formatting to create the URL path here.
2. The PathArg is a simple wrapper that customizes the formatting so
   that instead of just stringifying a value it also percent encodes it.
3. The return value of the get method is a ApiResult<ApiResponse>
   which provides a handy convert() method which does both error
   handling and deserialization.

How does the JSON handling take place here? The answer is that
convert() can do that. Because Vec<ReleaseInfo> has an automatic
deserializer implemented.

The Error Onion

The bulk of the complexity is hidden behind multiple layers of error
handling. It took me quite a long time to finally come up with this
design which is why I’m particularly happy with finally having found one I
really like. The reason error handling is so tricky with HTTP requests is
because you want to have both the flexibility of responding to specific
error conditions as well as automatically handling all the ones you are
not interested in.

The design I ended up with is that I have an ApiError type. All the
internal errors that the library could encounter (curl errors etc.) are
automatically converted into an ApiError. If you send a request the
return value is as such Result<ApiResponse, ApiError>. However the
trick here is that at this level no HTTP error (other than connection
errors) is actually stored as ApiError. Instead also a failed
response (because for instance of a 404) is stored as the actual response
object.

On the response object you can check the status of the response with these
methods:

pub fn status(&self) -> u32 { self.status }
pub fn failed(&self) -> bool { self.status >= 400 && self.status <= 600 }
pub fn ok(&self) -> bool { !self.failed() }

However what’s nice is that most of the time you don’t have to do any of
this. The response method also provides a method to conver non successful
responses into errors like this:

pub fn to_result(self) -> ApiResult<ApiResponse> {
    if self.ok() {
        return Ok(self);
    }
    if let Ok(err) = self.deserialize::<ErrorInfo>() {
        if let Some(detail) = err.detail {
            return Err(ApiError::Http(self.status(), detail));
        }
    }
    Err(ApiError::Http(self.status(), "generic error".into()))
}

This method consumes the response and depending on the condition of the
response returns different results. If everything was fine the response
is returned unchanged. However if there was an error we first try to
deserialize the body with our own ErrorInfo which is the JSON response
our API returns or otherwise we fall back to a generic error message and
the status code.

What’s deserialize? It just invokes serde for deserialization:

pub fn deserialize<T: Deserialize>(&self) -> ApiResult<T> {
    Ok(serde_json::from_reader(match self.body {
        Some(ref body) => body,
        None => &b""[..],
    })?)
}

One thing you can see here is that the body is buffered into memory
entirely. I was torn on this in the beginning but it actually turns out
to make the API significantly nicer because it allows you to reason about
the response better. Without buffering up everything in memory it becomes
much harder to do conditional things based on the body. For the cases
where we cannot deal with this limitation I have extra methods to stream
the incoming data.

On deserialization we match on the body. The body is an Option<Vec<u8>>
here which we convert into a &[u8] which satisfies the Read
interface which we can then use for deserialization.

The nice thing about the aforementioned to_result method is that it
works just so nice. The common case is to convert something into a result
and to then deserialize the response if everything is fine. Which is why
we have this convert method:

pub fn convert<T: Deserialize>(self) -> ApiResult<T> {
    self.to_result().and_then(|x| x.deserialize())
}

Complex Uses

There are some really nice uses for this. For instance here is how we
check for updates from the GitHub API:

pub fn get_latest_release(&self) -> ApiResult<Option<(String, String)>>
{
    let resp = self.get("https://api.github.com/repos/getsentry/sentry-cli/releases/latest")?;
    if resp.status() != 404 {
        let info : GitHubRelease = resp.to_result()?.convert()?;
        for asset in info.assets {
            if asset.name == REFERENCE_NAME {
                return Ok(Some((
                    info.tag_name,
                    asset.browser_download_url
                )));
            }
        }
    }
    Ok(None)
}

Here we silently ignore a 404 but otherwise we parse the response as
GitHubRelease structure and then look through all the assets. The call
to to_result does nothing on success but it will handle all the other
response errors automatically.

To get an idea how the structures like GitHubRelease are defined, this
is all that is needed:

#[derive(Debug, Deserialize)]
struct GitHubAsset {
    browser_download_url: String,
    name: String,
}

#[derive(Debug, Deserialize)]
struct GitHubRelease {
    tag_name: String,
    assets: Vec<GitHubAsset>,
}

Curl Handle Management

One thing that is not visible here is how I manage the curl handles. Curl
is a C library and the Rust binding to it is quite low level. While it’s
well typed and does not require unsafe code to use, it still feels very
much like a C library. In particular there is a curl “easy” handle object
you are supposed to keep hanging around between requests to take advantage
of keepalives. However the handles are stateful. Readers of this blog
are aware that there are few things I hate as much as unnecessary stateful
APIs. So I made it as stateless as possible.

The “correct” thing to do would be to have a pool of “easy” handles.
However in my case I never have more than one request outstanding at the
time so instead of going with something more complex I stuff away the
“easy” handle in a RefCell. A RefCell is a smart pointer that
moves the borrow semantics that rust normally requires at compile time to
runtime. This is rougly how this looks:

pub struct ApiRequest<'a> {
    handle: RefMut<'a, curl::easy::Easy>
}

pub struct Api {
    shared_handle: RefCell<curl::easy::Easy>,
    ...
}

impl Api {
    pub fn request(&self, method: Method, url: &str)
        -> ApiResult<ApiRequest<'a>>
    {
        let mut handle = self.shared_handle.borrow_mut();
        ApiRequest::new(handle, method, &url)
    }
}

This way if you call request twice you will get a runtime panic if the
last request is still outstanding. This is fine for what I do. The
ApiRequest object itself implements a builder like pattern where you
can modify the object with chaining calls. This is roughly how this looks
like when used for a more complex situation:

pub fn send_event(&self, event: &Event) -> ApiResult<String> {
    let dsn = self.config.dsn.as_ref().ok_or(Error::NoDsn)?;
    let event : EventInfo = self.request(Method::Post, &dsn.get_submit_url())?
        .with_header("X-Sentry-Auth", &dsn.get_auth_header(event.timestamp))?
        .with_json_body(&event)?
        .send()?.convert()?;
    Ok(event.id)
}

Lessons Learned

My key takeaways from doing this in Rust so far have been:

• Rust is definitely a great choice for building command line utilities.
  The ecosystem is getting stronger by the day and there are so many
  useful crates already for very common tasks.
• The cross platform support is superb. Getting the windows build going
  was easy cake compared to the terror you generally go through with
  other languages (including Python).
• serde is a pretty damn good library. It’s a shame it’s not as nice to
  use on stable rust. Can’t wait for this stuff to get more stable.
• Result objects in rust are great but sometimes it makes sense to not
  immediately convert data into a result object. I originally converted
  failure responses into errors immediately and that definitely hurt the
  convenience of the APIs tremendously.
• Don’t be afraid of using C libraries like curl instead of native
  Rust things. It turns out that Rust’s build support is pretty
  magnificent which makes installing the rust curl library
  straightforward. It even takes care of compiling curl itself on
  Windows.

If you want to see the code, the entire git repository of the client can
be found online: getsentry/sentry-cli.