Rust for Python Programmers

Syntax

The first difference you will notice as a Python programmer is the syntax.
Unlike Python, Rust is a language with lots of curly braces. However this
is for a good reason and that is that Rust has anonymous functions,
closures and lots of chaining that Python cannot support well. These
features are much easier to understand and write in a non indentation
based language. Let’s look at the same example in both languages.

First a Python example of printing “Hello World” three times:

def main():
    for count in range(3):
        print "{}. Hello World!".format(count)

And here is the same in Rust:

fn main() {
    for count in 0..3 {
        println!("{}. Hello World!", count);
    }
}

As you can see, quite similar. def becomes fn and colons become
braces. The other big difference syntax wise is that Rust requires type
information for parameters to function which is not something you do in
Python. In Python 3 type annotations are available which share the same
syntax as in Rust.

One new concept compared to Python are these functions with exclamation
marks at the end. Those are macros. A macro expands at compile time into
something else. This for instance is used for string formatting and
printing because this way the compiler can enforce correct format strings
at compile time. It does not accidentally happen that you mismatch the
types or number of arguments to a print function.

Traits vs Protocols

The most familiar yet different feature is object behavior. In Python a
class can opt into certain behavior by implementing special methods.
This is usually called “conforming to a protocol”. For instance to make
an object iterable it implements the __iter__ method that returns an
iterator. These methods must be implemented in the class itself and
cannot really be changed afterwards (ignoring monkeypatching).

In Rust the concept is quite similar but instead of special methods, it
uses traits. Traits are a bit different in that they accomplish the same
goal but the implementation is locally scoped and you can implement more
traits for types from another module. For instance if you want to give
integers a special behavior you can do that without having to change
anything about the integer type.

To compare this concept let’s see how to implement a type that can be
added to itself. First in Python:

class MyType(object):

    def __init__(self, value):
        self.value = value

    def __add__(self, other):
        if not isinstance(other, MyType):
            return NotImplemented
        return self.__class__(self.value + other.value)

And here is the same in Rust:

use std::ops::Add;

struct MyType {
    value: i32,
}

impl MyType {
    fn new(value: i32) -> MyType {
        MyType { value: value }
    }
}

impl Add for MyType {
    type Output = MyType;

    fn add(self, other: MyType) -> MyType {
        MyType { value: self.value + other.value }
    }
}

Here the Rust example looks a bit longer but it also comes with automatic
type handling which the Python example does not do. The first thing you
notice is that in Python the methods live on the class, whereas in Rust
the data and the operations live independently. The struct defines
the data layout and the impl MyType define methods the type itself
has, whereas impl Add for MyType implements the Add trait for that
type. For the Add implementation we also need to define the result
type of our add operations, but we avoid the extra complexity of having to
check the type at runtime like we have to do in Python.

Another difference is that in Rust the constructor is explicit whereas in
Python it’s quite magical. When you create an instance of an object in
Python it will eventually call __init__ to initialize the object,
whereas in Rust you just define a static method (by convention called
new) which allocates and constructs the object.

Error Handling

Error handling in Python and Rust is completely different. Whereas in
Python errors are thrown as exceptions, errors in Rust are passed back in
the return value. This might sound strange at first but it’s actually a
very nice concept. It’s pretty clear from looking at a function what
error it returns.

This works because a function in Rust can return a Result. A Result
is a parametrized type which has two sides: a success and a failure side.
For instance Result<i32, MyError> means that the function either
returns a 32bit integer in the success case or MyError if an error
happens. What happens if you need to return more than one error? This is
where things differ from a philosophical point of view.

In Python a function can fail with any error and there is nothing you can
do about that. If you ever used the Python “requests” library and you
caught down all request exceptions and then got annoyed that SSL errors
are not caught by this, you will understand the problem. There is very
little you can do if a library does not document what it returns.

In Rust the situation is very different. A function signature includes
the error. If you need to return two errors then the way to do this is to
make a custom error type and to convert internal errors into a better one.
For instance if you have an HTTP library and internally it might fail with
Unicode errors, IO errors, SSL errors, what have you, you need to convert
these errors into one error type specific to your library and users then
only need to deal with that. Rust provides error chaining that such an
error can still point back to the original error that created it if you
need to.

You can also at any point use the Box<Error> type which any error
converts into, if you are too lazy to make your own custom error type.

Where errors propagate invisibly in Python, errors propagate visibly in
Rust. What this means is that you can see whenever a function returns an
error even if you chose to not handle it there. This is enabled by the
try! macro. This example demonstrates this:

use std::fs::File;

fn read_file(path: &Path) -> Result<String, io::Error> {
    let mut f = try!(File::open(path));
    let mut rv = String::new();
    try!(f.read_to_string(&mut rv));
    Ok(rv)
}

Both File::open and read_to_string can fail with an IO error. The
try! macro will propagate the error upwards and cause an early return
from the function and unpack the success side. When returning the result
it needs to be wrapped in either Ok to indicate success or Err to
indicate failure.

The try! macro invokes the From trait to allow conversion of
errors. For instance you could change the return value from io::Error
to MyError and implement a conversion from io::Error to
MyError by implementing the From trait and it would be
automatically invoked there.

Alternatively you can change the return value from io::Error to
Box<Error> and any error can be returned. This way however you can
only reason about errors at runtime and no longer at compile time.

If you don’t want to handle an error and abort the execution instead, you
can unwrap() a result. That way you get the success value and if the
result was an error, then the program aborts.

Mutability and Ownership

The part where Rust and Python become completely different languages is
the concept of mutability and ownership. Python is a garbage collected
language and as a result pretty much everything can happen with the
objects at runtime. You can freely pass them around and it will “just
work”. Obviously you can still generate memory leaks but most problems
will be resolved for you automatically at runtime.

In Rust however there is no garbage collector, yet the memory management
still works automatically. This is enabled by a concept known as ownership
tracking. All things you can create are owned by another thing. If you
want to compare this to Python you could imagine that all objects in
Python are owned by the interpreter. In Rust ownership is much more
local. Function calls can have a list of objects in which case the
objects are owned by the list and the list is owned by the function’s
scope.

More complex ownership scenarios can be expressed by lifetime annotations
and the function signatures. For instance in the case of the Add
implementation in the previous example the receiver was called self
like in Python. However unlike in Python the value is “moved” into the
function whereas in Python the method is invoked with a mutable reference.
What this means is that in Python you could do something like this:

leaks = []

class MyType(object):
    def __add__(self, other):
        leaks.append(self)
        return self

a = MyType() + MyType()

Whenever you add an instance of MyType to another object you also leak
out self to a global list. That means if you run the above example you
have two references to the first instance of MyType: one is in
leaks the other is in a. In Rust this is impossible. There can
only ever be one owner. If you would append self to leaks the
compiler would “move” the value there and you could not return it from the
function because it was already moved elsewhere. You would have to move
it back first to return it (for instance by removing it from the list
again).

So what do you do if you need to have two references to an object? You
can borrow the value. You can have an unlimited number of immutable
borrows but you can only ever have one mutable borrow (and only if no
immutable borrows were given out).

Functions that operate on immutable borrows are marked as &self and
functions that need a mutable borrow are marked as &mut self. You can
only loan out references if you are the owner. If you want to move the
value out of the function (for instance by returning it) you cannot have
any outstanding loans and you cannot loan out values after having moved
ownership away from yourself.

This is a big change in how you think about programs but you will get used
to it.

Runtime Borrows and Mutible Owners

So far pretty much all this ownership tracking was verified at compile
time. But what if you cannot verify ownership at compile time? There you
have multiple options to your disposal. One example is that you can use a
mutex. A mutex allows you to guarantee at runtime that only one person
has a mutable borrow to an object but the mutex itself owns the object.
That way you can write code that access the same object but only ever once
thread can access it at the time.

As a result of this this also means that you cannot accidentally forget to
use a mutex and cause a data race. It would not compile.

But what if you want to program like in Python and you can’t find an owner
for memory? In that case you can put an object into a referenced counted
wrapper and loan it out at runtime this way. That way you get very close
to Python behavior just that you can cause cycles. Python breaks up
cycles in it’s garbage collector, Rust does not have an equivalent.

To show this in a better way, let’s go with a complex Python example and
the Rust equivalent:

from threading import Lock, Thread

def fib(num):
    if num < 2:
        return 1
    return fib(num - 2) + fib(num - 1)

def thread_prog(mutex, results, i):
    rv = fib(i)
    with mutex:
        results[i] = rv

def main():
    mutex = Lock()
    results = {}

    threads = []
    for i in xrange(35):
        thread = Thread(target=thread_prog, args=(mutex, results, i))
        threads.append(thread)
        thread.start()

    for thread in threads:
        thread.join()

    for i, rv in sorted(results.items()):
        print "fib({}) = {}".format(i, rv)

So what we do here is spawn 35 threads and make them compute in a very
terrible manner increasing Fibonacci numbers. Then we join the threads
and print the sorted results. One thing you immediately notice here is
that there is no intrinsic relationship between the mutex (the lock) and
the results array.

Here is the Rust example:

use std::sync::{Arc, Mutex};
use std::collections::BTreeMap;
use std::thread;

fn fib(num: u64) -> u64 {
    if num < 2 { 1 } else { fib(num - 2) + fib(num - 1) }
}

fn main() {
    let locked_results = Arc::new(Mutex::new(BTreeMap::new()));
    let threads : Vec<_> = (0..35).map(|i| {
        let locked_results = locked_results.clone();
        thread::spawn(move || {
            let rv = fib(i);
            locked_results.lock().unwrap().insert(i, rv);
        })
    }).collect();
    for thread in threads { thread.join().unwrap(); }
    for (i, rv) in locked_results.lock().unwrap().iter() {
        println!("fib({}) = {}", i, rv);
    }
}

The big differences to the Python version here is that we use a B-tree map
instead of a hash table and we put that into an Arc’ed mutex. What’s
that? First of all we use a B-tree because it sorts automatically which
is what we want here. Then we put it into a mutex so that we can at
runtime lock it. Relationship established. Lastly we put it into an Arc.
An Arc reference counts what it encloses. In this case the mutex. This
means that we can make sure the mutex gets deleted only after the last
thread finished running. Neat.

So here is how the code works: we count to 20 like in Python, and for each
of those numbers we run a local function. Unlike in Python we can use a
closure here. Then we make a copy of the Arc into the local thread. This
means that each thread sees it’s own version of the Arc (internally this
will increment the refcount and decrement automatically when the thread
dies). Then we spawn the thread with a local function. The move
tells us to move the closure into the thread. Then we run the Fibonacci
function in each thread. When we lock our Arc we get back a result we can
unwrap and the insert into. Ignore the unwrap for a moment, that’s
just how you convert explicit results into panics. However the point is
that you can only ever get the result map when you unlock the mutex. You
cannot accidentally forget to lock!

Then we collect all threads into a vector. Lastly we iterate over all
threads, join them and then print the results.

Two things of note here: there are very few visible types. Sure, there is
the Arc and the Fibonacci function takes unsigned 64bit integers, but
other than that, no types are visible. We can also use the B-tree map
here instead of a hashtable because Rust provides us with such a type.

Iteration works exactly the same as in Python. The only difference there
is that in Rust in this case we need to acquire the mutex because the
compiler cannot know that the threads finished running and the mutex is
not necessary. However there is an API that does not require this, it’s
just not stable yet in Rust 1.0.

Performance wise pretty much what you expect would happen. (This example
is intentionally terrible just to show how the threading works.)

Unicode

My favorite topic: Unicode 🙂 This is where Rust and Python differ quite
a bit. Python (both 2 and 3) have a very similar Unicode model which is
to map Unicode data against arrays of characters. In Rust however Unicode
strings are always stored as UTF-8. I have covered this in the past about
why this is a much better solution than what Python or C# are doing (see
also UCS vs UTF-8 as Internal String Encoding). What’s however very interesting about Rust
is how it deals with the ugly reality of our encoding world.

The first thing is that Rust is perfectly aware that operating system APIs
(both in Windows Unicode and Linux non-Unicode land) are pretty terrible.
Unlike Python however it does not try to force Unicode into these areas,
instead it has different string types that can (within reason) convert
between each other reasonably cheap. This works very well in practice and
makes string operations very fast.

For the vast majority of programs there is no encoding/decoding necessary
because they accept UTF-8, just need to run a cheap validation check,
process on UTF-8 strings and then don’t need an encode on the way out. If
they need to integrate with Windows Unicode APIs they internally use the
WTF-8 encoding which quite
cheaply can convert to UCS2 like UTF-16 and back.

At any point can you convert between Unicode and bytes and munch with the
bytes as you need. Then you can later run a validation step and ensure
that everything went as intended. This makes writing protocols both
really fast and really convenient. Compared this to the constant encoding
and decoding you have to deal with in Python just to support O(1)
string indexing.

Aside from a really good storage model for Unicode it also has lots of
APIs for dealing with Unicode. Either as part of the language or on the
excellent crates.io index. This
includes case folding, categorization, Unicode regular expressions,
Unicode normalization, well conforming URI/IRI/URL APIs, segmentation or
just simple things as name mappings.

What’s the downside? You can’t do "föo"[1] and expect 'ö' to come
back. But that’s not a good idea anyways.

As an example of how interaction with the OS works, here is an example
application that opens a file in the current working directory and prints
the contents and the filename:

use std::env;
use std::fs;

fn example() -> Result<(), Box<Error>> {
    let here = try!(env::current_dir());
    println!("Contents in: {}", here.display());
    for entry in try!(fs::read_dir(&here)) {
        let path = try!(entry).path();
        let md = try!(fs::metadata(&path));
        println!("  {} ({} bytes)", path.display(), md.len());
    }
    Ok(())
}

fn main() {
    example().unwrap();
}

All the IO operations use these Path objects that were also shown
before, which encapsulate the operating system’s internal path properly.
They might be bytes, unicode or whatever else the operating system uses
but the can be formatted properly by calling .display() on them which
return an object that can format itself into a string. This is convenient
because it means you never accidentally leak out bad strings like we do in
Python 3 for instance. There is a clear separation of concerns.

Distribution and Libraries

Rust comes with a combination of virtualenv+pip+setuptools called “cargo”.
Well, not entirely virtualenv as it can only work with one version of Rust
by default, but other than that it works as you expect. Even better than
in Python land can you depend on different versions of libraries and
depend on git repositories or the crates.io index. If you get rust from
the website it comes with the cargo command that does everything you
would expect.

Rust as a Python Replacement?

I don’t think there is a direct relationship between Python and Rust.
Python shines in scientific computing for instance and I don’t think that
this is something that can Rust tackle in the nearest future just because
of how much work that would be. Likewise there really is no point in
writing shell scripts in Rust when you can do that in Python. That being
said, I think like many Python programmers started to pick up Go, even
more will start to look at Rust for some areas where they previously used
Python for.

It’s a very powerful language, standing on strong foundations, under a
very liberal license, with a very friendly community and driving by a
democratic approach to language evolution.

Because Rust requires very little runtime support it’s very easy to use
via ctypes and CFFI with Python. I could very well envision a future
where there is a Python package that would allow the distribution of a
binary module written in Rust and callable from Python without any extra
work from the developer needed.