Dealing with the Python Import Blackbox

The Use Case

The problem arises when you start doing things and want to respond to
errors. A good example are imports where you try to import something and
if that fails you want to do something else. For instance you have a
module name as a string and you want to try to import that. If that
module does not exist (not if it fails to import!) you want to do
something else. Django’s middlewares for instance are defined as strings
in the configuration module and if there is a typo you want to tell the
users where the problem is.

If you import module A and if that does not exist you want to fall back to
module B, you don’t want to swallow the import error of module A since
that one might have been a dependency that failed loading.

Consider you have a module called foo that depends on a module named
bar. If foo does not exist you want to retry with simplefoo. This
is what nearly everybody is doing:

try:
    import foo
except ImportError:
    import simplefoo as foo

However if now foo is failing to import because bar is missing you get
the import error “No module named simplefoo” even though the correct error
would have been “No module named bar”.

The Problem

The problem is that Python does not provide you with information if the
module was not found or failed to import. In theory you could build
yourself something with the imp module that splits up finding and
loading but there are a handful of problems with that:

1. The Python import process is notoriously underspecified and exploited
   in various ways. Just because an importer says it finds a module it
   does not mean it can properly import it. For instance there are many
   finders that will tell you that find_module succeeded just to fail
   later with an error on load_module.
2. The Python import machinery is complex and even with the new
   importlib module everything but easy to use. To replicate the logic
   that Python is applying to locate modules you need around 80 lines of
   code, even with importlib available.
3. The import process is highly dynamic and there are various ways in
   which people can customize the importing, going beyond what is
   possible with regular import hooks by overriding __import__.

The second possibility that is actually in use sometimes is parsing the
error message of the import error. This however is a lost cause because
the error message is implementation defined and differs quite often. On
top of that is the import machinery in Python a recursive process and
gives very awkward results:

>>> import missing_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named missing_module

>>> import missing_package.missing_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named missing_package.missing_module

>>> import xml.missing_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named missing_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named missing_module

As you can see, the error message does not even include the whole import
path at all times. Sometimes the error message is something completely
unrelated, sometimes the whole error message is just the module name.
Sometimes it’s “No module named %s”, sometimes the module name is on
quotes. This is because various parts of the system can abort an import
process and since this is customizable …

Import Process Details

The way imports work is that at a very early point an entry in
sys.modules is created for the new module. When the module code is
executed it will be executed in a frame where the globals of the frame are
the dictionary of the module in sys.modules. As such this is valid in
Python:

import sys
a_value = [1, 2, 3]
this = sys.modules[__name__]
assert a_value is this.a_value

Now in theory one could think that if an import fails we will have a
partial entry in sys.modules left to introspect if the import failed at
a later point. This however is usually not the case because on import
errors caused by the actual importers an importer is required to remove
the entry in sys.modules again so we don’t have much luck there.

Consider this fail_module.py:

import sys

# this works
this = sys.modules['fail_module']

# this fails
import missing_module

If we however attempt to access fail_module later it will be gone:

>>> import sys
>>> import fail_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "fail_module.py", line 7, in <module>
    import missing_module
ImportError: No module named missing_module
>>> import sys
>>> 'fail_module' in sys.modules
False
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "fail_module.py", line 7, in <module>
    import missing_module
ImportError: No module named missing_module

Since we also can’t replace sys.modules with a custom data structure
where we get callbacks when things are inserted we have no chance there.

Sidechannels

I had to solve this problem again yesterday when I worked on a way to get
rid of namespace packages in Flask without
pissing existing users off. I think I found something that works reliable
enough where I don’t want to shoot myself for writing the code.

The idea is that if you get an import error you don’t only get an import
error but also a traceback object if you want. And that traceback object
has all the frames of the traceback linked to it. If you walk the
traceback you can find out if at any point the module you attempted to
import was involved. If that was the case, the module succeeded in
loading and something that it did resulted in an import error.

Now obviously there are downsides of this approach, so let’s go over them:

1. It assumes that the module we import does not override __name__.
   Since that is a horrible idea anyways that’s something we can ignore.
2. It assumes that there will be at least one traceback frame originating
   from that module. This will not be the case if that module was a C
   module that dynamically imported another module. This however is
   negligible since this is on the one hand a very uncommon thing to do
   and secondly this comes with its own set of problems.
3. It walks a traceback so your JIT will not be happy with that. On the
   other hand you should only import modules in non critical code paths
   anyways.

So how does the code look?

import sys

def import_module(module_name):
    try:
        __import__(module_name)
    except ImportError:
        exc_type, exc_value, tb_root = sys.exc_info()
        tb = tb_root
        while tb is not None:
            if tb.tb_frame.f_globals.get('__name__') == module_name:
                raise exc_type, exc_value, tb_root
            tb = tb.tb_next
        return None
    return sys.modules[module_name]

You can use it like this:

json = import_module('simplejson')
if json is None:
    json = import_module('json')
    if json is None:
        raise RuntimeError('Unable to find a json implementation')

Generally the implementation is straightforward. Try to import with
__import__, if that fails get the current traceback and see if any of
the frames originated in the module we tried to import. If that is the
case, we reraise the exception with the original traceback, otherwise just
return None to mark a missing module.

Since None has a special meaning in sys.modules which marks an import
error we know that an imported module never is None and we can use this
as return value to indicate a module that does not exist. If we would
instead raise an exception we would have the very same problem again since
exceptions bubble up and we don’t know if someone would handle it. So
raising something like ModuleNotFound instead of returning None would
cause troubles if the module we import recursively imports something with
import_module and does not handle the exception.

Why does it work?

Now you would think this only makes sense that it works, but it actually
surprised me that it does. The reason it surprises me is that Python
normally shuts down modules in a very weird way by setting all the values
in the global dictionary to None. Since the actual modules is long
gone when you get the import error you would think that the reference to
the globals you have is full of Nones and the names would never be
the module name.

To quote the documentation:

Starting with version 1.5, Python guarantees that globals whose name
begins with a single underscore are deleted from their module before
other globals are deleted; if no other references to such globals
exist, this may help in assuring that imported modules are still
available at the time when the __del__ method is called.

This however is only true when the module is shut down when the
interpreter is shutting down, not when the module is garbage collected.
And with that, the above hack works. If Python would do what the
documentation says in the module destructor instead of the interpreter
shutdown code our hack would not work.

Also this requires that a traceback object indeed still owns a reference
to f_globals. Now if you look at the traceback output itself you will
never see information that needs to be derived from the module global
dictionary so it appears to be implementation specific functionality that
is not guaranteed. However, and here is the catch. The import hook
protocol also specifies that a module can inject __loader__ into the
frame so that the source can be loaded from the __loader__ if the source
is not based on the filesystem. And for this to work the globals have to
be there. On top of that this also gives us confirmation that garbage
collected modules must not clear out their globals with Nones or we
would not be able to extract the sourcecode for certain import hooks when
an import error occurs since the loader would be gone.

And with that, the above hack suddenly looks quite reasonable and
supported again.