Be careful with exec and eval in Python

Behind the Scenes of Imports

Let’s start with everybody’s favourite topic: Performance. That’s
probably the most pointless argument against exec, but well, it’s
important to know though. But before that, let’s see what Python roughly
does if you import a module (import foo):

1. it locates the module (surprise). That happens by traversing the
   sys.path info in various ways. There is builtin import logic, there
   are import hooks and all in all there is a lot of magic involved I
   don’t want to go into. If you are curious, check this and this.
2. Now depending on the import hook responsible it might load bytecode
   (.pyc) or sourcecode (.py):
   1. If bytecode is available and the magic checksum matches the
      current Python interpreter’s version, the timestamp of the
      bytecode file is newer or equal to the source version (or the
      source does not exist) it will load that.
   2. If the bytecode is missing or outdated it will load the source
      file and compile that to bytecode. For that it checks magic
      comments in the file header for encoding settings and decodes
      according to those settings. It will also check if a special
      tab-width comment exists to treat tabs as something else than 8
      characters if necessary. Some import hooks will then generate
      .pyc files or store the bytecode somewhere else (__pycache__)
      depending on Python version and implementation.
3. The Python interpreter creates a new module object (you can do that on
   your own by calling imp.new_module or creating an instance of
   types.ModuleType. Those are equivalent) with a proper name.
4. If the module was loaded from a file the __file__ key is set. The
   import system will also make sure that __package__ and __path__
   are set properly if packages are involved before the code is executed.
   Import hooks will furthermore set the __loader__ variable.
5. The Python interpreter executes the bytecode in the context of the
   dictionary of the module. Thus the frame locals and frame globals for
   the executed code are the __dict__ attribute of that module.
6. The module is inserted into sys.modules.

Now first of all, none of the above steps ever passed a string to the
exec keyword or function. That’s obviously true because that happens
deep inside the Python interpreter unless you are using an import hook
written in Python. But even if the Python interpreter was written in
Python it would never pass a string to the exec function. So what would
you want to do if you want to get that string into bytecode yourself? You
would use the compile builtin:

>>> code = compile('a = 1 + 2', '<string>', 'exec')
>>> exec code
>>> print a
3

As you can see, exec happily executes bytecode too. Because the code
variable is actually an object of type code and not a string. The
second argument to compile is the filename hint. If we are compiling
from an actual string there we should provide a value enclosed in angular
brackets because this is what Python will do. <string> and
<stdin> are common values. If you do have a file to back this up,
please use the actual filename there. The last parameter is can be one of
'exec', 'eval' and 'single'. The first one is what exec is
using, the second is what the eval function uses. The difference is
that the first can contain statements, the second only expressions.
'single' is a form of hybrid mode which is useless for anything but
interactive shells. It exists solely to implement things like the
interactive Python shell and is very limited in use.

Here however we were already using a feature you should never, ever, ever
use: executing code in the calling code’s namespace. What should you do
instead? Execute against a new environment:

>>> code = compile('a = 1 + 2', '<string>', 'exec')
>>> ns = {}
>>> exec code in ns
>>> print ns['a']
3

Why should you do that? Cleaner for starters, also because exec without a
dictionary has to hack around some implementation details in the
interpreter. We will cover that later. For the moment: if you want to
use exec and you plan on executing that code more than once, make sure
you compile it into bytecode first and then execute that bytecode only and
only in a new dictionary as namespace.

In Python 3 the exec ... in statement disappeared and instead you can
use the new exec function which takes the globals and locals
dictionaries as parameters.

Performance Characteristics

Now how much faster is executing bytecode over creating bytecode and
executing that?:

$ python -mtimeit -s 'code = "a = 2; b = 3; c = a * b"' 'exec code'
10000 loops, best of 3: 22.7 usec per loop

$ python -mtimeit -s 'code = compile("a = 2; b = 3; c = a * b",
  "<string>", "exec")' 'exec code'
1000000 loops, best of 3: 0.765 usec per loop

32 times as fast for a very short code example. It becomes a lot worse
the more code you have. Why is that the case? Because parsing Python
code and converting that into Bytecode is an expensive operation compared
to evaluating the bytecode. That of course also affects execfile which
totally does not use bytecode caches, how should it. It’s not gonna
magically check if there is a .pyc file if you are passing the path to a
foo.py file.

Alright, lesson learned. compile + exec > exec. What else has to
be considered when using exec? The next thing you have to keep in mind
is that there is a huge difference between the global scope and the local
scope. While both the global scope and the local scope are using
dictionaries as a data storage, the latter actually is not. Local
variables in Python are just pulled from the frame local dictionary and
put there as necessary. For all calculations that happen between that,
the dictionary is never ever used. You can quickly verify this yourself.

Execute the following thing in the Python interpreter:

>>> a = 42
>>> locals()['a'] = 23
>>> a
23

Works as expected. Why? Because the interactive Python shell executes
code as part of the global namespace like any code outside of functions or
class declarations. The local scope is the global scope:

>>> globals() is locals()
True

Now what happens if we do this at function level?

>>> def foo():
...  a = 42
...  locals()['a'] = 23
...  return a
...
>>> foo()
42

How unfortunate. No magic variable changing for us. That however is only
partially correct. There is a Python opcode for synchronizing the frame
dictionary with the variables from the fast local slots. There are two
ways this synchronization can happen: from fast local to dictionary and
the other way round. The former is implicitly done for you when you call
locals() or access the f_locals attribute from a frame object, the
latter happens either explicitly when using some opcodes (which I don’t
think are used by Python as part of the regular compilation process but
nice for hacks) or when the exec statement is used in the frame.

So what are the performance characteristics of code executed in a global
scope versus code executed at a local scope? This is a lot harder to
measure because the timeit module does not allow us to execute code at
global scope by default. So we will need to write a little helper module
that emulates that:

code_global = compile('''
sum = 0
for x in xrange(500000):
    sum += x
''', '<string>', 'exec')
code_local = compile('''
def f():
    sum = 0
    for x in xrange(500000):
        sum += x
''', '<string>', 'exec')

def test_global():
    exec code_global in {}

def test_local():
    ns = {}
    exec code_local in ns
    ns['f']()

Here we compile two times the same algorithm into a string. One time
directly globally, one time wrapped into a function. Then we have two
functions. The first one executes that code in an empty dictionary, the
second executes the code in a new dictionary and then calls the function
that was declared. Let’s ask timeit how fast we are:

$ python -mtimeit -s 'from execcompile import test_global as t' 't()'
10 loops, best of 3: 67.7 msec per loop

$ python -mtimeit -s 'from execcompile import test_local as t' 't()'
100 loops, best of 3: 23.3 msec per loop

Again, an increase in performance [3]. Why is that? That has
to do with the fact that fast locals are faster than dictionaries (duh).
What is a fast local? In a local scope Python keeps track of the names of
variables it knows about. Each of that variable is assigned a number
(index). That index is used in an array of Python objects instead of a
dictionary. It will only fall back to the dictionary if this is necessary
(debugging purposes, exec statement used at local scope). Even though
exec still exists in Python 3 (as a function) you no longer it at a
local scope to override variables. The Python compiler does not check if
the exec builtin is used and will not unoptimize the scope because of
that [1].

All of the above knowledge is good to know if you plan on utilizing the
Python interpreter to interpret your own language by generating Python
code and compiling it to bytecode. That’s for instance how template
engines like Mako, Jinja2 or Genshi work internally in one way or another.

Semantics and Unwritten Conventions

However most people are using the exec statement for something else:
executing actual Python code from different locations. A very popular use
case is executing config files as Python code. That’s for example what
Flask does if you tell it to. That’s usually
okay because you don’t expect your config file to be a place where you
implement actual code. However there are also people that use exec to
load actual Python code that declares functions and classes. This is a
very popular pattern in some plugin systems and the web2py framework.

Why is that not a good idea? Because it breaks some (partially unwritten)
conventions about Python code:

1. Classes and functions belong into a module. That basic rule holds for
   all functions and classes imported from regular modules:

   >>> from xml.sax.saxutils import quoteattr
   >>> quoteattr.__module__
   'xml.sax.saxutils'
   

   Why is that important? Because that is how pickle works [2]:

   >>> pickle.loads(pickle.dumps(quoteattr))
   <function quoteattr at 0x1005349b0>
   >>> quoteattr.__module__ = 'fake'
   >>> pickle.loads(pickle.dumps(quoteattr))
   Traceback (most recent call last):
     ..
   pickle.PicklingError: Can't pickle quoteattr: it's not found as fake.quoteattr
   Traceback (most recent call last):
     ..
   pickle.PicklingError: Can't pickle quoteattr: it's not found as fake.quoteattr
   

   If you are using exec to execute Python code, be prepared that
   some modules like pickle, inspect, pkgutil, pydoc and probably some
   others that depend on those will not work as expected.

2. CPython has a cyclic garbage collector, classes can have destructors
   and interpreter shutdown breaks up cycles. What does it mean?

   • CPython uses refcounting internally. One (of many) downsides of
     refcounting is that it cannot detect circular dependencies between
     objects. Thus Python introduced a cyclic garbage collector at one
     point.
   • Python however also allows destructors on objects. Destructors
     however mean that the cyclic garbage collector will skip these
     objects because it does not know in what order it should delete
     these objects.

   Now let’s look at an innocent example:

   class Foo(object):
       def __del__(self):
           print 'Deleted'
   foo = Foo()
   

   Let’s execute that file:

   $ python test.py
   Deleted
   

   Looks good. Let’s try that with exec:

   >>> execfile('test.py', {})
   >>> execfile('test.py', {})
   >>> execfile('test.py', {})
   >>> import gc
   >>> gc.collect()
   27
   

   It clearly collected something, but it never collected our Foo
   instances. What the hell is happening? What’s happening is that
   there is an implicit cycle between foo, and the __del__ function
   itself. The function knows the scope it was created in and from
   __del__ -> global scope -> foo instance it has a nice cycle.

   Now now we know the cause, why doesn’t it happen if you have a module?
   The reason for that is that Python will do a trick when it shuts down
   modules. It will override all global values that do not begin with an
   underscore with None. We can easily verify that if we print the
   value of foo instead of 'Deleted':

   class Foo(object):
       def __del__(self):
           print foo
   foo = Foo()
   

   And of course it’s None:

   $ python test.py
   None
   

   So if we want to replicate that with exec or friends, we have to
   apply the same logic, but Python will not do that for us. If we are
   not careful this could lead to hard to spot memory leaks. And this is
   something many people rely on, because it’s documented behaviour.

3. Lifetime of objects. A global namespace sticks around from when it
   was imported to the point where the interpreter shuts down. With
   exec you as a user no longer know when this will happen. It might
   happen at a random point before. web2py is a common offender here.
   In web2py the magically executed namespace comes and goes each request
   which is very surprising behaviour for any experienced Python
   developer.

Python is not PHP

Don’t try to circumvent Python idioms because some other language does it
differently. Namespaces are in Python for a reason and just because it
gives you the tool exec it does not mean you should use that tool. C
gives you setjmp and longjmp and yet you will be very careful with
using it. The combination of exec and compile are a powerful tool for
anyone that wants to implement a domain specific language on top of Python
or for developers interested in extending (not circumventing) the Python
import system.

A python developer depends on imports doing what they are documented to do
and that the namespace has a specific initial value (namely that it’s
empty with the exception of a few internal variables such as __name__,
__loader__ etc.). A Python developer depends on being able to import
that module by a dotted name, on the fact that modules shut down in a
specific way, that they are cached in sys.modules etc.

Jacob Kaplan-Moss wrote a comment on Reddit
about the use of exec in web2py a while ago which I would recommend
reading. He brings up some very good points why changing the semantics of
a language is a bad idea.

However web2py and it’s use of execfile are not the only offenders in
the Python web community. Werkzeug has it’s fair share of abusing Python
conventions as well. We were shipping (and still do) an on-demand import
system which caused more problems than it solved and are currently in the
progress of moving away from it (despite all the pain this is for us).
Django abused Python internals as well. It was generating Python code on
the fly and totally changing semantics (to the point where imports
vanished without warning!). They learned their lesson as well and fixed
that problem in the magic removal branch. Same goes for web.py which was
abusing the print statement to write into an internal thread-local
buffer that was then sent out as response to the browser. Also something
that turned out to be a bad idea and was subsequently removed.

With that I encourage the web2py developers to reconsider their decision
on the use of the exec statement and using regular Python modules.

Because one of the things we all have to keep in mind: if a Python
developer starts his journeys in the twisted world of wrongly executed
Python modules they will be very confused when they continue their travels
in another Python environment. And having different semantics in
different frameworks/modules/libraries is very hurtful for Python as a
runtime and language.