Git-pandas caching for faster analysis

This post was originally published here

Git-pandas is a python library I wrote to help make analysis of git data easier when dealing with collections of repositories.  It makes a ton of cool stuff easier, like cumulative blame plots, but they can be kind of slow, especially with many large repositories. In the past we've made that work with running analyses offline, and by sampling, but really most of the work run-to-run is repeated.

Enter caching. There are a few places in the codebase that we can cache result-sets by revision key and get pretty significant performance boosts when using the library in something like gitnoc. And it turns out, it's pretty straight forward.

Currently in develop, we've got a new module with a custom python decorator to handle caching by different mechanisms:

def multicache(key_prefix, key_list, skip_if=None):
    def multicache_nest(func):
        def deco(self, *args, **kwargs):
            if self.cache_backend is None:
                return func(self, *args, **kwargs)
                if skip_if is not None:
                    if skip_if(kwargs):
                        return func(self, *args, **kwargs)

                key = key_prefix + self.repo_name + '_'.join([str(kwargs.get(k)) for k in key_list])
                    if isinstance(self.cache_backend, EphemeralCache):
                        ret = self.cache_backend.get(key)
                        return ret
                    elif isinstance(self.cache_backend, RedisDFCache):
                        ret = self.cache_backend.get(key)
                        return ret
                        raise ValueError('Unknown cache backend type')
                except CacheMissException as e:
                    ret = func(self, *args, **kwargs)
                    self.cache_backend.set(key, ret)
                    return ret

        return deco
    return multicache_nest

It looks pretty convoluted, but ends up being pretty useful.  It creates a decorator that we can use on any method in the Repository class, where one can specify a caching_backend (currently we have in-memory-ephemeral and redis based options), a key_prefix to use, a list of kwarg keys to use in the cache key, and optionally a lambda function to apply over the kwargs that returns whether to skip caching.

The lambda is in particular useful for cases we have like not wanting to cache the results for rev='HEAD', since that can change moment to moment.

Each of the two caching backends implements your basic get/set/purge functionality, and lets you set a maximum number of keys to have something like an LRU cache.

One interesting nugget from the Redis cache was that the objects we are caching are always, in git-pandas, pandas dataframes. To store those in Redis we can serialize/deserialize the dfs with:

# self._cache is a connection to redis
self._cache.set(k, v.to_msgpack(compress='zlib'), ex=self.ttl)
df = pd.read_msgpack(self._cache.get(k))

It's still being tested, and is probably one of the last things we will cram in before releasing git-pandas 2.0.0, so check out the repository over on github, try it out, and let me know what you think.

Related Posts

Intro to Python, Epilogue – Only the Beginning Congratulations, dear reader, on completing your quest. But be warned, opening the Door was but the first step of your journey. You now have an under...
Intro to Python, Chapter 8 – Opening the Door Baldric was suddenly standing in a crowd of soldiers. Not Imperial soldiers thankfully. These were more ragtag, wearing random bits of armor here and...
Intro to Python, Chapter 7 – Imports and Nesting Baldric had been a town guard for years, breaking up fights and catching thieves and helping the elderly cross the street. When the Empire arose and ...
Intro to Python, Chapter 6 – Built-in Functions and Methods Magic-wielding escapees were desperately deflecting the Imperial attacks on the fortress’ supports as best they could. Ranks of longbowmen rain...