Implementing a factory for creating decorators.

This is the third post in my series of blog posts about Python decorators and how I believe they are generally poorly implemented. It follows on from the previous post titled 'The interaction between decorators and descriptors', with the very first post in the series being 'How you implemented your Python decorator is wrong'.

In the very first post I described a number of ways in which the traditional way that Python decorators are implemented is lacking. These were:

• Preservation of function __name__ and __doc__.
• Preservation of function argument specification.
• Preservation of ability to get function source code.
• Ability to apply decorators on top of other decorators that are implemented as descriptors.

In the followup post I described a pattern for implementing a decorator which built on top of what is called an object proxy, with the object proxy solving the first three issues. The final issue was dealt with by creating a function wrapper using the object proxy, which was implemented as a descriptor, and which performed object binding when a wrapper was used on class methods. This combination of the object proxy and a descriptor ensured that introspection continued to work properly and that the execution model of the Python object model was also respected.

The issue at this point was how to make the solution more usable, eliminating the boiler plate and minimising the amount of code that someone implementing a decorator would need to write.

In this post I will describe one such approach to simplifying the task of creating a decorator based on this pattern. This will be done by using a decorator as a factory to create decorators, requiring a user to only have to supply a single wrapper function which does the actual work of invoking the wrapped function, inserting any extra work that the specific decorator is intended to carry out as necessary.

Pattern for implementing the decorator

Just to refresh where we got to last time, we had an implementation of an object proxy as:

class object_proxy(object): 

    def __init__(self, wrapped):
        self.wrapped = wrapped
        try:
            self.__name__= wrapped.__name__
        except AttributeError:
            pass 

    @property
    def __class__(self):
        return self.wrapped.__class__  

    def __getattr__(self, name):
        return getattr(self.wrapped, name)

As pointed out the last time, this is a minimal representation of what it does. In practice it actually needs to do a lot more than this if it is to serve as a general purpose object proxy usable in the more generic use case of monkey patching.

The decorator itself would then be implemented per the pattern:

class bound_function_wrapper(object_proxy): 

    def __init__(self, wrapped):
        super(bound_function_wrapper, self).__init__(wrapped) 

    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs) 

class function_wrapper(object_proxy): 

    def __init__(self, wrapped):
       super(function_wrapper, self).__init__(wrapped) 

    def __get__(self, instance, owner):
        wrapped = self.wrapped.__get__(
instance, owner)
        return bound_function_wrapper(wrapped) 

    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs) 

When the wrapper is applied to a normal function, the __call__() method of the wrapper is used. If the wrapper is applied to a method of a class, the __get__() method is called when the attribute is accessed, which returns a new bound wrapper and the __call__() method of that is invoked instead when a call is made. This allows our wrapper to be used around descriptors as it propagates the descriptor protocol, also binding the wrapped object as necessary.

A decorator for creating decorators

So we have a pattern for implementing a decorator that appears to work correctly, but as already mentioned, needing to do all that each time is more work than we really want. What we can do therefore is create a decorator to help us create decorators. This would reduce the code we need to write for each decorator to a single function, allowing us to simplify the code to just:

@decorator
def my_function_wrapper(wrapped, args, kwargs):
    return wrapped(*args, **kwargs) 

@my_function_wrapper
def function():
    pass

What would this decorator factory need to look like?

As it turns out, our decorator factory is quite simple and isn't really much different to using a partial(), combining our new wrapper argument from when the decorator is defined, with the wrapped function when the decorator is used and passing them into our function wrapper object.

def decorator(wrapper):
    @functools.wraps(wrapper)
    def _decorator(wrapped):
        return function_wrapper(wrapped, wrapper)
    return _decorator

We now just need to amend our function wrapper implementation to delegate the actual execution of the wrapped object to the user supplied decorator wrapper function.

class bound_function_wrapper(object_proxy): 

    def __init__(self, wrapped, wrapper):
        super(bound_function_wrapper, self).__init__(wrapped)
        self.wrapper = wrapper 

    def __call__(self, *args, **kwargs):
        return self.wrapper(self.wrapped, args, kwargs) 

class function_wrapper(object_proxy): 

    def __init__(self, wrapped, wrapper):
        super(function_wrapper, self).__init__(wrapped)
        self.wrapper = wrapper 

    def __get__(self, instance, owner):
        wrapped = self.wrapped.__get__(instance, owner)
        return bound_function_wrapper(wrapped, self.wrapper) 

    def __call__(self, *args, **kwargs):
        return self.wrapper(self.wrapped, args, kwargs)

The __call__() method of our function wrapper, for when it is used around a normal function, now just calls the user supplied decorator wrapper function with the wrapped function and arguments, leaving the calling of the wrapped function up to the user supplied decorator wrapper function.

In the case where binding a function, the wrapper is also passed to the bound wrapper. The bound wrapper is more or less the same, with the __call__() method delegating to the user supplied decorator wrapper function.

So we can make creating decorators easier using a factory. Lets now check that this will in fact work in all cases in which it could be applied and also see what other problems we can find and whether we can improve on those situations as well.

Decorating methods of classes

The first such area which can cause problems is creating a single decorator that can work on both normal functions and instance methods of classes.

To test out how our new decorator works, we can print out the args passed to the wrapper when the wrapped function is called and can compare the results.

@decorator
def my_function_wrapper(wrapped, args, kwargs):
    print('ARGS', args)
    return wrapped(*args, **kwargs)

First up lets try wrapping a normal function:

@my_function_wrapper
def function(a, b):
    pass 

>>> function(1, 2)
ARGS (1, 2)

As would be expected, just the two arguments passed when the function is called are output.

What about when wrapping an instance method?

class Class(object):
    @my_function_wrapper
    def function_im(self, a, b):
        pass 

c = Class() 

>>> c.function_im()
ARGS (1, 2)

Once again just the two arguments passed when the instance method is called are displayed. How the decorator works for both the normal function and the instance method is therefore the same.

The problem here is what if the user within their decorator wrapper function wanted to know what the actual instance of the class was. We have lost that information when the function was bound to the instance of the class as it is now associated with the bound function passed in, rather than the argument list.

To solve this problem we can remember what the instance was that was passed to the __get__() method when it was called to bind the function. This can then be passed through to the bound wrapper when it is created.

class bound_function_wrapper(object_proxy): 

    def __init__(self, wrapped, instance, wrapper):
        super(bound_function_wrapper, self).__init__(wrapped)
        self.instance = instance
        self.wrapper = wrapper 

    def __call__(self, *args, **kwargs):
        return self.wrapper(self.wrapped, self.instance, args, kwargs) 

class function_wrapper(object_proxy): 

    def __init__(self, wrapped, wrapper):
        super(function_wrapper, self).__init__(wrapped)
        self.wrapper = wrapper 

    def __get__(self, instance, owner):
        wrapped = self.wrapped.__get__( instance, owner)
        return bound_function_wrapper(wrapped, instance, self.wrapper) 

    def __call__(self, *args, **kwargs):
        return self.wrapper(self.wrapped, None, args, kwargs)

In the bound wrapper, the instance pointer can then be passed through to the decorator wrapper function as an extra argument. To be uniform for the case of a normal function, in the top level wrapper we pass None for this new instance argument.

We can now modify our wrapper function for the decorator to output both the instance and the arguments passed.

@decorator
def my_function_wrapper(wrapped, instance, args, kwargs):
    print('INSTANCE', instance)
    print('ARGS', args)
    return wrapped(*args, **kwargs) 

>>> function(1, 2)
INSTANCE None
ARGS (1, 2) 

>>> c.function_im(1, 2)
INSTANCE <__main__.Class object at 0x1085ca9d0>
ARGS (1, 2)

This change therefore allows us to be able to distinguish between a normal function call and an instance method call within the one decorator wrapper function. The reference to the instance is even passed separately so we don't have to juggle with the arguments to move it out of the way for an instance method when calling the original wrapped function.

Now there is one final scenario in which an instance method can be called which we still need to check. This is calling an instance method by calling the function on the class and passing the object instance explicitly as the first argument.

>>> Class.function_im(c, 1, 2)
INSTANCE None
ARGS (<__main__.Class object at 0x1085ca9d0>, 1, 2)

Unfortunately passing in the instance explicitly as an argument against the function from the class, results in the instance passed to the decorator wrapper function being None, with the reference to the instance getting passed through as the first argument instead. This isn't really a desirable outcome.

To deal with this variation, we can check for instance being None before calling the decorator wrapper function and pop the instance off the start of the argument list. We then use a partial to bind the instance to the wrapped function ourselves and call the decorator wrapper function.

class bound_function_wrapper(object_proxy): 

    def __call__(self, *args, **kwargs):
        if self.instance is None:
            instance, args = args[0], args[1:]
            wrapped = functools.partial(self.wrapped, instance)
            return self.wrapper(wrapped, instance, args, kwargs) 

        return self.wrapper(self.wrapped, self.instance, args, kwargs)

We then get the same result no matter whether the instance method is called via the class or not.

>>> Class.function_im(c, 1, 2)
INSTANCE <__main__.Class object at 0x1085ca9d0>
ARGS (1, 2)

So everything works okay for instance methods, with the argument list seen by the decorator wrapper function being the same as if a normal function had been wrapped. At the same time though, by virtue of the new instance argument, we can if need be act on the instance of a class where the decorator was applied to an instance method of a class.

What about other method types that a class can have, specifically class method and static methods.

class Class(object):
    @my_function_wrapper
    @classmethod
    def function_cm(cls, a, b):
        pass 

>>> Class.function_cm(1, 2)
INSTANCE 1
ARGS (2,)

As can be seen, this fiddle has though upset things for when we have a class method, also causing the same issue for a static method. In both those cases the instance would initially have been passed as None when the function was bound. The result is that the real first argument ends up as the instance, which is obviously going to be quite wrong.

What to do?

A universal decorator

So we aren't quite there yet, but what are we trying to achieve in even trying to do this? What was wrong with our initial pattern for the decorator?

The ultimate goal here is what I call a universal decorator. A single decorator that can be applied to a normal function, an instance method, a class method, a static method or even a class, with the decorator being able to determine at the point it was called the context in which it was used.

Right now with the way that decorators are normally implemented this is not possible. Instead different decorators are provided to be used in the different contexts, meaning duplication of the code into each, or the use of hacks to try and convert decorators created for one purpose so they can be used in a different context.

What I am instead aiming for is the ability to do:

@decorator
def universal(wrapped, instance, args, kwargs):
    if instance is None:
        if inspect.isclass(wrapped):
            # class.
        else:
            # function or staticmethod.
    else:
        if inspect.isclass(instance):
            # classmethod.
        else:
            # instancemethod.

At this point we have got things working for normal functions and instance methods, we just now need to work out how to handle class methods, static methods and the scenario where a decorator is applied to a class.

The next post in this series will continue to pursue this goal and describe how our decorator can be tweaked further to get there.