The interaction between decorators and descriptors.

This is the second post in my series of blog posts about Python decorators and how I believe they are generally poorly implemented. It follows on from the first post titled 'How you implemented your Python decorator is wrong'.

In that first post I described a number of ways in which the traditional way that Python decorators are implemented is lacking. These were:

• Preservation of function __name__ and __doc__.
• Preservation of function argument specification.
• Preservation of ability to get function source code.
• Ability to apply decorators on top of other decorators that are implemented as descriptors.

I described previously how functools.wraps() attempts to solve the problem with preservation of the introspection of the __name__ and __doc__ attributes, but highlight one case in Python 2 where it can fail, and also note that it doesn't help with the preservation of the function argument specification nor the ability to access the source code.

In this post I want to focus mainly on the last of the issues above. That is the interaction between decorators and descriptors, where a function wrapper is applied to a Python object which is actually a descriptor.

What are descriptors?

I am not going to give an exhaustive analysis of what descriptors are or how they work so if you want to understand them in depth, I would suggest reading up about them elsewhere.

In short though, a descriptor is an object attribute with “binding behaviour”, one whose attribute access has been overridden by methods in the descriptor protocol. Those methods are __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor.

• obj.attribute

      --> attribute.__get__(obj, type(obj))

• obj.attribute = value
  
    --> attribute.__set__(obj, value)

• del obj.attribute
          
      --> attribute.__delete__(obj)

What this means is that if an attribute of a class has any of these special methods defined, when the corresponding operation is performed on that attribute of a class, then those methods will be called instead of the default action. This allows an attribute to override how those operations are going to work.

You may well be thinking that you have never made use of descriptors, but fact is that function objects are actually descriptors. When a function is originally added to a class definition it is as a normal function. When you access that function using a dotted attribute path, you are invoking the __get__() method to bind the function to the class instance, turning it into a bound method of that object.

def f(obj): pass
>>> hasattr(f, '__get__')
True 

>>> f
<function f at 0x10e963cf8> 

>>> obj = object()
>>> f.__get__(obj, type(obj))
<bound method object.f of <object object at 0x10e8ac0b0>>

So when calling a method of a class, it is not the __call__() method of the original function object that is called, but the __call__() method of the temporary bound object that is created as a result of accessing the function.

You of course don't usually see all these intermediary steps and just see the outcome.

>>> class Object(object):
...   def f(self): pass 

>>> obj = Object()
>>> obj.f
<bound method Object.f of <__main__.Object object at 0x10abf29d0>>

Looking back now at the example given in the first blog post where we wrapped a decorator around a class method, we encountered the error:

class Class(object):
    @function_wrapper
    @classmethod
    def cmethod(cls):
        pass 

Class.cmethod() 

Traceback (most recent call last):
  File "classmethod.py", line 15, in <module>
    Class.cmethod()
  File "classmethod.py", line 6, in _wrapper
    return wrapped(*args, **kwargs)
TypeError: 'classmethod' object is not callable

The problem with this example was that for the @classmethod decorator to work correctly, it is dependent on the descriptor protocol being applied properly. This is because the __call__() method only exists on the result returned by __get__() when it is called, there is no __call__() method on the @classmethod decorator itself.

More specifically, the simple type of decorator that people normally use is not itself honouring the descriptor protocol and applying that to the wrapped object to yield the bound function object which should actually be called. Instead it is simply calling the wrapped object directly, which will fail if it doesn't have a __call__().

Why then does applying a decorator to a normal instance method still work?

This still works because a normal function still has a __call__() method. In bypassing the descriptor protocol of the wrapped function it is calling this. Although the binding protocol is side stepped, things still work out because the wrapper will pass the 'self' argument for the instance explicitly as the first argument when calling the original unbound function object.

For a normal instance method the result in this situation is effectively the same. It only falls apart when the wrapped object, as in the case of @classmethod, are dependent on the descriptor protocol being applied correctly.

Wrappers as descriptors

The way to solve this problem where the wrapper is not honouring the descriptor protocol and performing binding on the wrapped object in the case of a method on a class, is for wrappers to also be descriptors.

class bound_function_wrapper(object): 

    def __init__(self, wrapped):
        self.wrapped = wrapped 

    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs) 

class function_wrapper(object): 

    def __init__(self, wrapped):
        self.wrapped = wrapped 

    def __get__(self, instance, owner):
        wrapped = self.wrapped.__get__(
instance, owner)
        return bound_function_wrapper(wrapped) 

    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs)

If the wrapper is applied to a normal function, the __call__() method of the wrapper is used. If the wrapper is applied to a method of a class, the __get__() method is called, which returns a new bound wrapper and the __call__() method of that is invoked instead. This allows our wrapper to be used around descriptors as it propagates the descriptor protocol.

So since using a function closure will ultimately fail if used around a decorator which is implemented as a descriptor, the situation we therefore have is that if we want everything to work, then decorators should always use a class based wrapper, where the class implements the descriptor protocol as shown.

The question now is how do we address the other issues that were listed.

We solved naming using functools.wrap()/functools.update_wrapper() before, but what do they do and can we still use them.

Well wraps() just uses update_wrapper(), so we just need to look at it.

WRAPPER_ASSIGNMENTS = ('__module__',
       '__name__', '__qualname__', '__doc__',
       '__annotations__')

WRAPPER_UPDATES = ('__dict__',) 

def update_wrapper(wrapper, wrapped,
        assigned = WRAPPER_ASSIGNMENTS,
        updated = WRAPPER_UPDATES): 

    wrapper.__wrapped__ = wrapped 

    for attr in assigned:
        try:
            value = getattr(wrapped, attr)
        except AttributeError:
            pass
        else:
            setattr(wrapper, attr, value) 

    for attr in updated:
        getattr(wrapper, attr).update(
                getattr(wrapped, attr, {}))

What is shown here is what is in Python 3.3, although that actually has a bug in it, which is fixed in Python 3.4. :-)

Looking at the body of the function, three things are being done. First off a reference to the wrapped function is saved as __wrapped__. This is the bug, as it should be done last.

The second is to copy those attributes such as __name__ and __doc__.

Finally the third thing is to copy the contents of __dict__ from the wrapped function into the wrapper, which could actually result in quite a lot of objects needing to be copied.

If we are using a function closure or straight class wrapper this copying is able to be done at the point that the decorator is applied.

With the wrapper being a descriptor though, it technically now also needs to be done in the bound wrapper.

class bound_function_wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped
        functools.update_wrapper(self, wrapped) 

class function_wrapper(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped
        functools.update_wrapper(self, wrapped)

As the bound wrapper is created every time the wrapper is called for a function bound to a class, this is going to be too slow. We need a more performant way of handling this.

Transparent object proxy

The solution to the performance issue is to use what is called an object proxy. This is a special wrapper class which looks and behaves like what it wraps.

class object_proxy(object): 

    def __init__(self, wrapped):
        self.wrapped = wrapped
        try:
            self.__name__= wrapped.__name__
        except AttributeError:
            pass 

    @property
    def __class__(self):
        return self.wrapped.__class__ 

    def __getattr__(self, name):
        return getattr(self.wrapped, name)

A fully transparent object proxy is a complicated beast in its own right, so I am going to gloss over the details for the moment and cover it in a separate blog post at some point.

The above example though is a minimal representation of what it does. In practice it actually needs to do a lot more than this though if it is to serve as a general purpose object proxy usable in the more generic use case of monkey patching.

In short though, it copies limited attributes from the wrapped object to itself, and otherwise uses special methods, properties and __getattr__() to fetch attributes from the wrapped object only when required thereby avoiding the need to copy across lots of attributes which may never actually be accessed.

What we now do is derive our wrapper class from the object proxy and do away with calling update_wrapper().

class bound_function_wrapper(object_proxy):
    def __init__(self, wrapped):
        super(bound_function_wrapper, self).__init__(wrapped)

    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs)  

class function_wrapper(object_proxy):
    def __init__(self, wrapped):
       super(function_wrapper, self).__init__(wrapped)

    def __get__(self, instance, owner):
        wrapped = self.wrapped.__get__(
instance, owner)
        return bound_function_wrapper(wrapped) 

    def __call__(self, *args, **kwargs):
        return self.wrapped(*args, **kwargs) 

In doing this, attributes like __name__ and __doc__, when queried from the wrapper, return the values from the wrapped function. We don't therefore as a result have the problem we did before where details were being returned from the wrapper instead.

Using a transparent object proxy in this way also means that calls like inspect.getargspec() and inspect.getsource() will now work and return what we expect. So we have actually managed to solve those two problems at the same time without any extra effort, which is a bonus.

Making this all more usable

Although this pattern addresses the problems which were originally identified, it consists of a lot of boiler plate code. Further, you now have two places in the code where the wrapped function is actually being called where you would need to insert the code to implement what the decorator was intended to do.

Replicating this every time you need to implement a decorator would therefore be a bit of a pain.

What we can instead do is wrap this all up and package it up into a decorator factory, thereby avoiding the need for this all to be done manually each time. How to do that will be the subject of the next blog post in this series.

From that point we can start to look at how we can further improve the functionality and introduce new capabilities which are generally hard to pull off with the way that decorators are normally implemented.

And before people start to complain that using this pattern is going to be too slow in the general use case, I will also address that in a future post as well, so just hold your complaints for now.