Intercepting Built-in Operation Attributes

When I introduced __getattr__ and __getattribute__, I stated that they intercept undefined and all attribute fetches, respectively, which makes them ideal for delegation-based coding patterns. While this is true for normally named attributes, their behavior needs some additional clarification: for method-name attributes implicitly fetched by built-in operations, these methods may not be run at all. This means that operator overloading method calls cannot be delegated to wrapped objects unless wrapper classes somehow redefine these methods themselves.

For example, attribute fetches for the __str__, __add__, and __getitem__ methods run implicitly by printing, + expressions, and indexing, respectively, are not routed to the generic attribute interception methods in 3.0. Specifically:

广告：个人专属 VPN，独立 IP，无限流量，多机房切换，还可以屏蔽广告和恶意软件，每月最低仅 5 美元

 

 
  • In Python 3.0, neither __getattr__ nor __getattribute__ is run for such attributes.
  • In Python 2.6, __getattr__ is run for such attributes if they are undefined in the class.
  • In Python 2.6, __getattribute__ is available for new-style classes only and works as it does in 3.0.

In other words, in Python 3.0 classes (and 2.6 new-style classes), there is no direct way to generically intercept built-in operations like printing and addition. In Python 2.X, the methods such operations invoke are looked up at runtime in instances, like all other attributes; in Python 3.0 such methods are looked up in classes instead.

This change makes delegation-based coding patterns more complex in 3.0, since they cannot generically intercept operator overloading method calls and route them to an embedded object. This is not a showstopper—wrapper classes can work around this constraint by redefining all relevant operator overloading methods in the wrapper itself, in order to delegate calls. These extra methods can be added either manually, with tools, or by definition in and inheritance from common superclasses. This does, however, make wrappers more work than they used to be when operator overloading methods are a part of a wrapped object’s interface.

Keep in mind that this issue applies only to __getattr__ and __getattribute__. Because properties and descriptors are defined for specific attributes only, they don’t really apply to delegation-based classes at all—a single property or descriptor cannot be used to intercept arbitrary attributes. Moreover, a class that defines both operator overloading methods and attribute interception will work correctly, regardless of the type of attribute interception defined. Our concern here is only with classes that do not have operator overloading methods defined, but try to intercept them generically.

Consider the following example, the file getattr.py, which tests various attribute types and built-in operations on instances of classes containing __getattr__ and __getattribute__ methods:

class GetAttr:
    eggs = 88             # eggs stored on class, spam on instance
    def __init__(self):
       self.spam = 77
    def __len__(self):    # len here, else __getattr__ called with __len__
        print('__len__: 42')
        return 42
    def __getattr__(self, attr):     # Provide __str__ if asked, else dummy func
        print('getattr: ' + attr)
        if attr == '__str__':
            return lambda *args: '[Getattr str]'
        else:
            return lambda *args: None

class GetAttribute(object):          # object required in 2.6, implied in 3.0
    eggs = 88                        # In 2.6 all are isinstance(object) auto
    def __init__(self):              # But must derive to get new-style tools,
        self.spam = 77               # incl __getattribute__, some __X__ defaults
    def __len__(self):
        print('__len__: 42')
        return 42
    def __getattribute__(self, attr):
        print('getattribute: ' + attr)
        if attr == '__str__':
            return lambda *args: '[GetAttribute str]'
        else:
            return lambda *args: None

for Class in GetAttr, GetAttribute:
    print('\n' + Class.__name__.ljust(50, '='))

    X = Class()
    X.eggs               # Class attr
    X.spam               # Instance attr
    X.other              # Missing attr
    len(X)               # __len__ defined explicitly

    try:                 # New-styles must support [], +, call directly: redefine
        X[0]             # __getitem__?
    except:
        print('fail []')

    try:
        X + 99           # __add__?
    except:
        print('fail +')

    try:
        X()              # __call__?  (implicit via built-in)
    except:
        print('fail ()')
    X.__call__()         # __call__?  (explicit, not inherited)

    print(X.__str__())   # __str__?   (explicit, inherited from type)
    print(X)             # __str__?   (implicit via built-in)

When run under Python 2.6, __getattr__ does receive a variety of implicit attribute fetches for built-in operations, because Python looks up such attributes in instances normally. Conversely, __getattribute__ is not run for any of the operator overloading names, because such names are looked up in classes only:

C:\misc> c:\python26\python getattr.py

GetAttr===========================================
getattr: other
__len__: 42
getattr: __getitem__
getattr: __coerce__
getattr: __add__
getattr: __call__
getattr: __call__
getattr: __str__
[Getattr str]
getattr: __str__
[Getattr str]

GetAttribute======================================
getattribute: eggs
getattribute: spam
getattribute: other
__len__: 42
fail []
fail +
fail ()
getattribute: __call__
getattribute: __str__
[GetAttribute str]
<__main__.GetAttribute object at 0x025EA1D0>

Note how __getattr__ intercepts both implicit and explicit fetches of __call__ and __str__ in 2.6 here. By contrast, __getattribute__ fails to catch implicit fetches of either attribute name for built-in operations.

Really, the __getattribute__ case is the same in 2.6 as it is in 3.0, because in 2.6 classes must be made new-style by deriving from object to use this method. This code’s object derivation is optional in 3.0 because all classes are new-style.

When run under Python 3.0, though, results for __getattr__ differ—none of the implicitly run operator overloading methods trigger either attribute interception method when their attributes are fetched by built-in operations. Python 3.0 skips the normal instance lookup mechanism when resolving such names:

C:\misc> c:\python30\python getattr.py

GetAttr===========================================
getattr: other
__len__: 42
fail []
fail +
fail ()
getattr: __call__
<__main__.GetAttr object at 0x025D17F0>
<__main__.GetAttr object at 0x025D17F0>

GetAttribute======================================
getattribute: eggs
getattribute: spam
getattribute: other
__len__: 42
fail []
fail +
fail ()
getattribute: __call__
getattribute: __str__
[GetAttribute str]
<__main__.GetAttribute object at 0x025D1870>

We can trace these outputs back to prints in the script to see how this works:

 

 
  • __str__ access fails to be caught twice by __getattr__ in 3.0: once for the built-in print, and once for explicit fetches because a default is inherited from the class (really, from the built-in object, which is a superclass to every class).
  • __str__ fails to be caught only once by the __getattribute__ catchall, during the built-in print operation; explicit fetches bypass the inherited version.
  • __call__ fails to be caught in both schemes in 3.0 for built-in call expressions, but it is intercepted by both when fetched explicitly; unlike with __str__, there is no inherited __call__ default to defeat __getattr__.
  • __len__ is caught by both classes, simply because it is an explicitly defined method in the classes themselves—its name it is not routed to either __getattr__ or __getattribute__ in 3.0 if we delete the class’s __len__ methods.
  • All other built-in operations fail to be intercepted by both schemes in 3.0.

Again, the net effect is that operator overloading methods implicitly run by built-in operations are never routed through either attribute interception method in 3.0: Python 3.0 searches for such attributes in classes and skips instance lookup entirely.

This makes delegation-based wrapper classes more difficult to code in 3.0—if wrapped classes may contain operator overloading methods, those methods must be redefined redundantly in the wrapper class in order to delegate to the wrapped object. In general delegation tools, this can add many extra methods.

Of course, the addition of such methods can be partly automated by tools that augment classes with new methods (the class decorators and metaclasses of the next two chapters might help here). Moreover, a superclass might be able to define all these extra methods once, for inheritance in delegation-based classes. Still, delegation coding patterns require extra work in 3.0.

For a more realistic illustration of this phenomenon as well as its workaround, see the Private decorator example in the following chapter. There, we’ll see that it’s also possible to insert a __getattribute__ in the client class to retain its original type, although this method still won’t be called for operator overloading methods; printing still runs a __str__ defined in such a class directly, for example, instead of routing the request through __getattribute__.

As another example, the next section resurrects our class tutorial example. Now that you understand how attribute interception works, I’ll be able to explain one of its stranger bits.

Note

For an example of this 3.0 change at work in Python itself, see the discussion of the 3.0 os.popen object in Chapter 14. Because it is implemented with a wrapper that uses __getattr__ to delegate attribute fetches to an embedded object, it does not intercept the next(X) built-in iterator function in Python 3.0, which is defined to run __next__. It does, however, intercept and delegate explicit X.__next__() calls, because these are not routed through the built-in and are not inherited from a superclass like __str__ is.

This is equivalent to __call__ in our example—implicit calls for built-ins do not trigger __getattr__, but explicit calls to names not inherited from the class type do. In other words, this change impacts not only our delegators, but also those in the Python standard library! Given the scope of this change, it’s possible that this behavior may evolve in the future, so be sure to verify this issue in later releases.