First-Class Functions

Posted 2021-03-13 neozheng

The "First-Class object" in Python:

• Created at runtime
• Assigned to a variable or element in a data structure
• Pass as an augument to a function
• Return as the result of a function

Integers, strings and dictionaries are other examples of first-class objects in Python.

The __doc__ attribute is used to generate the help text of an object.

 

1. High-Order Functions （高阶函数）

A function that takes a function as argument or returns a function as the result is a high-order function. One example is the built-in function sorted: an optional key argument lets you provide a function to be applied to each item for sorting. And any one-argument function can be used as the key. 

all(iterable): 

# Returns True if every element of the iterables is truthy; all([]) returns True.

any(iterable):

# Returns True if any element of the iterable is truthy; any([]) returns False.

 

2. Anonymous Functions

The best use of anonymous functions is in the context of an argument list. Outside the limited context of arguments to high-order functions, anonymous functions are rarely useful in Python.

The lambda syntax is just syntactic sugar: a lambda expression creates a function object just like the def statement.

 

3. The Seven Flavors of Callable Objects

The call operator(i.e., ()) may be applied to other objects beyond user-defined funcitons. To determine whether an object is callable, use the callable() built-in function.

The Python Data Model documentation lists seven callable types:

3.1 User-defined functions

# Create with def statement or lambda expressions.

3.2 Built-in functions

# A function implemented in C (for CPython), like len or time.strftime

3.3 Built-in methods

# Methods implemented in C, like dict.get

3.4 Methods 

# Functions defined in the body of a class.

3.5 Classes 

# When invoked, a class runs its __new__ method to create an instance, then __init__ to initialize it, and finally the instance is returned to the caller. Because there is no ‘new‘ operator in Python, calling a class is like calling a function.

3.6 Class instances

# If a class defines a __call__ method, then its instances may be invoked as functions. 

3.7 Generator functions

# Functions or methods that use the ‘yield‘ keyword. When called, generator functions return a generator object.

Generator functions can be also used as coroutines.

 

User-Defined Callable Types

import random


class BingoCage:

    def __init__(self, items):
        self._items = list(items)
        random.shuffle(self._items)  # shuffle is guaranteed to work because self._items is a list.

    def pick(self):
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError(‘pick from empty BingoCage‘)

    def __call__(self, *args, **kwargs):
        return self.pick()  # Shortcut to bingo.pick(): bingo()


"""
>>> from bingocall import BingoCage
>>> bingo = BingoCage(range(3))
>>> bingo.pick()
2
>>> bingo()
1
>>> callable(bingo)
True
>>> 
"""

 

4. Function Introspection

Example 5-9. Listing attributes of functions that don‘t exist in plain instances.

In [62]: class C: pass

In [63]: obj = C()

In [64]: def func(): pass

In [66]: sorted(set(dir(func)) - set(dir(obj)))
Out[66]:
[‘__annotations__‘,
 ‘__call__‘,
 ‘__closure__‘,
 ‘__code__‘,
 ‘__defaults__‘,
 ‘__get__‘,
 ‘__globals__‘,
 ‘__kwdefaults__‘,
 ‘__name__‘,
 ‘__qualname__‘]

"""
Using set difference, generate a sorted list of the attributes that exist in a function but not in an instance of a bare class.
"""

 

5. From Positional to Keyword-only Parameters

Example 5-10. tag generates html; a keyword-only argument cls is used to pass "class" attributes as a workaround because class is a keyword in Python.

def tag(name, *content, cls=None, **attrs):
    """Generate one or more HTML tags"""
    if cls is not None:
        attrs[‘class‘] = cls

    if attrs:
        attr_str = ‘‘.join(‘ %s="%s"‘ % (attr, value) for attr, value in sorted(attrs.items()))
    else:
        attr_str = ‘‘

    if content:
        return ‘
‘.join(‘<%s %s>%s</%s>‘ % (name, attr_str, c, name) for c in content)
    else:
        return ‘<%s%s />‘ % (name, attr_str)


# Some of the many ways of calling the tag function.
"""
In [90]: tag(‘br‘)                                                                                                                                            
Out[90]: ‘<br />‘

In [91]: tag(‘p‘, ‘hello‘)                                                                                                                                    
Out[91]: ‘<p >hello</p>‘

In [92]: print(tag(‘p‘, ‘hello‘, ‘world‘))                                                                                                                    
<p >hello</p>
<p >world</p>

In [93]: tag(‘p‘, ‘hello‘, id=3)                                                                                                                              
Out[93]: ‘<p  id="3">hello</p>‘

In [94]: print(tag(‘p‘, ‘hello‘, ‘world‘, cls=‘sidebar‘))                                                                                                     
<p  class="sidebar">hello</p>
<p  class="sidebar">world</p>

In [95]: tag(content=‘testing‘, name=‘img‘)                                                                                                                   
Out[95]: ‘<img content="testing" />‘

In [96]: my_tag = {‘name‘: ‘img‘, ‘title‘: ‘Sunset Boulevard‘, 
    ...:           ‘src‘: ‘sunset.jpg‘, ‘cls‘: ‘framed‘}                                                                                                      

In [97]: tag(**my_tag)  # Prefixing the my_tag dict with ** passes all its items as separate arguments, which are then bound to the named parameters, with the
    ...:  remaining parameters caught by **attrs . 
    ...:                                                                                                                                                      
Out[97]: ‘<img class="framed" src="sunset.jpg" title="Sunset Boulevard" />‘

"""

"""
To specify keyword-only arguments when defining a function, name them after the argument prefixed with * . (keyword-only argument-- 强制关键字参数)
"""

 

6. Retrieving Information About Parameters

Within a function object, the __defaults__ attribute holds a tuple with the default values of positional and keyword arguments. The defaults for keyword-only arguments appear in __kwdefaults__ . The names of the arguments, however, are found within the __code__ attribute, which is a reference to code object with many attributes of its own.

Example 5-15. Function to shorten a string by clipping at a space near the desired length.

def clip(text, max_len=10):
    """Return text clipped at the last space before or after max_len
    """
    end = None
    if len(text) > max_len:
        space_before = text.rfind(‘ ‘, 0, max_len)
        if space_before >= 0:
            end = space_before
        else:
            space_after = text.rfind(‘ ‘, max_len)
            if space_after >= 0:
                end = space_after
    if end is None:  # no spaces were found
        end = len(text)

    return text[:end].rstrip()


"""
# Extracting information about the function arguments:
>>> from clip import clip
>>> clip.__defaults__
(10,)
>>> clip.__code__
<code object clip at 0x102c14390, file ".../ch5-first-class-functions/clip.py", line 1>
>>> clip.__code__.co_varnames
(‘text‘, ‘max_len‘, ‘end‘, ‘space_before‘, ‘space_after‘)
>>> clip.__code__.co_argcount
2
>>> 
"""

"""
Within a function object, the __defaults__  attribute holds a tuple with the default values of positional and keyword
arguments. The default for keyword-only arguments appear in __kwdefaults__ . The names of the arguments, however, are
found within the __code__ attribute, which is reference to a code object with many attributes of its own.
"""

"""
The argument names appear in __code__.co_varnames, but that also includes the names of the local variables created in
the body of the function. Therefore, the argument names are the first N strings, where N is given by 
__code__.co_argcount which --by the way-- does not include any variable arguments prefixed with * or **. The default
values are identified only by their position in the __defaults__ tuple, so to link each with the respective argument,
you have to scan from last to first.  
In the example, we have two arguments, text and max_len, and one default, 80, so it must belong to the last argument,
max_len.
"""

 

Example 5-17. Extracting the function signature.

>>> from clip import clip
>>> from inspect import signature
>>> sig = signature(clip)
>>> sig
<Signature (text, max_len=10)>
>>> str(sig)
‘(text, max_len=10)‘

>>> for name, param in sig.parameters.items():
...     print(param.kind, ‘:‘, name, ‘=‘, param.default)
...
POSITIONAL_OR_KEYWORD : text = <class ‘inspect._empty‘>
POSITIONAL_OR_KEYWORD : max_len = 10


"""
inspect.signature returns an inspect.Signature object, which has a parameters attribute that lets you read an ordered
mapping of names to inspect.Parameter objects. Each Parameter instance has attributes such as name, default and kind.
The special value inspect._empty denotes parameters with no default.
"""

 

Example 5-18. Binding the function signature from the tag function in Example 5-10 to a dict of argument.

"""Binding the function signature from the tag function to a dict of arguments"""

"""
The inspect.Signature object hjas a bind method that takes any number of arguments and binds them to the parameters in
the signature, applying the usual rules for matching actual arguments to formal parameters. This can be used by a 
framework to validate arguments prior to the actual function invocation.
"""

>>> import inspect
>>> from html_tags import tag
>>> sig = inspect.signature(tag)
>>> my_tag = {‘name‘:‘img‘, ‘title‘:‘Sunset Boulevard‘, ‘src‘:‘sunset.jpg‘, ‘cls‘:‘framed‘}
>>> bound_args = sig.bind(**my_tag)         # Pass a dict of arguments to .bind()
>>> bound_args
<BoundArguments (name=‘img‘, cls=‘framed‘, attrs={‘title‘: ‘Sunset Boulevard‘, ‘src‘: ‘sunset.jpg‘})>
>>>
>>> for name, value in bound_args.arguments.items():        # bound_args.argument is an OrderDict, which display the names and values of the arguments.
...     print(name, ‘=‘, value)
...
name = img
cls = framed
attrs = {‘title‘: ‘Sunset Boulevard‘, ‘src‘: ‘sunset.jpg‘}
>>>
>>> del my_tag[‘name‘]
>>> bound_args = sig.bind(**my_tag)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2965, in bind
    return args[0]._bind(args[1:], kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/inspect.py", line 2880, in _bind
    raise TypeError(msg) from None
TypeError: missing a required argument: ‘name‘

 

7. Function Annotations

Python 3 provides syntax to attach metadata to the parameters of a function declaration and its return value.

Example 5-19. Annotated clip function.

def clip(text: str, max_len: ‘int > 0‘ = 10) -> str:        # The Annotated function declaration
    """Return text clipped at the last space before or after max_len
    """
    end = None
    if len(text) > max_len:
        space_before = text.rfind(‘ ‘, 0, max_len)
        if space_before >= 0:
            end = space_before
        else:
            space_after = text.rfind(‘ ‘, max_len)
            if space_after >= 0:
                end = space_after
    if end is None:  # no spaces were found
        end = len(text)

    return text[:end].rstrip()


"""
Each argument in the function declaration may have an annotation expression preceded by : . If there is a default value, 
the annotation goes between the argument name and the = sign. To annotate the return value, add -> and another expression
between the ) and the : at the tail of the function declaration. The expressions may be of any type. The most common 
types used in annotations are classes, like str or int, or strings, like ‘int > 0‘, as seen in the annotation for max_len
in this example.

No processing is done with the annotations. They are merely stored in __annotations__ attribute of the function, a dict:
"""

# 运行示例结果：
"""
>>> clip.__annotations__
{‘text‘: <class ‘str‘>, ‘max_len‘: ‘int > 0‘, ‘return‘: <class ‘str‘>}
>>> 
"""

"""
The item with key ‘return‘ holds the return value annotation marked with -> in the function declaration in this example.
"""

"""
The only thing Python does with annotations is to store them in the __annotations__ attribute of the function. Nothing
else: no checks, enforcement, validation, or any other action is performed. In other words, annotations have no meaning
to the Python interpreter. They are just metadata that may be used by tools,such as IDEs, framework, and decorators.
"""

 

Example 5-20. Extracting annotations from the function signature.

>>> from clip_annot import clip
>>> from inspect import signature
>>> sig = signature(clip)
>>> sig.return_annotation
<class ‘str‘>
>>> for param in sig.parameters.values():
...     note = repr(param.annotation).ljust(13)
...     print(note, ‘:‘, param.name, ‘=‘, param.default)
...
<class ‘str‘> : text = <class ‘inspect._empty‘>
‘int > 0‘     : max_len = 10
>>>

"""
The signature function returns a Signature object, which has a return_annotation attribute and a parameters dictionary
mapping parameter names to Parameter objects. Each Parameter object has its own annotation attribute.
"""

 

8. The operator Module

Example 5-21. Factorial implemented with reduce and an anonymous function.

from functools import reduce
from operator import mul


def fact(n):
    return reduce(mul, range(n, n + 1))


"""
The operator module provides function equivalents for dozens of arithmetic operators.
"""

 

itemgetter , attrgetter , methodcaller:

Example 5-23. Demo of itemgetter to sort a list of tuples.

"""
Another group of one-trick lambdas that operator replaces are functions to pick items from sequences or read attributes
from objects: itemgetter and attrgetter actually build custom functions to do that.
"""

metro_data = [
    (‘Tokyo‘, ‘JP‘, 36.933, (35.689722, 139.691667)),
    (‘Delhi NCP‘, ‘IN‘, 21.935, (28.613889, 77.208889)),
    (‘Mexico City‘, ‘MX‘, 20.142, (14.433333, -99.133333)),
    (‘New York-Newark‘, ‘US‘, 20.104, (40.808611, -74.020386)),
    (‘Sau Paulo‘, ‘BR‘, 19.649, (-23.547778, -46.635833)),
]

from operator import itemgetter
for city in sorted(metro_data, key=itemgetter(1)):
    print(city)


# 运行结果：
"""
(‘Sau Paulo‘, ‘BR‘, 19.649, (-23.547778, -46.635833))
(‘Delhi NCP‘, ‘IN‘, 21.935, (28.613889, 77.208889))
(‘Tokyo‘, ‘JP‘, 36.933, (35.689722, 139.691667))
(‘Mexico City‘, ‘MX‘, 20.142, (14.433333, -99.133333))
(‘New York-Newark‘, ‘US‘, 20.104, (40.808611, -74.020386))
"""

"""
This example show a common use of itemgetter: sorting a list of tuples by the value of one field. In this example, the
cities are printed sorted by country code (field 1). Essentially, itemgetter(1) does the same as lambda fields:fields[1]
: create a function that returns the item at index 1.
"""

"""
If you pass multiple index arguments to itemgetter, the function it builds will return tuples with the extracted values:
"""

cc_name = itemgetter(1, 0)
for city in metro_data:
    print(cc_name(city))

# 运行结果：
"""
(‘JP‘, ‘Tokyo‘)
(‘IN‘, ‘Delhi NCP‘)
(‘MX‘, ‘Mexico City‘)
(‘US‘, ‘New York-Newark‘)
(‘BR‘, ‘Sau Paulo‘)
"""

"""
Because itemgetter uses the [] operator, it supports not only sequences but also mapping and any class that implements
__getitem__ .
"""

 

Example 5-24. Demo of attrgetter to process a previously defined list of nametuple called metro_data.

"""
A sibling of itemgetter ios attrgetter, which creates functions to extract object attributes by name. If you pass
attrgetter several attribute names as arguments, it also returns a tuple of values. In addition, if any argument name
contains a . (dot), attrgetter navigates through nested objects to retrieve the attribute.
"""


>>> from collections import namedtuple
>>>
>>> metro_data = [
...     (‘Tokyo‘, ‘JP‘, 36.933, (35.689722, 139.691667)),
...     (‘Delhi NCP‘, ‘IN‘, 21.935, (28.613889, 77.208889)),
...     (‘Mexico City‘, ‘MX‘, 20.142, (14.433333, -99.133333)),
...     (‘New York-Newark‘, ‘US‘, 20.104, (40.808611, -74.020386)),
...     (‘Sau Paulo‘, ‘BR‘, 19.649, (-23.547778, -46.635833)),
... ]
>>>
>>> LatLong = namedtuple(‘LatLong‘, ‘lat long‘)
>>> Metropolis = namedtuple(‘Metropolis‘, ‘name cc pop coord‘)
>>>
>>> metro_areas = [
...     Metropolis(name, cc, pop, LatLong(lat, long))
...     for name, cc, pop, (lat, long) in metro_data
...     # nested tuple unpacking to extract (lat, long) and use them to build the LatLong for
...     # the coord attribute of Metropolis
... ]
>>>
>>> metro_areas[0].coord.lat
35.689722
>>>
>>> from operator import attrgetter
>>> name_lat = attrgetter(‘name‘, ‘coord.lat‘)      # Define an attrgetter to retrieve the name and the coord.lat nested attribute.
>>>
>>> for city in sorted(metro_areas, key=attrgetter(‘coord.lat‘)):   # Use attrgetter to sort list of cities by latitude.
...     print(name_lat(city))       # use the attrgetter defined to show only city name and latitude.
...
(‘Sau Paulo‘, -23.547778)
(‘Mexico City‘, 14.433333)
(‘Delhi NCP‘, 28.613889)
(‘Tokyo‘, 35.689722)
(‘New York-Newark‘, 40.808611)
>>>

 

Example 5-25. Demo of methodcaller: second test shows the binding of extra arguments.

>>> from operator import methodcaller
>>>
>>> s = ‘The time has come‘
>>> upcase = methodcaller(‘upper‘)
>>> upcase(s)
‘THE TIME HAS COME‘
>>>
>>> hiphenate = methodcaller(‘replace‘, ‘ ‘, ‘-‘)      # the second test shows the binding of extra arguments.
>>> hiphenate(s)
‘The-time-has-come‘
>>>

"""
The function methodcaller creates calls a method by name on the object given as argument.
methodcaller 创建的函数能够通过名字调用所传实参对应对象的一个方法
"""

 

9. Freezing Arguments with functools.partial

Example 5-26. Using partial to use a two-argument function where a one-argument callable is required.

from operator import mul
from functools import partial

triple = partial(mul, 3)    # create new triple function from mul, binding first positional arguments to 3.
triple(7)

list(map(triple, range(1, 10)))

"""
The function module brings together a handful of high-order functions. One of the most useful is partial and its
variation, partialmethod.

functools.partial is a high-order function that allows partial application of a function. Given a function, a partial
application produces a new callable with some of the arguments of the original function fixed. This is useful to adapt
a function that takes one or more arguments to an API that requires a callback with fewer arguments.
"""

 

Example 5-27. Building a convenient Unicode normalizing function with partial.

"""Building a convenient Unicode normalizing function with partial"""

>>> import unicodedata, functools
>>>
>>> nfc = functools.partial(unicodedata.normalize, ‘NFC‘)
>>> s1 = ‘café‘
>>> s2 = ‘cafeu0301‘
>>> s1, s2
(‘café‘, ‘cafe?‘)
>>> s1 == s2
False
>>> nfc(s1) == nfc(s2)
True
>>>

 

Example 5-28. Demo of partial appplied to the function tag from the previous example.

"""
This example shows the use of partial with tag function in previous example, to freeze one positional argument and
one keyword argument.
"""

>>> from html_tags import tag
>>> tag
<function tag at 0x102ed3268>
>>> from functools import partial
>>> picture = partial(tag, ‘img‘, cls=‘pic-frame‘)      # Create picture function from tag by fixing the first positional argument with ‘img‘ and the cls keyword argument with ‘pic-frame‘
>>> picture(src=‘wumpus.jpeg‘)
‘<img class="pic-frame" src="wumpus.jpeg" />‘
>>> picture     # partial() returns a functools.partial object.
functools.partial(<function tag at 0x102ed3268>, ‘img‘, cls=‘pic-frame‘)
>>> picture.func        # A functools.partial object has attributes providing access to the original function and the fixed arguments.
<function tag at 0x102ed3268>
>>> picture.args
(‘img‘,)
>>> picture.keywords
{‘cls‘: ‘pic-frame‘}
>>>


"""
partial takes a callable as first argument, followed by an arbitrary number of positional and keyword arguments to bind.
"""

"""
The functools.partialmethod function does the same job as partial, but is designed to work with methods.
"""

 

Chapter Summary

The goal of this chapter was to explore the first-class nature of function in Python. The main ideas are that you can assign functions to variables, pass them to other functions, store them in data structures, and access attributes, allowing frameworks and tools to act on that information.