Generators

Generators provide an elegant way to write simple and efficient code for functions that return a sequence of elements. Based on the yield statement, they allow you to pause a function and return an intermediate result. The function saves its execution context and can be resumed later, if necessary.

1. Creating generators

Generators were introduced in Python a long time ago, with the idea of introducing iteration in Python while improving the performance of the program (by using less memory) at the same time.

The idea of a generator is to create an object that is iterable, and, while it’s being iterated, will produce the elements it contains, one at a time. The main use of generators is to save memory: instead of having a very large list of elements in memory, holding everything at once, we have an object that knows how to produce each particular element, one at a time, as they are required.

In many cases, the resources required to process one element are less than the resources required to store whole sequences. Therefore, they can be kept low, making the program more efficient.

This feature enables lazy computations or heavyweight objects in memory, in a similar manner to what other functional programming languages (Haskell, for instance) provide. It would even be possible to work with infinite sequences because the lazy nature of generators allows for such an option.

A common use case is to stream data buffers with generators (for example, from files). They can be paused, resumed, and stopped whenever necessary at any stage of the data processing pipeline without any need to load whole datasets into the program’s memory

1.1. A first look at generators

Let’s start with an example. The problem at hand now is that we want to process a large list of records and get some metrics and indicators over them. Given a large data set with information about purchases, we want to process it in order to get the lowest sale, highest sale, and the average price of a sale.

For the simplicity of this example, we will assume a CSV with only two fields, in the following format:

<purchase_date>, <price>
...

We are going to create an object that receives all the purchases, and this will give us the necessary metrics. We could get some of these values out of the box by simply using the min() and max() built-in functions, but that would require iterating all of the purchases more than once, so instead, we are using our custom object, which will get these values in a single iteration.

The code that will get the numbers for us looks rather simple. It’s just an object with a method that will process all prices in one go, and, at each step, will update the value of each particular metric we are interested in. First, we will show the first implementation in the following listing, and, later on (once we have seen more about iteration), we will revisit this implementation and get a much better (and compact) version of it. For now, we are settling on the following:

class PurchasesStats:
    def __init__(self, purchases):
        self.purchases = iter(purchases)
        self.min_price: float = None
        self.max_price: float = None
        self._total_purchases_price: float = 0.0
        self._total_purchases = 0
        self._initialize()

    def _initialize(self):
        try:
            first_value = next(self.purchases)
        except StopIteration:
            raise ValueError("no values provided")

        self.min_price = self.max_price = first_value
        self._update_avg(first_value)

    def process(self):
        for purchase_value in self.purchases:
            self._update_min(purchase_value)
            self._update_max(purchase_value)
            self._update_avg(purchase_value)

        return self

    def _update_min(self, new_value: float):
        if new_value < self.min_price:
            self.min_price = new_value

    def _update_max(self, new_value: float):
        if new_value > self.max_price:
            self.max_price = new_value

    @property
    def avg_price(self):
        return self._total_purchases_price / self._total_purchases

    def _update_avg(self, new_value: float):
        self._total_purchases_price += new_value
        self._total_purchases += 1

    def __str__(self):
        return (
            f"{self.__class__.__name__}({self.min_price}, "
            f"{self.max_price}, {self.avg_price})"
        )

This object will receive all the totals for the purchases and process the required values. Now, we need a function that loads these numbers into something that this object can process. Here is the first version:

def _load_purchases(filename):
    purchases = []
    with open(filename) as f:
        for line in f:
            *_, price_raw = line.partition(",")
            purchases.append(float(price_raw))

    return purchases

This code works; it loads all the numbers of the file into a list that, when passed to our custom object, will produce the numbers we want. It has a performance issue, though. If you run it with a rather large dataset, it will take a while to complete, and it might even fail if the dataset is large enough as to not fit into the main memory.

If we take a look at our code that consumes this data, it is processing the purchases, one at a time, so we might be wondering why our producer fits everything in memory at once. It is creating a list where it puts all of the content of the file, but we know we can do better.

The solution is to create a generator. Instead of loading the entire content of the file in a list, we will produce the results one at a time. The code will now look like this:

def load_purchases(filename):
    with open(filename) as f:
        for line in f:
            *_, price_raw = line.partition(",")
            yield float(price_raw)

If you measure the process this time, you will notice that the usage of memory has dropped significantly. We can also see how the code looks simpler: there is no need to define the list (therefore, there is no need to append to it), and that the return statement also disappeared.

In this case, the load_purchases function is a generator function, or simply a generator.

In Python, the mere presence of the keyword yield in any function makes it a generator, and, as a result, when calling it, nothing other than creating an instance of the generator will happen:

>>> load_purchases("file")
<generator object load_purchases at 0x...>

A generator object is an iterable (we will revisit iterables in more detail later on), which means that it can work with for loops. Notice how we did not have to change anything on the consumer code: our statistics processor remained the same, with the for loop unmodified, after the new implementation.

Working with iterables allows us to create these kinds of powerful abstractions that are polymorphic with respect to for loops. As long as we keep the iterable interface, we can iterate over that object transparently.

1.2. Generator expressions

Generators save a lot of memory, and since they are iterators, they are a convenient alternative to other iterables or containers that require more space in memory such as lists, tuples, or sets.

Much like these data structures, they can also be defined by comprehension, only that it is called a generator expression (there is an ongoing argument about whether they should be called generator comprehensions).

In the same way, we would define a list comprehension. If we replace the square brackets with parenthesis, we get a generator that results from the expression. Generator expressions can also be passed directly to functions that work with iterables, such as sum(), and, max():

>>> [x**2 for x in range(10)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>>> (x**2 for x in range(10))
<generator object <genexpr> at 0x...>
>>> sum(x**2 for x in range(10))
285

Note

Always pass a generator expression, instead of a list comprehension, to functions that expect iterables, such as min(), max(), and sum(). This is more efficient and pythonic.

It is also worth mentioning, that we can only iterate 1 time over generators:

>>> a = (x for x in range(3))
>>> a
<generator object <genexpr> at 0x7f95ece4dad0>
>>> for x in a:
...     print(x)
...
0
1
2

>>> next(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Tip

It is better to have a lot of simple iterable functions that work over sequences of values than a complex function that computes the result for one value at a time.

2. Iterating idiomatically

In this section, we will first explore some idioms that come in handy when we have to deal with iteration in Python. These code recipes will help us get a better idea of the types of things we can do with generators (especially after we have already seen generator expressions), and how to solve typical problems in relation to them.

Once we have seen some idioms, we will move on to exploring iteration in Python in more depth, analyzing the methods that make iteration possible, and how iterable objects work.

2.1. Idioms for iteration

We are already familiar with the built-in enumerate() function that, given an iterable, will return another one on which the element is a tuple, whose first element is the enumeration of the second one (corresponding to the element in the original iterable):

>>> list(enumerate("abcdef"))
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]

We wish to create a similar object, but in a more low-level fashion; one that can simply create an infinite sequence. We want an object that can produce a sequence of numbers, from a starting one, without any limits.

An object as simple as the following one can do the trick. Every time we call this object, we get the next number of the sequence ad infinitum:

class NumberSequence:
    def __init__(self, start=0):
        self.current = start

    def next(self):
        current = self.current
        self.current += 1
        return current

Based on this interface, we would have to use this object by explicitly invoking its next() method:

>>> seq = NumberSequence()
>>> seq.next()
0
>>> seq.next()
1
>>> seq2 = NumberSequence(10)
>>> seq2.next()
10
>>> seq2.next()
11

But with this code, we cannot reconstruct the enumerate() function as we would like to, because its interface does not support being iterated over a regular Python for loop, which also means that we cannot pass it as a parameter to functions that expect something to iterate over. Notice how the following code fails:

>>> list(zip(NumberSequence(), "abcdef"))
Traceback (most recent call last):
File "...", line 1, in <module>
TypeError: zip argument #1 must support iteration

The problem lies in the fact that NumberSequence does not support iteration. To fix this, we have to make the object an iterable by implementing the magic method __iter__(). We have also changed the previous next() method, by using the magic method __next__, which makes the object an iterator:

class SequenceOfNumbers:
    def __init__(self, start=0):
        self.current = start

    def __next__(self):
        current = self.current
        self.current += 1
        return current

    def __iter__(self):
        return self

This has an advantage: not only can we iterate over the element, we also don’t even need the next() method any more because having __next__() allows us to use the next() built-in function:

>>> list(zip(SequenceOfNumbers(), "abcdef"))
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]
>>> seq = SequenceOfNumbers(100)
>>> next(seq)
100
>>> next(seq)
101

2.1.1. The next() function

The next() built-in function will advance the iterable to its next element and return it:

>>> word = iter("hello")
>>> next(word)
'h'
>>> next(word)
'e'

If the iterator does not have more elements to produce, the StopIteration exception is raised:

>>> ...
>>> next(word)
'o'
>>> next(word)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

This exception signals that the iteration is over and that there are no more elements to consume.

If we wish to handle this case, besides catching the StopIteration exception, we could provide this function with a default value in its second parameter. Should this be provided, it will be the return value in lieu of throwing StopIteration:

>>> next(word, "default value")
'default value'

2.1.2. Using a generator

The previous code can be simplified significantly by simply using a generator. Generator objects are iterators. This way, instead of creating a class, we can define a function that yield the values as needed:

def sequence(start=0):
    while True:
        yield start
        start += 1

Remember that from our first definition, the yield keyword in the body of the function makes it a generator. Because it is a generator, it’s perfectly fine to create an infinite loop like this, because, when this generator function is called, it will run all the code until the next yield statement is reached. It will produce its value and suspend there:

>>> seq = sequence(10)
>>> next(seq)
10
>>> next(seq)
11
>>> list(zip(sequence(), "abcdef"))
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]

2.1.3. Itertools

Working with iterables has the advantage that the code blends better with Python itself because iteration is a key component of the language. Besides that, we can take full advantage of the itertools module. Actually, the sequence() generator we just created is fairly similar to itertools.count(). However, there is more we can do.

One of the nicest things about iterators, generators, and itertools, is that they are composable objects that can be chained together.

For instance, getting back to our first example that processed purchases in order to get some metrics, what if we want to do the same, but only for those values over a certain threshold? The naive approach of solving this problem would be to place the condition while iterating:

def process(self):
    for purchase in self.purchases:
        if purchase > 1000.0:
        ...

This is not only non-Pythonic, but it’s also rigid (and rigidity is a trait that denotes bad code). It doesn’t handle changes very well. What if the number changes now? Do we pass it by parameter? What if we need more than one? What if the condition is different (less than, for instance)? Do we pass a lambda?

These questions should not be answered by this object, whose sole responsibility is to compute a set of well-defined metrics over a stream of purchases represented as numbers. And, of course, the answer is no. It would be a huge mistake to make such a change (once again, clean code is flexible, and we don’t want to make it rigid by coupling this object to external factors). These requirements will have to be addressed elsewhere.

It’s better to keep this object independent of its clients. The less responsibility this class has, the more useful it will be for more clients, hence enhancing its chances of being reused.

Instead of changing this code, we’re going to keep it as it is and assume that the new data is filtered according to whatever requirements each customer of the class has.

For instance, if we wanted to process only the first 10 purchases that amount to more than 1,000, we would do the following:

>>> from itertools import islice
>>> purchases = islice(filter(lambda p: p > 1000.0, purchases), 10)
>>> stats = PurchasesStats(purchases).process()

There is no memory penalization for filtering this way because since they all are generators, the evaluation is always lazy. This gives us the power of thinking as if we had filtered the entire set at once and then passed it to the object, but without actually fitting everything in memory.

2.1.4. Simplifying code through iterators

Now, we will briefly discuss some situations that can be improved with the help of iterators, and occasionally the itertools module. After discussing each case, and its proposed optimization, we will close each point with a corollary.

2.1.4.1. Repeated iterations

Now that we have seen more about iterators, and introduced the itertools module, we can show you how one of the first examples of this chapter (the one for computing statistics about some purchases), can be dramatically simplified:

def process_purchases(purchases):
    min_iter, max_iter, avg_iter = itertools.tee(purchases, 3)
    return min(min_iter), max(max_iter), median(avg_iter)

In this example, itertools.tee will split the original iterable into three new ones. We will use each of these for the different kinds of iterations that we require, without needing to repeat three different loops over purchases.

The reader can simply verify that if we pass an iterable object as the purchases parameter, this one is traversed only once (thanks to the itertools.tee function), which was our main requirement. It is also possible to verify how this version is equivalent to our original implementation. In this case, there is no need to manually raise ValueError because passing an empty sequence to the min() function will do the same.

Note

If you are thinking about running a loop over the same object more than one time, stop and think if itertools.tee can be of any help.

2.1.4.2. Nested loops

In some situations, we need to iterate over more than one dimension, looking for a value, and nested loops come as the first idea. When the value is found, we need to stop iterating, but the break keyword doesn’t work entirely because we have to escape from two (or more) for loops, not just one.

What would be the solution for this? A flag signaling escape? No. Raising an exception? No, this would be the same as the flag, but even worse because we know that exceptions are not to be used for control flow logic. Moving the code to a smaller function and return it? Close, but not quite.

The answer is, whenever possible, flat the iteration to a single for loop. This is the kind of code we would like to avoid:

def search_nested_bad(array, desired_value):
    coords = None
    for i, row in enumerate(array):
        for j, cell in enumerate(row):
            if cell == desired_value:
                coords = (i, j)
                break
        if coords is not None:
            break

    if coords is None:
        raise ValueError(f"{desired_value} not found")

    logger.info("value %r found at [%i, %i]", desired_value, *coords)
    return coords

And here is a simplified version of it that does not rely on flags to signal termination, and has a simpler, more compact structure of iteration:

def _iterate_array2d(array2d):
    for i, row in enumerate(array2d):
        for j, cell in enumerate(row):
            yield (i, j), cell

def search_nested(array, desired_value):
    try:
        coord = next(coord for (coord, cell) in _iterate_array2d(array) if cell == desired_value)
    except StopIteration:
        raise ValueError(f"{desired_value} not found")

    logger.info(f"value {desired_value} found at {coords}")
    return coord

It’s worth mentioning how the auxiliary generator that was created works as an abstraction for the iteration that’s required. In this case, we just need to iterate over two dimensions, but if we needed more, a different object could handle this without the client needing to know about it. This is the essence of the iterator design pattern, which, in Python, is transparent, since it supports iterator objects automatically, which is the topic covered in the next section.

Note

Try to simplify the iteration as much as possible with as many abstractions as are required, flatting the loops whenever possible.

2.2. The iterator pattern in Python

Here, we will take a small detour from generators to understand iteration in Python more deeply. Generators are a particular case of iterable objects, but iteration in Python goes beyond generators, and being able to create good iterable objects will give us the chance to create more efficient, compact, and readable code.

In the previous code listings, we have been seeing examples of iterable objects that are also iterators, because they implement both the __iter__() and __next__() magic methods. While this is fine in general, it’s not strictly required that they always have to implement both methods, and here we’ll show the subtle differences between an iterable object (one that implements __iter__) and an iterator (that implements __next__).

We also explore other topics related to iterations, such as sequences and container objects.

2.2.1. The interface for iteration

An iterable is an object that supports iteration, which, at a very high level, means that we can run a for .. in ... loop over it, and it will work without any issues. However, iterable does not mean the same as iterator.

Generally speaking, an iterable is just something we can iterate, and it uses an iterator to do so. This means that in the __iter__ magic method, we would like to return an iterator, namely, an object with a __next__() method implemented.

An iterator is an object that only knows how to produce a series of values, one at a time, when it’s being called by the already explored built-in next() function. While the iterator is not called, it’s simply frozen, sitting idly by until it’s called again for the next value to produce. In this sense, generators are iterators.

In the following code, we will see an example of an iterator object that is not iterable: it only supports invoking its values, one at a time. Here, the name sequence refers just to a series of consecutive numbers, not to the sequence concept in Python, which will we explore later on:

class SequenceIterator:
    def __init__(self, start=0, step=1):
        self.current = start
        self.step = step

    def __next__(self):
        value = self.current
        self.current += self.step
        return value

Notice that we can get the values of the sequence one at a time, but we can’t iterate over this object (this is fortunate because it would otherwise result in an endless loop):

>>> si = SequenceIterator(1, 2)
>>> next(si)
1
>>> next(si)
3
>>> next(si)
5
>>> for _ in SequenceIterator(): pass
...
Traceback (most recent call last):
...
TypeError: 'SequenceIterator' object is not iterable

The error message is clear, as the object doesn’t implement __iter__().

Just for explanatory purposes, we can separate the iteration in another object (again, it would be enough to make the object implement both __iter__ and __next__, but doing so separately will help clarify the distinctive point we’re trying to make in this explanation).

2.2.2. Sequence objects as iterables

As we have just seen, if an object implements the __iter__() magic method, it means it can be used in a for loop. While this is a great feature, it’s not the only possible form of iteration we can achieve. When we write a for loop, Python will try to see if the object we’re using implements __iter__, and, if it does, it will use that to construct the iteration, but if it doesn’t, there are fallback options.

If the object happens to be a sequence (meaning that it implements __getitem__() and __len__() magic methods), it can also be iterated. If that is the case, the interpreter will then provide values in sequence, until the IndexError exception is raised, which, analogous to the aforementioned StopIteration, also signals the stop for the iteration.

With the sole purpose of illustrating such a behavior, we run the following experiment that shows a sequence object that implements map() over a range of numbers:

class MappedRange:
    """Apply a transformation to a range of numbers."""
    def __init__(self, transformation, start, end):
        self._transformation = transformation
        self._wrapped = range(start, end)

    def __getitem__(self, index):
        value = self._wrapped.__getitem__(index)
        result = self._transformation(value)
        logger.info(f"Index {index}: {result}")
        return result

    def __len__(self):
        return len(self._wrapped)

Keep in mind that this example is only designed to illustrate that an object such as this one can be iterated with a regular for loop. There is a logging line placed in the __getitem__ method to explore what values are passed while the object is being iterated, as we can see from the following test:

>>> mr = MappedRange(abs, -10, 5)
>>> mr[0]
Index 0: 10
10
>>> mr[-1]
Index -1: 4
4
>>> list(mr)
Index 0: 10
Index 1: 9
Index 2: 8
Index 3: 7
Index 4: 6
Index 5: 5
Index 6: 4
Index 7: 3
Index 8: 2
Index 9: 1
Index 10: 0
Index 11: 1
Index 12: 2
Index 13: 3
Index 14: 4
[10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4]

As a word of caution, it’s important to highlight that while it is useful to know this, it’s also a fallback mechanism for when the object doesn’t implement __iter__, so most of the time we’ll want to resort to these methods by thinking in creating proper sequences, and not just objects we want to iterate over.

Note

When thinking about designing an object for iteration, favor a proper iterable object (with __iter__), rather than a sequence that can coincidentally also be iterated.

3. Coroutines

As we already know, generator objects are iterables. They implement __iter__() and __next__(). This is provided by Python automatically so that when we create a generator object function, we get an object that can be iterated or advanced through the next() function.

Besides this basic functionality, they have more methods so that they can work as coroutines. Here, we will explore how generators evolved into coroutines to support the basis of asynchronous programming before we go into more detail in the next section, where we explore the new features of Python and the syntax that covers programming asynchronously. The basic methods added to support coroutines are as follows:

• .close()

• .throw(ex_type[, ex_value[, ex_traceback]])

• .send(value)

3.1. The methods of the generator interface

In this section, we will explore what each of the aforementioned methods does, how it works, and how it is expected to be used. By understanding how to use these methods, we will be able to make use of simple coroutines.

Later on, we will explore more advanced uses of coroutines, and how to delegate to sub- generators (coroutines) in order to refactor code, and how to orchestrate different coroutines.

3.1.1. close()

When calling this method, the generator will receive the GeneratorExit exception. If it’s not handled, then the generator will finish without producing any more values, and its iteration will stop.

This exception can be used to handle a finishing status. In general, if our coroutine does some sort of resource management, we want to catch this exception and use that control block to release all resources being held by the coroutine. In general, it is similar to using a context manager or placing the code in the finally block of an exception control, but handling this exception specifically makes it more explicit.

In the following example, we have a coroutine that makes use of a database handler object that holds a connection to a database, and runs queries over it, streaming data by pages of a fixed length (instead of reading everything that is available at once):

def stream_db_records(db_handler):
    try:
        while True:
            yield db_handler.read_n_records(10)
    except GeneratorExit:
        db_handler.close()

At each call to the generator, it will return 10 rows obtained from the database handler, but when we decide to explicitly finish the iteration and call close(), we also want to close the connection to the database:

>>> streamer = stream_db_records(DBHandler("testdb"))
>>> next(streamer)
[(0, 'row 0'), (1, 'row 1'), (2, 'row 2'), (3, 'row 3'), ...]
>>> next(streamer)
[(0, 'row 0'), (1, 'row 1'), (2, 'row 2'), (3, 'row 3'), ...]
>>> streamer.close()
INFO:...:closing connection to database 'testdb'

Use the close() method on generators to perform finishing-up tasks when needed.

3.1.2. throw(ex_type[, ex_value[, ex_traceback]])

This method will throw the exception at the line where the generator is currently suspended. If the generator handles the exception that was sent, the code in that particular except clause will be called, otherwise, the exception will propagate to the caller.

Here, we are modifying the previous example slightly to show the difference when we use this method for an exception that is handled by the coroutine, and when it’s not:

class CustomException(Exception):
    pass

def stream_data(db_handler):
    while True:
        try:
            yield db_handler.read_n_records(10)
        except CustomException as e:
            logger.info(f"controlled error {e}, continuing")
        except Exception as e:
            logger.info(f"unhandled error {e}, stopping")
            db_handler.close()
        break

Now, it is a part of the control flow to receive a CustomException, and, in such a case, the generator will log an informative message (of course, we can adapt this according to our business logic on each case), and move on to the next yield statement, which is the line where the coroutine reads from the database and returns that data.

This particular example handles all exceptions, but if the last block (except Exception:) wasn’t there, the result would be that the generator is raised at the line where the generator is paused (again, the yield), and it will propagate from there to the caller:

>>> streamer = stream_data(DBHandler("testdb"))
>>> next(streamer)
[(0, 'row 0'), (1, 'row 1'), (2, 'row 2'), (3, 'row 3'), (4, 'row 4'), ...]
>>> next(streamer)
[(0, 'row 0'), (1, 'row 1'), (2, 'row 2'), (3, 'row 3'), (4, 'row 4'), ...]
>>> streamer.throw(CustomException)
WARNING:controlled error CustomException(), continuing
[(0, 'row 0'), (1, 'row 1'), (2, 'row 2'), (3, 'row 3'), (4, 'row 4'), ...]
>>> streamer.throw(RuntimeError)
ERROR:unhandled error RuntimeError(), stopping
INFO:closing connection to database 'testdb'
Traceback (most recent call last):
...
StopIteration

When our exception from the domain was received, the generator continued. However, when it received another exception that was not expected, the default block caught where we closed the connection to the database and finished the iteration, which resulted in the generator being stopped. As we can see from the StopIteration that was raised, this generator can’t be iterated further.

3.1.3. send(value)

In the previous example, we created a simple generator that reads rows from a database, and when we wished to finish its iteration, this generator released the resources linked to the database. This is a good example of using one of the methods that generators provide (close), but there is more we can do.

An obvious of such a generator is that it was reading a fixed number of rows from the database.

We would like to parametrize that number so that we can change it throughout different calls. Unfortunately, the next() function does not provide us with options for that. But luckily, we have send():

def stream_db_records(db_handler):
    retrieved_data = None
    previous_page_size = 10

    try:
        while True:
            page_size = yield retrieved_data
            if page_size is None:
                page_size = previous_page_size

            previous_page_size = page_size
            retrieved_data = db_handler.read_n_records(page_size)

    except GeneratorExit:
        db_handler.close()

The idea is that we have now made the coroutine able to receive values from the caller by means of the send() method. This method is the one that actually distinguishes a generator from a coroutine because when it’s used, it means that the yield keyword will appear on the right-hand side of the statement, and its return value will be assigned to something else.

In coroutines, we generally find the yield keyword to be used in the following form: receive = yield produced

The yield, in this case, will do two things. It will send produced back to the caller, which will pick it up on the next round of iteration (after calling next(), for example), and it will suspend there. At a later point, the caller will want to send a value back to the coroutine by using the send() method. This value will become the result of the yield statement, assigned in this case to the variable named receive.

Sending values to the coroutine only works when this one is suspended at a yield statement, waiting for something to produce. For this to happen, the coroutine will have to be advanced to that status. The only way to do this is by calling next() on it. This means that before sending anything to the coroutine, this has to be advanced at least once via the next() method. Failure to do so will result in an exception:

>>> c = coro()
>>> c.send(1)
Traceback (most recent call last):
...
TypeError: can't send non-None value to a just-started generator

Important

Always remember to advance a coroutine by calling next() before sending any values to it.

Back to our example. We are changing the way elements are produced or streamed to make it able to receive the length of the records it expects to read from the database.

The first time we call next(), the generator will advance up to the line containing yield; it will provide a value to the caller (None, as set in the variable), and it will suspend there).

From here, we have two options. If we choose to advance the generator by calling next(), the default value of 10 will be used, and it will go on with this as usual. This is because next() is technically the same as send(None), but this is covered in the if statement that will handle the value that we previously set.

If, on the other hand, we decide to provide an explicit value via send(<value>), this one will become the result of the yield statement, which will be assigned to the variable containing the length of the page to use, which, in turn, will be used to read from the database.

Successive calls will have this logic, but the important point is that now we can dynamically change the length of the data to read in the middle of the iteration, at any point.

Now that we understand how the previous code works, most Pythonistas would expect a simplified version of it (after all, Python is also about brevity and clean and compact code):

def stream_db_records(db_handler):
    retrieved_data = None
    page_size = 10
    try:
        while True:
            page_size = (yield retrieved_data) or page_size
            retrieved_data = db_handler.read_n_records(page_size)
    except GeneratorExit:
        db_handler.close()

This version is not only more compact, but it also illustrates the idea better. The parenthesis around the yield makes it clearer that it’s a statement (think of it as if it were a function call), and that we are using the result of it to compare it against the previous value.

This works as we expect it does, but we always have to remember to advance the coroutine before sending any data to it. If we forget to call the first next(), we’ll get a TypeError. This call could be ignored for our purposes because it doesn’t return anything we’ll use.

It would be good if we could use the coroutine directly, right after it is created without having to remember to call next() the first time, every time we are going to use it. Some authors devised an interesting decorator to achieve this. The idea of this decorator is to advance the coroutine, so the following definition works automatically:

@prepare_coroutine
def stream_db_records(db_handler):
    retrieved_data = None
    page_size = 10
    try:
        while True:
            page_size = (yield retrieved_data) or page_size
            retrieved_data = db_handler.read_n_records(page_size)
    except GeneratorExit:
    db_handler.close()

>>> streamer = stream_db_records(DBHandler("testdb"))
>>> len(streamer.send(5))
5

3.2. More advanced coroutines

So far, we have a better understanding of coroutines, and we are able to create simple ones to handle small tasks. We can say that these coroutines are, in fact, just more advanced generators (and that would be right, coroutines are just fancy generators), but, if we actually want to start supporting more complex scenarios, we usually have to go for a design that handles many coroutines concurrently, and that requires more features.

When handling many coroutines, we find new problems. As the control flow of our application becomes more complex, we want to pass values up and down the stack (as well as exceptions), be able to capture values from sub-coroutines we might call at any level, and finally schedule multiple coroutines to run toward a common goal.

To make things simpler, generators had to be extended once again. This is addressed by changing the semantic of generators so that they are able to return values, and introducing the new yield from construction.

3.2.1. Returning values in coroutines

As introduced at the beginning, the iteration is a mechanism that calls next() on an iterable object many times until a StopIteration exception is raised.

So far, we have been exploring the iterative nature of generators: we produce values one at a time, and, in general, we only care about each value as it’s being produced at every step of the for loop. This is a very logical way of thinking about generators, but coroutines have a different idea; even though they are technically generators, they weren’t conceived with the idea of iteration in mind, but with the goal of suspending the execution of a code until it’s resumed later on.

This is an interesting challenge; when we design a coroutine, we usually care more about suspending the state rather than iterating (and iterating a coroutine would be an odd case). The challenge lies in that it is easy to mix them both. This is because of a technical implementation detail; the support for coroutines in Python was built upon generators.

If we want to use coroutines to process some information and suspend its execution, it would make sense to think of them as lightweight threads (or green threads, as they are called in other platforms). In such a case, it would make sense if they could return values, much like calling any other regular function.

But let’s remember that generators are not regular functions, so in a generator, the construction value = generator() will do nothing other than create a generator object. What would be the semantics for making a generator return a value? It will have to be after the iteration is done.

When a generator returns a value, its iteration is immediately stopped (it can’t be iterated any further). To preserve the semantics, the StopIteration exception is still raised, and the value to be returned is stored inside the exception object. It’s the responsibility of the caller to catch it.

In the following example, we are creating a simple generator that produces two values and then returns a third. Notice how we have to catch the exception in order to get this value, and how it’s stored precisely inside the exception under the attribute named value:

>>> def generator():
...     yield 1
...     yield 2
...     return 3
...
>>> value = generator()
>>> next(value)
1
>>> next(value)
2
>>> try:
...     next(value)
... except StopIteration as e:
...     print(f">>>>>> returned value {e.value}")
...
>>>>>> returned value 3

3.2.2. Delegating into smaller coroutines: the yield from syntax

The previous feature is interesting in the sense that it opens up a lot of new possibilities with coroutines (generators), now that they can return values. But this feature, by itself, would not be so useful without proper syntax support, because catching the returned value this way is a bit cumbersome.

This is one of the main features of the yield from syntax. Among other things (that we’ll review in detail), it can collect the value returned by a sub-generator. Remember that we said that returning data in a generator was nice, but that, unfortunately, writing statements as value = generator() wouldn’t work. Well, writing it as value = yield from generator() would.

3.2.2.1. The simplest use of yield from

In its most basic form, the new yield from syntax can be used to chain generators from nested for loops into a single one, which will end up with a single string of all the values in a continuous stream.

The canonical example is about creating a function similar to itertools.chain() from the standard library. This is a very nice function because it allows you to pass any number of iterables and will return them all together in one stream.

The naive implementation might look like this:

def chain(*iterables):
    for it in iterables:
        for value in it:
            yield value

It receives a variable number of iterables, traverses through all of them, and since each value is iterable, it supports a for... in.. construction, so we have another for loop to get every value inside each particular iterable, which is produced by the caller function. This might be helpful in multiple cases, such as chaining generators together or trying to iterate things that it wouldn’t normally be possible to compare in one go (such as lists with tuples, and so on).

However, the yield from syntax allows us to go further and avoid the nested loop because it’s able to produce the values from a sub-generator directly. In this case, we could simplify the code like this:

def chain(*iterables):
    for it in iterables:
        yield from it

Notice that for both implementations, the behavior of the generator is exactly the same:

>>> list(chain("hello", ["world"], ("tuple", " of ", "values.")))
['h', 'e', 'l', 'l', 'o', 'world', 'tuple', ' of ', 'values.']

This means that we can use yield from over any other iterable, and it will work as if the top-level generator (the one the yield from is using) were generating those values itself.

This works with any iterable, and even generator expressions aren’t the exception. Now that we’re familiar with its syntax, let’s see how we could write a simple generator function that will produce all the powers of a number (for instance, if provided with all_powers(2, 3), it will have to produce 2^0, 2^1,… 2^3 ):

def all_powers(n, pow):
    yield from (n ** i for i in range(pow + 1))

While this simplifies the syntax a bit, saving one line of a for statement isn’t a big advantage, and it wouldn’t justify adding such a change to the language.

Indeed, this is actually just a side effect and the real raison d’être of the yield from construction is what we are going to explore in the following two sections.

3.2.2.2. Capturing the value returned by a sub-generator

In the following example, we have a generator that calls another two nested generators, producing values in a sequence. Each one of these nested generators returns a value, and we will see how the top-level generator is able to effectively capture the return value since it’s calling the internal generators through yield from:

def sequence(name, start, end):
    logger.info(f"{name} started at {start}")
    yield from range(start, end)
    logger.info(f"{name} finished at {end}")
    return end

def main():
    step1 = yield from sequence("first", 0, 5)
    step2 = yield from sequence("second", step1, 10)
    return step1 + step2

This is a possible execution of the code in main while it’s being iterated:

>>> g = main()
>>> next(g)
INFO:generators_yieldfrom_2:first started at 0
0
>>> next(g)
1
>>> next(g)
2
>>> next(g)
3
>>> next(g)
4
>>> next(g)
INFO:generators_yieldfrom_2:first finished at 5
INFO:generators_yieldfrom_2:second started at 5
5
>>> next(g)
6
>>> next(g)
7
>>> next(g)
8
>>> next(g)
9
>>> next(g)
INFO:generators_yieldfrom_2:second finished at 10
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 15

The first line of main delegates into the internal generator, and produces the values, extracting them directly from it. This is nothing new, as we have already seen. Notice, though, how the sequence() generator function returns the end value, which is assigned in the first line to the variable named step1, and how this value is correctly used at the start of the following instance of that generator.

In the end, this other generator also returns the second end value, and the main generator, in turn, returns the sum of them, which is the value we see once the iteration has stopped.

Tip

We can use yield from to capture the last value of a coroutine after it has finished its processing.

3.2.2.3. Sending and receiving data to and from a sub-generator

Now, we will see the other nice feature of the yield from syntax, which is probably what gives it its full power. As we have already introduced when we explored generators acting as coroutines, we know that we can send values and throw exceptions at them, and, in such cases, the coroutine will either receive the value for its internal processing, or it will have to handle the exception accordingly.

If we now have a coroutine that delegates into other ones (such as in the previous example), we would also like to preserve this logic. Having to do so manually would be quite complex if we didn’t have this handled by yield from automatically.

In order to illustrate this, let’s keep the same top-level generator (main) unmodified with respect to the previous example (calling other internal generators), but let’s modify the internal generators to make them able to receive values and handle exceptions. The code is probably not idiomatic, only for the purposes of showing how this mechanism works:

def sequence(name, start, end):
    value = start
    logger.info("%s started at %i", name, value)

    while value < end:
        try:
            received = yield value
            logger.info("%s received %r", name, received)
            value += 1

        except CustomException as e:
            logger.info("%s is handling %s", name, e)
            received = yield "OK"

    return end

Now, we will call the main coroutine, not only by iterating it, but also by passing values and throwing exceptions at it to see how they are handled inside sequence :

>>> g = main()
>>> next(g)
INFO: first started at 0
0
>>> next(g)
INFO: first received None
1
>>> g.send("value for 1")
INFO: first received 'value for 1'
2
>>> g.throw(CustomException("controlled error"))
INFO: first is handling controlled error
'OK'
... # advance more times
INFO:second started at 5
5
>>> g.throw(CustomException("exception at second generator"))
INFO: second is handling exception at second generator
'OK'

This example is showing us a lot of different things. Notice how we never send values to sequence, but only to main, and even so, the code that is receiving those values is the nested generators. Even though we never explicitly send anything to sequence, it’s receiving the data as it’s being passed along by yield from .

The main coroutine calls two other coroutines internally, producing their values, and it will be suspended at a particular point in time in any of those. When it’s stopped at the first one, we can see the logs telling us that it is that instance of the coroutine that received the value we sent. The same happens when we throw an exception to it. When the first coroutine finishes, it returns the value that was assigned in the variable named step1, and passed as input for the second coroutine, which will do the same (it will handle the send() and throw() calls, accordingly).

The same happens for the values that each coroutine produces. When we are at any given step, the return from calling send() corresponds to the value that the subcoroutine (the one that main is currently suspended at) has produced. When we throw an exception that is being handled, the sequence coroutine produces the value OK, which is propagated to the called (main), and which in turn will end up at main’s caller.

4. Asynchronous programming

With the constructions we have seen so far, we are able to create asynchronous programs in Python. This means that we can create programs that have many coroutines, schedule them to work in a particular order, and switch between them when they’re suspended after a yield from has been called on each of them.

The main advantage that we can take out of this is the possibility of parallelizing I/O operations in a non-blocking way. What we would need is a low-level generator (usually implemented by a third-party library) that knows how to handle the actual I/O while the coroutine is suspended. The idea is for the coroutine to effect suspension so that our program can handle another task in the meantime. The way the application would retrieve the control back is by means of the yield from statement, which will suspend and produce a value to the caller (as in the examples we saw previously when we used this syntax to alter the control flow of the program).

This is roughly the way asynchronous programming had been working in Python for quite a few years, until it was decided that better syntactic support was needed.

The fact that coroutines and generators are technically the same causes some confusion. Syntactically (and technically), they are the same, but semantically, they are different. We create generators when we want to achieve efficient iteration. We typically create coroutines with the goal of running non-blocking I/O operations.

While this difference is clear, the dynamic nature of Python would still allow developers to mix these different type of objects, ending up with a runtime error at a very late stage of the program. Remember that in the simplest and most basic form of the yield from syntax, we used this construction over iterables (we created a sort of chain function applied over strings, lists, and so on). None of these objects were coroutines, and it still worked. Then, we saw that we can have multiple coroutines, use yield from to send the value (or exceptions), and get some results back. These are clearly two very different use cases, however, if we write something along the lines of the following statement: result = yield from iterable_or_awaitable()

It’s not clear what iterable_or_awaitable returns. It can be a simple iterable such as a string, and it might still be syntactically correct. Or, it might be an actual coroutine. The cost of this mistake will be paid much later.

For this reason, the typing system in Python had to be extended. Before Python 3.5, coroutines were just generators with a @coroutine decorator applied, and they were to be called with the yield from syntax. Now, there is a specific type of object, that is, a coroutine.

This change heralded, syntax changes as well. The await and async def syntax were introduced. The former is intended to be used instead of yield from, and it only works with awaitable objects (which coroutines conveniently happen to be). Trying to call await with something that doesn’t respect the interface of an awaitable will raise an exception. The async def is the new way of defining coroutines, replacing the aforementioned decorator, and this actually creates an object that, when called, will return an instance of a coroutine.

Without going into all the details and possibilities of asynchronous programming in Python, we can say that despite the new syntax and the new types, this is not doing anything fundamentally different from concepts we have covered.

The idea of programming asynchronously in Python is that there is an event loop (typically asyncio because it’s the one that is included in the standard library, but there are many others that will work just the same) that manages a series of coroutines. These coroutines belong to the event loop, which is going to call them according to its scheduling mechanism. When each one of these runs, it will call our code (according to the logic we have defined inside the coroutine we programmed), and when we want to get control back to the event loop, we call await <coroutine>, which will process a task asynchronously. The event loop will resume and another coroutine will take place while that operation is left running.

In practice, there are more particularities and edge cases that are beyond the scope. It is, however, worth mentioning that these concepts are related to the ideas introduced in this chapter and that this arena is another place where generators demonstrate being a core concept of the language, as there are many things constructed on top of them.