You may have heard that functions with yield are called generators in Python. What is a generator?

Let us set aside generators for now and use a common programming problem to demonstrate the concept of yield.

How to Generate the Fibonacci Sequence

The Fibonacci sequence is a very simple recursive sequence. Except for the first and second numbers, any number can be obtained by adding the previous two numbers. Using a computer program to output the first N numbers of the Fibonacci sequence is a very simple problem, and many beginners can easily write the following function:

Listing 1. Simply output the first N numbers of the Fibonacci sequence

Executing the above code, we can obtain the following output:

1 
1 
2 
3 
5

The result is fine, but experienced developers will point out that directly using print in the fab function to print numbers makes the function poorly reusable, because the fab function returns None, and other functions cannot obtain the sequence generated by this function.

To improve the reusability of the fab function, it is best not to print the sequence directly, but to return a List. The following is the second version of the rewritten fab function:

Listing 2. Second version of outputting the first N numbers of the Fibonacci sequence

You can print the List returned by the fab function in the following way:

1 
1 
2 
3 
5

The rewritten fab function meets the reusability requirement by returning a List, but more experienced developers will point out that the memory occupied by this function during execution increases as the parameter max increases. To control memory usage, it is best not to use a List

to store intermediate results, but rather to iterate through an iterable object. For example, in Python 2.x, the code:

Listing 3. Iterating through an iterable object

for i in range(1000): pass

will generate a List with 1000 elements, while the code:

for i in xrange(1000): pass

will not generate a List with 1000 elements, but instead returns the next value in each iteration, occupying very little memory space. This is because xrange does not return a List, but returns an iterable object.

Using iterable, we can rewrite the fab function as a class that supports iterable. The following is the third version of Fab:

Listing 4. Third version

The Fab class continuously returns the next number of the sequence through next(), and memory usage remains constant:

1 
1 
2 
3 
5

However, this version rewritten using a class is far less concise than the first version of the fab function. If we want to maintain the conciseness of the first version of the fab function while also achieving the iterable effect, yield comes into play:

Listing 5. Fourth version using yield

Compared with the first version, the fourth version of fab simply changes print b to yield b, achieving the iterable effect while maintaining conciseness.

Calling the fourth version of fab is exactly the same as calling the second version of fab:

1 
1 
2 
3 
5

Simply put, the role of yield is to turn a function into a generator. A function with yield is no longer an ordinary function. The Python interpreter will treat it as a generator. Calling fab(5) will not execute the fab function, but will return an iterable object! When the for loop executes, each iteration executes the code inside the fab function. When it reaches yield b, the fab function returns an iteration value. On the next iteration, the code continues executing from the statement after yield b, and the function's local variables appear exactly the same as before the last interruption. The function then continues executing until it encounters yield again.

You can also manually call the next() method of fab(5) (because fab(5) is a generator object that has a next() method), which allows us to see the execution flow of fab more clearly:

Listing 6. Execution flow

When the function finishes execution, the generator automatically raises a StopIteration exception, indicating that the iteration is complete. In a for loop, there is no need to handle the StopIteration exception; the loop will end normally.

We can draw the following conclusions:

A function with yield is a generator. Unlike ordinary functions, generating a generator looks like a function call, but it does not execute any function code until next() is called on it (in a for loop, next() is called automatically). Although the execution flow still follows the function's flow, it is interrupted every time it reaches a yield statement and returns an iteration value. The next time execution resumes, it continues from the statement following yield. It looks as if a function has been interrupted by yield several times during normal execution, and each interruption returns the current iteration value through yield.

The benefits of yield are obvious. Rewriting a function as a generator gives it iteration capability. Compared with using a class instance to save state and calculate the next value for next(), it is not only concise in code, but also exceptionally clear in execution flow.

How can we determine whether a function is a special generator function? We can use isgeneratorfunction to judge:

Listing 7. Using isgeneratorfunction to judge

We should pay attention to distinguishing between fab and fab(5). fab is a generator function, while fab(5) is a generator returned by calling fab, just like the difference between a class definition and a class instance:

Listing 8. Class definition and class instance

Each call to the fab function generates a new generator instance, and the instances do not affect each other:

The Role of return

In a generator function, if there is no return, it executes by default until the function completes. If return is encountered during execution, it directly raises StopIteration to terminate the iteration.

Another Example

Another example of yield comes from file reading. If you directly call the read() method on a file object, it can cause unpredictable memory usage. A good approach is to use a fixed-length buffer to continuously read file content. Through yield, we no longer need to write an iteration class for file reading, and can easily implement file reading:

Listing 9. Another example of yield

The above only briefly introduces the basic concepts and usage of yield. yield has more powerful uses in Python 3, which we will discuss in subsequent articles.

Note: All the code in this article has been tested and passed in Python 2.7.

Original URL: https://www.ibm.com/developerworks/cn/opensource/os-cn-python-yield/