Generators: Lazy Iteration with yield

Building a 10-million-item list so you can iterate over it once is a waste of memory and a waste of time. Most of the time you don't need the whole list. You need the next item, then the one after that, and so on. A generator produces items on demand: it computes the next item when asked, then pauses until it is asked again.

yield is the keyword that turns a function into a generator. Once you understand it, you can process a 50 GB log file through ten chained steps using only a few megabytes of RAM. The same idea (produce one item at a time, only when asked) is behind itertools, reading files line by line, async iteration, and the "pipeline" patterns in Python.

1. The Problem With Eager Lists

def squares(n):
    return [x*x for x in range(n)]      # builds the whole list in memory

total = sum(squares(10_000_000))         # ~400 MB before sum even starts

The list comprehension allocates every square, all at once, even though sum only needs one at a time. For large n your program becomes slow or runs out of memory, and it doesn't need to. The fix is to compute values lazily, only when they are needed.

def squares(n):
    for x in range(n):
        yield x*x                        # produce ONE value, then pause

total = sum(squares(10_000_000))         # constant memory

yield is the only change. The function now returns a generator object. It produces the 10 million squares one at a time, and each square is freed as soon as sum has used it.

2. yield Turns a Function Into a Generator

The presence of any yield in a function body makes the whole function a generator function. Calling it doesn't run the body. It returns a generator object that you iterate over.

def count_up_to(n):
    print("starting")
    i = 1
    while i <= n:
        yield i
        i += 1
    print("done")

g = count_up_to(3)
print(g)                            # <generator object count_up_to at 0x...>
                                    # NOTHING printed yet, because the body has not run

print(next(g))                      # starting   |   1
print(next(g))                      #            |   2
print(next(g))                      #            |   3
print(next(g))                      # done       |   StopIteration

Each next(g) runs the body until the next yield, gives you the yielded value, and pauses. The generator keeps all its local variables and its position in the code. The next next(g) continues from the line right after the yield.

When the function reaches its end (or a return), Python raises StopIteration to show that the generator is exhausted.

3. Iterating Generators

You almost never call next() directly. for, list(), sum(), min(), max(), any(), all(), and most of itertools all read iterables the same way: call next() in a loop, catch StopIteration, stop.

def count_up_to(n):
    i = 1
    while i <= n:
        yield i
        i += 1

for x in count_up_to(5):
    print(x)                        # 1, 2, 3, 4, 5

print(list(count_up_to(5)))         # [1, 2, 3, 4, 5]
print(sum(count_up_to(100)))        # 5050

A for loop is the standard way to read all values from a generator. Use next() only when you want exactly one value (for example, to read the first item, then loop over the rest).

4. Generator Expressions

A generator expression looks like a list comprehension, but with round brackets instead of square ones. It is lazy.

squares_list = [x*x for x in range(1_000_000)]      # builds a list of 1,000,000 numbers
squares_gen  = (x*x for x in range(1_000_000))      # builds a generator

print(sum(squares_list))            # works
print(sum(squares_gen))             # also works, constant memory

When a generator expression is the only argument to a function, you can leave out its parentheses. The function call's own parentheses are enough:

total = sum(x*x for x in range(1_000_000))          # no second pair of parentheses needed
maximum = max(len(line) for line in open("log.txt"))

Curly braces give you {x*x for x in range(5)}, a set comprehension. With key: value, it's a dict comprehension. Generator expressions are the only comprehension with round brackets.

5. yield from: Delegating to a Sub-Iterator

When you want to yield every item from another iterable, the long form is a for loop:

def chain_two(a, b):
    for x in a:
        yield x
    for x in b:
        yield x

yield from does the same in one line:

def chain_two(a, b):
    yield from a
    yield from b

print(list(chain_two([1, 2, 3], "ab")))     # [1, 2, 3, 'a', 'b']

yield from also passes on send(), throw(), and return values from sub-generators, which is useful when you combine generators. For now, think of it as the clean way to yield every item of another iterable.

6. Memory: The Main Reason

import sys

list_comp = [x*x for x in range(1_000_000)]
gen_exp   = (x*x for x in range(1_000_000))

print(sys.getsizeof(list_comp))     # 8_448_728   (~8.4 MB)
print(sys.getsizeof(gen_exp))       #       208   (208 bytes)

The generator is roughly 40,000× smaller. It stores how to compute the values (the loop state and the expression), not the results. Each result is computed, consumed, and discarded.

sys.getsizeof only measures the outer object, not what it contains. The list's 8 MB is the array of pointers to the int objects, and the generator's 208 B is the generator frame. Either way, the difference is very large.

7. Real Generators: Reading a Huge File

This is a classic Python pattern. The file object returned by open() is already a lazy iterator over lines.

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:                       # one line at a time, not the whole file
            yield line.rstrip("\n")

def only_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

def with_lineno(lines):
    for i, line in enumerate(lines, start=1):
        yield f"{i:6d}: {line}"

# Pipeline: read, keep only errors, add line numbers, print
for entry in with_lineno(only_errors(read_lines("server.log"))):
    print(entry)

For a 10 GB log, the code and the memory use are the same. Each line passes through every stage exactly once and is then freed. This pattern (generators feeding generators) works like a UNIX pipe, where each command passes its output to the next.

8. Pipelines: Composing Generators

def numbers():
    n = 1
    while True:                              # an infinite generator is fine, because it is lazy
        yield n
        n += 1

def squared(source):
    for x in source:
        yield x * x

def take(source, n):
    for i, x in enumerate(source):
        if i >= n:
            return
        yield x

pipeline = take(squared(numbers()), 5)
print(list(pipeline))                        # [1, 4, 9, 16, 25]

There are three stages (produce, transform, limit), connected by function calls. Nothing is computed until list() starts asking for values. Infinite generators are safe as long as a later stage stops the iteration (take, itertools.islice, a break, etc.).

itertools has many lazy iterators that fit into pipelines: islice, chain, groupby, takewhile, dropwhile, tee, accumulate. See the Itertools lesson, which is mostly about combining them.

9. send(), throw(), close(): Two-Way Generators

Generators were extended in PEP 342 to be two-way. The caller can send values into the generator at each yield:

def echo():
    while True:
        received = yield
        print(f"got: {received}")

g = echo()
next(g)                                      # start the generator (run it to the first yield)
g.send("hello")                              # got: hello
g.send("world")                              # got: world
g.close()                                    # raises GeneratorExit inside the generator to end it

g.throw(SomeException) raises an exception inside the generator at the paused yield. The generator can handle it or let it propagate. g.close() raises GeneratorExit, giving the generator a chance to clean up.

This two-way yield was Python's original "coroutine" mechanism. In modern Python, async/await replaces nearly every use of it. Learn to recognise the syntax in older code, but don't use it in new code.

10. itertools: Lazy Iterators in the Standard Library

Most of itertools is lazy iterators written in C. They are fast, and you can combine them like generators.

from itertools import islice, chain, count, takewhile

# First 5 squares of an infinite counter
print(list(islice((x*x for x in count(1)), 5)))    # [1, 4, 9, 16, 25]

# Concatenate without copying
print(list(chain([1, 2], (3, 4), {5, 6})))         # [1, 2, 3, 4, 5, 6]

# Take while a condition holds
print(list(takewhile(lambda x: x < 10, count(1)))) # [1, 2, 3, 4, 5, 6, 7, 8, 9]

When you find yourself writing a complicated while loop that tracks its own state, check itertools first. There is often a one-line solution. See the Itertools lesson.

11. Async Generators: A Short Preview

async def plus yield gives you an async generator, consumed with async for. They are used for streaming async I/O: paginated APIs, websockets, server-sent events. You'll meet them when you reach asyncio. They work the same way as normal generators (lazy, one item at a time, paused between yields), but you can use await inside them.

Common Mistakes

1. Treating a generator like a list: it works only once

g = (x*x for x in range(5))
print(list(g))                      # [0, 1, 4, 9, 16]
print(list(g))                      # []          (the generator is already used up!)

A generator can be iterated exactly once. The second loop gets an empty iterator. If you need to iterate twice:

# Option A: keep the recipe and re-call it
def squares(n):
    for x in range(n):
        yield x*x

print(list(squares(5)))             # fresh
print(list(squares(5)))             # fresh again

# Option B: materialise once
data = list(x*x for x in range(5))  # now a real list, so you can loop over it many times

2. len(generator) doesn't work

g = (x for x in range(5))
print(len(g))                       # TypeError: object of type 'generator' has no len()

Generators don't know their length, because they produce values on demand. If you need a count, either build a list (len(list(g)), but that uses up the generator) or count as you go (sum(1 for _ in g)).

3. Using a generator when you need to re-iterate

results = (process(item) for item in source)
if any(r > threshold for r in results):     # this loops over results once
    for r in results:                       # results is already empty here!
        ...

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def process(*_a, **_kw):
    print('-> process() called')
    return _AutoMock('process()')
source = ["alpha", "beta", "gamma"]
threshold = _AutoMock('threshold')

If you need to iterate again, build a list once (results = list(...)) or write a generator function that you can call again. itertools.tee can split one generator into N independent iterators, but it stores values in memory and is rarely the right choice.

4. Confusing comprehension brackets

[x for x in r]          # list
(x for x in r)          # generator
{x for x in r}          # set
{k: v for k, v in r}    # dict

r = [("alpha", 1), ("beta", 2), ("gamma", 3)]

Round brackets in a comprehension always mean generator expression, never a tuple. There is no "tuple comprehension" syntax. To build a tuple, write tuple(x for x in r).

5. Holding files open without with

def lines(path):
    f = open(path)
    for line in f:
        yield line
    f.close()                       # never reached if the consumer breaks early

If the consumer stops iterating partway (break, an exception, islice), f.close() never runs. Use a with block instead. When the generator is closed (by g.close(), or when Python cleans up the generator object), GeneratorExit is raised inside it, the with block exits, and the file is closed.

def lines(path):
    with open(path) as f:           # always closes the file, even if the loop stops early
        for line in f:
            yield line

🎯 Your Turn: Stream a CSV in Chunks

Write read_csv_chunks(path, chunk_size=1000) that yields lists of up to chunk_size rows from a CSV file. Each row is a list of strings. Pair it with csv.reader (no need to parse manually). The function must:

• be a generator (use yield, not return),
• never hold more than chunk_size + 1 rows in memory at once,
• yield a final partial chunk if the file's row count isn't a multiple of chunk_size,
• use a with block to close the file even if the consumer stops early,
• skip the header row if has_header=True is passed.

import csv

def read_csv_chunks(path, chunk_size=1000, has_header=False):
    # TODO 1: open the file inside a `with` block
    # TODO 2: build a csv.reader over the file
    # TODO 3: if has_header, advance past the first row
    # TODO 4: accumulate rows into a chunk list; yield when it reaches chunk_size
    # TODO 5: after the loop, yield any remaining partial chunk
    ...

# Usage with itertools.islice to grab only the first few chunks:
from itertools import islice
for chunk in islice(read_csv_chunks("big.csv", chunk_size=500, has_header=True), 3):
    print(f"got {len(chunk)} rows; first row: {chunk[0]}")

import csv

def read_csv_chunks(path, chunk_size=1000, has_header=False):
    """Yield lists of up to chunk_size rows from a CSV file."""
    with open(path, newline="", encoding="utf-8") as f:
        reader = csv.reader(f)
        if has_header:
            next(reader, None)                  # skip header if present

        chunk = []
        for row in reader:
            chunk.append(row)
            if len(chunk) == chunk_size:
                yield chunk
                chunk = []

        if chunk:                               # final partial chunk
            yield chunk


# Demo with a tiny in-memory CSV
import io, csv as _csv
sample = io.StringIO("name,age\nalice,30\nbob,25\ncarol,28\ndan,40\nellen,22\n")
# (use the generator pattern the same way; here we read from StringIO)

def read_csv_chunks_from(fileobj, chunk_size, has_header=False):
    reader = _csv.reader(fileobj)
    if has_header: next(reader, None)
    chunk = []
    for row in reader:
        chunk.append(row)
        if len(chunk) == chunk_size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk

print(list(read_csv_chunks_from(sample, chunk_size=2, has_header=True)))
# [[['alice', '30'], ['bob', '25']],
#  [['carol', '28'], ['dan', '40']],
#  [['ellen', '22']]]

What You Learned

• A function with yield is a generator function. Calling it returns a generator object, but the body doesn't run until you iterate.
• Each yield pauses the function, hands a value to the consumer, and resumes from the same line on the next next().
• Generator expressions (x for x in ...) look like list comprehensions, but they are lazy. They use round brackets.
• yield from sub delegates to a sub-iterator in one line.
• Generators use constant memory regardless of the size of the sequence they produce.
• Connect generators into pipelines (take(squared(numbers()))), like a UNIX pipe.
• Generators work only once: after one full iteration they are empty. Call the generator function again, or build a list with list(...), if you need to iterate again.
• len() doesn't work on generators. Neither does indexing. They're streams, not collections.
• itertools is the standard library's generator toolkit. Use it before writing manual loops.
• Always wrap file I/O inside a generator in with open(...) so cleanup runs on early exit.
• send(), throw() and close() exist but are rarely needed. async/await is the modern replacement for two-way generators.

Next: Lambda Expressions: the one-line anonymous function, and when it is useful with map, filter, and key=.