Introduction

Generator functions are ideal when you want to process sequences lazily instead of building full collections in memory. They let you produce values one item at a time with yield, preserving execution state between iterations. In practical systems, generators improve memory usage, enable stream processing, and make data pipelines composable.

Lazy Evaluation and Memory Efficiency

A generator computes values only when requested by iteration.

1def squares(n: int):
2    for i in range(n):
3        yield i * i
4
5for value in squares(5):
6    print(value)

Compared with list construction, this avoids storing all results at once.

1# eager
2values = [i * i for i in range(10_000_000)]
3
4# lazy
5gen_values = (i * i for i in range(10_000_000))

If a caller consumes only first few items, the rest is never computed.

Streaming Large Files

Generators are a strong fit for log and data-file processing.

1def error_lines(path: str):
2    with open(path, "r", encoding="utf-8") as fh:
3        for line in fh:
4            if "ERROR" in line:
5                yield line.rstrip("\n")
6
7for line in error_lines("app.log"):
8    print(line)

This keeps memory bounded even for multi-gigabyte files.

Building Composable Pipelines

Generators can be chained so each stage transforms data incrementally.

1def parse_ints(lines):
2    for line in lines:
3        line = line.strip()
4        if line:
5            yield int(line)
6
7
8def only_even(nums):
9    for n in nums:
10        if n % 2 == 0:
11            yield n
12
13
14def running_sum(nums):
15    total = 0
16    for n in nums:
17        total += n
18        yield total
19
20source = ["1", "2", "3", "4", "5", "6"]
21pipeline = running_sum(only_even(parse_ints(source)))
22print(list(pipeline))

Pipeline style is easy to test because each stage has a focused responsibility.

Infinite and On-Demand Sequences

Generators can model conceptually infinite sequences safely, as long as consumers bound iteration.

1def fibonacci():
2    a, b = 0, 1
3    while True:
4        yield a
5        a, b = b, a + b
6
7fib = fibonacci()
8for _ in range(10):
9    print(next(fib))

This is cleaner than building ever-growing lists when only next values are needed.

Backpressure-Friendly Data Flow

In producer-consumer workflows, generators naturally apply backpressure. The producer runs only when consumer asks for next item.

1def produce_jobs():
2    for i in range(1, 6):
3        print("producing", i)
4        yield {"id": i, "payload": i * 10}
5
6
7def consume_jobs(jobs):
8    for job in jobs:
9        print("consumed", job["id"], "value", job["payload"])
10
11consume_jobs(produce_jobs())

This pull-based flow avoids unnecessary buffering in many single-process tasks.

Stateful Generators and send

Generators can also receive values via send, enabling lightweight state machines.

1def accumulator():
2    total = 0
3    while True:
4        value = yield total
5        if value is None:
6            return
7        total += value
8
9acc = accumulator()
10print(next(acc))    # initial total
11print(acc.send(5))
12print(acc.send(7))

This is less common than modern async patterns but can be useful for custom protocols.

Generators vs Async Generators

Regular generators are synchronous and suited for CPU or file iteration. Async generators are for asynchronous sources such as network events.

1import asyncio
2
3async def async_numbers():
4    for i in range(3):
5        await asyncio.sleep(0.1)
6        yield i

Choosing the right type avoids mixing blocking and non-blocking execution models.

When Generators Are Not Ideal

Avoid generators when you need:

• random indexing access
• repeated full passes over stable data
• serialization of complete datasets without re-iteration

In those cases, materialized collections such as lists, arrays, or dataframes may be better.

Testing Generator Logic

Generator tests should validate yielded sequence and exhaustion behavior.

def test_squares():
    gen = squares(3)
    assert list(gen) == [0, 1, 4]

Also test partial consumption if pipeline behavior depends on laziness.

Common Pitfalls

A common pitfall is trying to reuse an exhausted generator and expecting data again.

Another pitfall is converting generators to lists immediately, which removes memory advantages.

A third pitfall is placing side effects in transformation stages, making pipeline behavior harder to reason about.

Teams also create infinite generators without bounded consumers, causing runaway loops.

Summary

• Generator functions enable lazy, on-demand iteration with yield.
• They are excellent for large-file processing and composable data pipelines.
• They reduce memory pressure by avoiding full materialization.
• They support advanced patterns such as infinite sequences and stateful send workflows.
• Use generators where sequential consumption matters, and switch to materialized structures when random access is needed.