Introduction

Generators are one of Python's most useful tools for working with sequences lazily. Instead of building all values up front, a generator produces them one at a time as iteration advances.

That small change has a big effect on memory use, performance characteristics, and the way data flows through a program.

What a Generator Is

A generator is an iterator created either by a function that uses yield or by a generator expression. When Python reaches yield, it returns a value to the caller and pauses the function state. On the next iteration, execution resumes exactly where it left off.

Basic example:

1def count_up_to(limit):
2    current = 1
3    while current <= limit:
4        yield current
5        current += 1
6
7
8for number in count_up_to(3):
9    print(number)

Output:

1
2
3

The function does not build a list [1, 2, 3]. It yields each number only when requested by the loop.

Why Generators Matter

The main advantage is lazy evaluation. A list computes and stores every element immediately. A generator computes the next element only when iteration asks for it.

That matters when:

• the sequence is very large
• values are expensive to compute
• data arrives from a file, socket, or stream
• you want to compose processing steps without temporary lists

For example, summing squares with a list creates all squares first:

total = sum([x * x for x in range(1_000_000)])

Using a generator expression avoids building the intermediate list:

total = sum(x * x for x in range(1_000_000))

Both produce the same result, but the generator version usually uses less memory.

Generator Functions and State

When a normal function returns, its local state disappears. A generator function is different. It keeps enough state to continue later.

1def running_totals(values):
2    total = 0
3    for value in values:
4        total += value
5        yield total
6
7
8for total in running_totals([5, 10, 3]):
9    print(total)

The local variable total survives across yield points. That is why generators are so useful for incremental computation.

Generator Expressions

Python also supports generator expressions, which look like list comprehensions without the outer brackets.

List comprehension:

numbers = [x * 2 for x in range(5)]

Generator expression:

numbers = (x * 2 for x in range(5))

The first creates a list immediately. The second creates a generator object that yields values on demand.

You can inspect that behavior directly:

1gen = (x * 2 for x in range(3))
2
3print(next(gen))
4print(next(gen))
5print(next(gen))

This prints 0, then 2, then 4.

Generators in Pipelines

Generators become especially powerful when chained together. Each stage can process one item at a time instead of materializing the full dataset at every step.

1def read_numbers():
2    for line in ["1", "2", "3", "4"]:
3        yield int(line)
4
5
6def even_numbers(values):
7    for value in values:
8        if value % 2 == 0:
9            yield value
10
11
12def squared(values):
13    for value in values:
14        yield value * value
15
16
17result = list(squared(even_numbers(read_numbers())))
18print(result)  # [4, 16]

This style is common in data processing because it keeps each step focused and memory-friendly.

Generators Are Single-Use Iterators

One important property surprises many beginners: generators are consumed as you iterate them.

1gen = (x for x in range(3))
2
3print(list(gen))  # [0, 1, 2]
4print(list(gen))  # []

After the first pass, the generator is exhausted. If you need to iterate again, recreate it or store the values in a reusable collection such as a list.

Common Pitfalls

The biggest pitfall is expecting a generator to behave like a list. You cannot index it directly, and once consumed, it does not reset automatically.

Another issue is debugging. Because generators are lazy, bugs may appear later than expected, only when iteration actually happens.

Developers also sometimes choose generators for tiny fixed datasets where a list would be clearer. Laziness is useful, but it is not automatically the best choice for every piece of code.

Finally, remember that generator expressions are not magic performance boosts. They reduce memory pressure, but if the computation itself is expensive, that cost still exists.

Summary

• Generators produce values lazily, one item at a time.
• Use yield in a function or a generator expression to create them.
• They are great for large sequences, streams, and processing pipelines.
• Generators preserve state between yield points but are single-use iterators.
• Choose generators when lazy evaluation improves clarity or memory use, not just because they are available.