Python Generators: Creating Memory-Efficient Iterators with yield

1. Syntax & Core Mechanics: The yield Statement 🔝

Functions containing yield return a generator object implementing the iterator protocol without executing code until next() is invoked:

# 1. Defining a generator function with yield & stepping with next()
def countdown(n):
    while n > 0:
        yield n
        n -= 1

timer = countdown(3)
print("Generator object: ", timer)
print("First next() call: ", next(timer))
print("Second next() call:", next(timer))
print("Third next() call: ", next(timer))

Terminal Output:

Generator object:  <generator object countdown at 0x7f...>
First next() call:  3
Second next() call: 2
Third next() call:  1

2. Consuming Generators with for Loops 🔝

Generators can be iterated with a for loop, which invokes next() each cycle and handles the StopIteration exception automatically:

# 2. Consuming generators with for loops (auto-handles StopIteration)
def number_sequence(limit):
    for i in range(limit):
        yield i

for num in number_sequence(4):
    print("Streamed number:", num)

Terminal Output:

Streamed number: 0
Streamed number: 1
Streamed number: 2
Streamed number: 3

3. Functions vs. Generators: Memory & Lifecycle 🔝

Contrasting Fibonacci generation using an eager list function versus a lazy generator stream:

# 3. List accumulator vs. Generator: Memory & Architecture
def fibonacci_list(n):
    result, a, b = [], 0, 1
    for _ in range(n):
        result.append(b)
        a, b = b, a + b
    return result

def fibonacci_gen(n):
    a, b = 0, 1
    for _ in range(n):
        yield b
        a, b = b, a + b

print("List (all in RAM):  ", fibonacci_list(6))
print("Generator (stream):", list(fibonacci_gen(6)))

Terminal Output:

List (all in RAM):   [1, 1, 2, 3, 5, 8]
Generator (stream): [1, 1, 2, 3, 5, 8]

4. Memory Efficiency & RAM Benchmarking 🔝

Generators eliminate intermediate buffer allocations when processing large datasets. Measured with Python's sys.getsizeof():

# 4. RAM footprint benchmark using sys.getsizeof()
import sys

num_list = [x * 2 for x in range(1_000_000)]
num_gen = (x * 2 for x in range(1_000_000))

print("List memory:     ", sys.getsizeof(num_list), "bytes")
print("Generator memory:", sys.getsizeof(num_gen), "bytes")

Terminal Output:

List memory:      8448728 bytes
Generator memory: 112 bytes

5. Advanced Two-Way Coroutines: send() and close() 🔝

Using val = yield total, callers pass data into running generators with send(val) or stop execution with close():

# 5. Advanced methods: send() and close()
def running_total():
    total = 0
    while True:
        value = yield total
        if value is not None:
            total += value

acc = running_total()
print("Initial primed yield:", next(acc))
print("After sending 10:   ", acc.send(10))
print("After sending 25:   ", acc.send(25))
acc.close()

Terminal Output:

Initial primed yield: 0
After sending 10:    10
After sending 25:    35

6. Delegating Sub-Generators with yield from 🔝

The yield from expression establishes a delegation pipeline between a generator and child sub-generators, flattening nested sequences cleanly:

# 6. Delegating sub-generators with yield from
def sub_a():
    yield "A1"
    yield "A2"

def sub_b():
    yield "B1"
    yield "B2"

def combined_stream():
    yield "Start"
    yield from sub_a()
    yield from sub_b()
    yield "End"

print("Flattened sequence:", list(combined_stream()))

Terminal Output:

Flattened sequence: ['Start', 'A1', 'A2', 'B1', 'B2', 'End']

7. Production Recipes 🔝

Streaming large files chunk-by-chunk using a fixed-size buffer to prevent RAM exhaustion:

# 7. Production Recipe: Chunked streaming file reader
import io

def read_in_chunks(file_obj, chunk_size=12):
    while True:
        chunk = file_obj.read(chunk_size)
        if not chunk:
            break
        yield chunk

stream = io.StringIO("PLUS2NET_DATA_STREAM_CHUNK_PROCESSING_DEMO")
for i, block in enumerate(read_in_chunks(stream, chunk_size=12), start=1):
    print(f"Chunk {i}: {block}")

Terminal Output:

Chunk 1: PLUS2NET_DAT
Chunk 2: A_STREAM_CHU
Chunk 3: NK_PROCESSIN
Chunk 4: G_DEMO

Building a stream processing pipeline by chaining modular generator stages:

# 8. Production Recipe: Multi-stage stream processing pipeline
def log_source():
    yield from [
        "INFO Server online",
        "WARN High CPU load",
        "ERROR DB connection failed",
        "ERROR Disk timeout"
    ]

def filter_errors(records):
    for line in records:
        if "ERROR" in line:
            yield line.split("ERROR ")[1]

pipeline = filter_errors(log_source())
for error_msg in pipeline:
    print("Caught error:", error_msg)

Terminal Output:

Caught error: DB connection failed
Caught error: Disk timeout

8. Common Developer Pitfalls & Traps 🔝

Generators are single-pass iterables. Once exhausted, a second iteration yields an empty sequence:

# 9. Common traps: Generator exhaustion (single-pass streams)
def count_two():
    yield 1
    yield 2

g = count_two()
print("First pass: ", list(g))
print("Second pass:", list(g))

Terminal Output:

First pass:  [1, 2]
Second pass: []

9. Interactive Practice & Companion Jupyter Notebook 🔝

Practice generator exercises, memory benchmarks, and coroutines in our companion Jupyter Notebook:

Self-Verifying Assertion Suite

Run these unit checks directly in Python to verify generator mechanics:

# 10. Self-verifying unit test suite for generator mechanics
# Test 1: Basic yield sequence
def gen_three():
    yield 10
    yield 20
    yield 30

g = gen_three()
assert next(g) == 10
assert next(g) == 20
assert next(g) == 30

# Test 2: StopIteration on exhaustion
try:
    next(g)
    assert False
except StopIteration:
    pass

# Test 3: Generator expression
sq = (x * x for x in [1, 2, 3])
assert list(sq) == [1, 4, 9]

# Test 4: yield from delegation
def sub():
    yield 'a'
    yield 'b'

def parent():
    yield from sub()

assert list(parent()) == ['a', 'b']

print("All generator unit assertions successfully verified!")

Terminal Output:

All generator unit assertions successfully verified!

10. Frequently Asked Questions (FAQ) 🔝

Further Reading & Related Tutorials 🔝

• Python Iterators Overview — Understanding iterables, iterators, and the iterator protocol.
• Python next() Built-in Function — Advancing iterators and handling default fallbacks.
• Python Functions Tutorial — Function declarations, parameters, and return statements.
• Python for Loops — Iterating over sequences and stream generators.
• Python range() Built-in Function — Memory-efficient integer range sequences.
• Python List Comprehension — Contrasting eager lists with lazy generator expressions.
• Python File Read Operations — Efficiently reading files line-by-line and in chunks.
• Python Built-in Functions Guide — Master reference for all Python built-in operations.

Subhendu Mohapatra

Author & Instructor at plus2net

I write and maintain practical tutorials on Python, PHP, SQL, JavaScript, HTML, jQuery, and web development at plus2net. The tutorials focus on clear explanations, working examples, and code that readers can test and adapt while learning.

About the author   Watch on YouTube

Connect: