Process a list in batches in Python

Some work is done better in groups than one item at a time: saving records to a database, or sending requests to a service that accepts at most 100 items per request. This program splits a list into batches of a fixed size and passes each batch to a function.

The program

def process_in_batches(items, size, process):
    if size < 1:
        raise ValueError("size must be at least 1")
    for start in range(0, len(items), size):
        process(items[start:start + size])


def save_batch(batch):
    print(f"Saving {len(batch)} records: {batch}")


records = list(range(1, 12))
process_in_batches(records, 4, save_batch)

Saving 4 records: [1, 2, 3, 4]
Saving 4 records: [5, 6, 7, 8]
Saving 3 records: [9, 10, 11]

How it works

• range(0, len(items), size) gives the start of each batch: 0, 4, 8.
• items[start:start + size] takes one batch. A slice that goes past the end of the list stops at the end without an error, so the last batch has fewer items (here 3).
• process is a function passed in as an argument. The batching code does not need to know what happens to each batch, so you can reuse it for any kind of work.
• With a size of 0, range() raises a ValueError. With a negative size, the loop does nothing and gives no error at all. The function checks the size first and gives a clear message.

Another way: itertools.batched

Python 3.12 added itertools.batched(). It gives tuples, and it works with any iterable, not only lists. That includes a file, which you can then process in batches of lines without reading it all into memory:

from itertools import batched

names = ["Ada", "Linus", "Grace", "Alan", "Hedy", "Margaret", "Guido"]
for number, batch in enumerate(batched(names, 3), start=1):
    print(f"Batch {number}: {batch}")

Batch 1: ('Ada', 'Linus', 'Grace')
Batch 2: ('Alan', 'Hedy', 'Margaret')
Batch 3: ('Guido',)