Comprehensions: Lists, Dicts, Sets, Generators
1 · The lesson
readA comprehension is a one-line, declarative way to build a list, set, dict, or generator from another iterable. You describe what you want (what each element looks like, and where the values come from), and Python builds it. The imperative for loop describes how, and the comprehension describes what.
Once you know the four forms, you can replace many of the loops you used to write with one line.
1. The Four Forms
The same syntax gives you four kinds of container:
nums = [1, 2, 3, 4, 5] squares_list = [n * n for n in nums] # list: [1, 4, 9, 16, 25] squares_set = {n * n for n in nums} # set: {1, 4, 9, 16, 25} squares_dict = {n: n * n for n in nums} # dict: {1: 1, 2: 4, ...} squares_gen = (n * n for n in nums) # generator: makes each value only when asked
The brackets pick the type. The rest is identical: expression for variable in iterable. The generator form doesn't build anything until you iterate over it. Section 8 covers this.
2. Why Comprehensions
Compare the imperative form to the declarative one:
# Imperative style: 4 lines that keep changing a result list doubled = [] for n in nums: doubled.append(n * 2) # Declarative style: one line, no result list to change, no .append doubled = [n * 2 for n in nums]
setup added so this can run · re-runs 1 earlier block on this page first
nums = ["alpha", "beta", "gamma"] # Earlier blocks on this page that this one builds on, re-run quietly first. _pm_earlier = [ 'nums = [1, 2, 3, 4, 5]\n\nsquares_list = [n * n for n in nums] # list: [1, 4, 9, 16, 25]\nsquares_set = {n * n for n in nums} # set: {1, 4, 9, 16, 25}\nsquares_dict = {n: n * n for n in nums} # dict: {1: 1, 2: 4, ...}\nsquares_gen = (n * n for n in nums) # generator: makes each value only when asked', ] def _pm_run_earlier(_srcs, _g): import sys as _s, io as _io, warnings as _w for _src in _srcs: _o, _e = _s.stdout, _s.stderr _s.stdout = _s.stderr = _io.StringIO() try: with _w.catch_warnings(): _w.simplefilter('ignore') exec(compile(_src, '<earlier block>', 'exec'), _g) except (Exception, SystemExit): pass finally: _s.stdout, _s.stderr = _o, _e if 'matplotlib.pyplot' in _s.modules: _s.modules['matplotlib.pyplot'].close('all') # their plots belong to their own blocks _pm_run_earlier(_pm_earlier, globals()) del _pm_earlier, _pm_run_earlier
The comprehension is shorter. It is also slightly faster, because it doesn't look up and call .append on every pass. And it reads as one sentence: "doubled values, one for each n in nums". You think about how each value is transformed, not about the loop steps.
3. Filtering with if
Append an if after the for to skip items:
evens = [n for n in range(20) if n % 2 == 0] # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18] long_names = [name for name in ["Al", "Alice", "Bo", "Charlie"] if len(name) > 3] # ['Alice', 'Charlie']
You can write several ifs in a row. They are combined with logical AND:
divisible_by_six = [n for n in range(50) if n % 2 == 0 if n % 3 == 0] # same as divisible_by_six = [n for n in range(50) if n % 2 == 0 and n % 3 == 0]
Two ifs are easy to read. Three are a warning sign: split the logic up.
4. Conditional Expression Inside the Result
There are two places if can appear in a comprehension, and they mean different things.
nums = [-3, -1, 0, 2, 5] # Filtering: some items are skipped positives = [n for n in nums if n > 0] # [2, 5] # Conditional expression: keeps every item and changes it based on a condition clamped = [n if n > 0 else 0 for n in nums] # [0, 0, 0, 2, 5]
The first drops negatives. The second replaces them with zero. The position of the if is what changes:
[expr for x in it if cond]:ifafter thefor→ filter.[expr_a if cond else expr_b for x in it]: a conditional expression insideexpr→ transform.
You can combine them to filter and transform:
# Square positives only, drop the rest [n * n for n in nums if n > 0] # [4, 25] # Square positives, turn the others into 0, keep every item [n * n if n > 0 else 0 for n in nums] # [0, 0, 0, 4, 25]
setup added so this can run · re-runs 2 earlier blocks on this page first
nums = [3, -1, 4, -1, 5] # Earlier blocks on this page that this one builds on, re-run quietly first. _pm_earlier = [ 'nums = [1, 2, 3, 4, 5]\n\nsquares_list = [n * n for n in nums] # list: [1, 4, 9, 16, 25]\nsquares_set = {n * n for n in nums} # set: {1, 4, 9, 16, 25}\nsquares_dict = {n: n * n for n in nums} # dict: {1: 1, 2: 4, ...}\nsquares_gen = (n * n for n in nums) # generator: makes each value only when asked', 'nums = [-3, -1, 0, 2, 5]\n\n# Filtering: some items are skipped\npositives = [n for n in nums if n > 0] # [2, 5]\n\n# Conditional expression: keeps every item and changes it based on a condition\nclamped = [n if n > 0 else 0 for n in nums] # [0, 0, 0, 2, 5]', ] def _pm_run_earlier(_srcs, _g): import sys as _s, io as _io, warnings as _w for _src in _srcs: _o, _e = _s.stdout, _s.stderr _s.stdout = _s.stderr = _io.StringIO() try: with _w.catch_warnings(): _w.simplefilter('ignore') exec(compile(_src, '<earlier block>', 'exec'), _g) except (Exception, SystemExit): pass finally: _s.stdout, _s.stderr = _o, _e if 'matplotlib.pyplot' in _s.modules: _s.modules['matplotlib.pyplot'].close('all') # their plots belong to their own blocks _pm_run_earlier(_pm_earlier, globals()) del _pm_earlier, _pm_run_earlier
5. Nested Comprehensions: Matrices
A comprehension whose expression is itself a comprehension builds a 2D structure.
# 3x4 multiplication table table = [[r * c for c in range(1, 5)] for r in range(1, 4)] # [[1, 2, 3, 4], # [2, 4, 6, 8], # [3, 6, 9, 12]]
Read it outside-in: for each row r, build a list of r * c for each column c. The outer for runs slowest, and the inner runs fastest.
6. Flattening: Two fors, Same Line
When both fors sit at the top level, you get flattening, not nesting:
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] flat = [item for row in matrix for item in row] # [1, 2, 3, 4, 5, 6, 7, 8, 9]
Read the for parts from left to right, in the same order as you would write the nested loop:
# Equivalent imperative form flat = [] for row in matrix: # first for for item in row: # second for flat.append(item) # the expression on the left
setup added so this can run · re-runs 1 earlier block on this page first
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] # Earlier blocks on this page that this one builds on, re-run quietly first. _pm_earlier = [ 'matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]\n\nflat = [item for row in matrix for item in row]\n# [1, 2, 3, 4, 5, 6, 7, 8, 9]', ] def _pm_run_earlier(_srcs, _g): import sys as _s, io as _io, warnings as _w for _src in _srcs: _o, _e = _s.stdout, _s.stderr _s.stdout = _s.stderr = _io.StringIO() try: with _w.catch_warnings(): _w.simplefilter('ignore') exec(compile(_src, '<earlier block>', 'exec'), _g) except (Exception, SystemExit): pass finally: _s.stdout, _s.stderr = _o, _e if 'matplotlib.pyplot' in _s.modules: _s.modules['matplotlib.pyplot'].close('all') # their plots belong to their own blocks _pm_run_earlier(_pm_earlier, globals()) del _pm_earlier, _pm_run_earlier
The order of the fors is the same in both forms. The only difference is that the comprehension puts the body (item) at the start.
7. Set and Dict Comprehensions
Set comprehensions deduplicate as they build:
words = ["Hello", "world", "hello", "WORLD"] unique_lower = {w.lower() for w in words} # {'hello', 'world'}
Two operations (transform and remove duplicates) happen in a single expression. The imperative form needs a loop and an explicit set.add call.
Dict comprehensions need a key and a value separated by ::
prices = {"apple": 1.20, "banana": 0.40, "cherry": 3.00}
# Transform values
doubled = {k: v * 2 for k, v in prices.items()} # {'apple': 2.4, 'banana': 0.8, 'cherry': 6.0}
upper = {k.upper(): v for k, v in prices.items()} # {'APPLE': 1.2, ...}
# Swap keys and values
inverse = {v: k for k, v in prices.items()} # {1.2: 'apple', ...}
# Filter
cheap = {k: v for k, v in prices.items() if v < 1.0} # {'banana': 0.4}
# Build from scratch
squares = {n: n * n for n in range(5)} # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}{v: k for k, v in d.items()} is the standard Pythonic way to swap a dict's keys and values. It only works well if the values are unique and hashable. If two keys have the same value, you get one entry with no warning, and the last key wins.
8. Generator Expressions: Lazy by Default
Replace the [ ] with ( ) and you get a generator: an iterator that produces one value at a time, only when it is asked for the next one. It doesn't build the full sequence in memory.
# List: builds all 1,000,000 squares in memory total = sum([n * n for n in range(1_000_000)]) # Generator: makes one square at a time, and sum() adds each one as it arrives total = sum(n * n for n in range(1_000_000))
Both give the same answer, but the generator uses far less memory. When you pass the values to sum, any, all, min, max, "".join(...), or any function that iterates only once, the generator form is almost always the right choice. When the generator is the only argument, you can leave out its own parentheses:
# All of these work with no extra parentheses sum(n * n for n in range(100)) any(x < 0 for x in nums) all(s.isalpha() for s in words) max(len(line) for line in open("data.txt")) "\n".join(line.strip() for line in lines)
setup added so this can run · defines nums, words, lines
nums = [3, -1, 4, -1, 5] words = ["alpha", "beta", "gamma"] lines = ["alpha", "beta", "gamma"]
Generators are single-use. After you have iterated over one, it is empty, and you can't start again. If you need to go through the data twice, use a list.
9. Walrus in a Comprehension (3.8+)
The := operator assigns a value and returns it in one step. Inside a comprehension, it lets you compute something once and use it twice: once in the filter and once in the result.
import re lines = ["age=42", "broken", "name=Alice", "also-broken"] pairs = [(m.group(1), m.group(2)) for line in lines if (m := re.match(r"(\w+)=(\w+)", line))] # [('age', '42'), ('name', 'Alice')]
setup added so this can run · defines m
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) m = _AutoMock('m')
Without the walrus you'd either call re.match twice or precompute with a regular loop. Use it for expensive or complex expressions. For simple ones, the harder-to-read code isn't worth it.
10. When Not to Use a Comprehension
Comprehensions are not always the best choice. Use a regular for loop when:
- The logic spans more than one readable line. If your comprehension wraps onto three lines and is hard to read, the loop is clearer.
- You need multiple levels of
if/elifin the result expression. Chained conditional expressions inside a comprehension are very hard to read later. - The body has side effects: file writes,
print, mutations. Comprehensions are for producing a collection. Throwing away a list ofNones just to run side effects is a misuse:
python
# Bad: this builds a list of None values and then throws it away
[print(x) for x in items]
# Correct
for x in items:
print(x)
- You need exception handling per element. A
try/exceptdoesn't fit inside a comprehension.
The rule: a comprehension should describe one transformation in one line. When it grows longer than that, go back to a loop.
Common Mistakes
1. Confusing filter-if with conditional-expression-if
nums = [-2, -1, 0, 1, 2] # The position of the if matters. These two lines do different things. [n if n > 0 else 0 for n in nums] # [0, 0, 0, 1, 2] (changes items) [n for n in nums if n > 0] # [1, 2] (filters items)
Filter goes after the for. Conditional expression goes before the for, inside the result slot.
2. Using [ ] when you wanted ( )
# Builds a list of 10 million numbers in memory just to sum them total = sum([n for n in range(10_000_000)]) # A generator streams the numbers, so memory use stays small total = sum(n for n in range(10_000_000))
If the comprehension is feeding into a single-pass consumer (sum, any, all, max, join...), prefer the generator form.
3. Nesting too deep
# Don't do this result = [[x * y for x in row if x > 0] for row in matrix if sum(row) > 10]
setup added so this can run · defines matrix, y
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) matrix = ["alpha", "beta", "gamma"] y = _AutoMock('y')
That's two fors, two ifs, and an arithmetic expression in one line. Six months from now, nobody (including you) will be able to read it easily. Break it into a function with a regular nested loop and a clear name.
4. Name shadowing: mostly fine, with one quirk
In Python 3, comprehensions have their own scope. The loop variable doesn't exist outside the comprehension:
i = "outer" squares = [i * i for i in range(5)] print(i) # 'outer' (not 4)
This is good: the outer i doesn't change by surprise. But the iterable of the first for is evaluated in the enclosing scope:
i = [1, 2, 3] result = [x for x in i for x in range(x)] # the outer `i` is the source list
This is confusing, so don't do it. Don't use the same name for the source and the loop variable.
5. Comprehensions for side effects
# Wrong: this builds a useless list of None values [file.write(line) for line in lines] # Right for line in lines: file.write(line)
setup added so this can run · defines lines, file
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) lines = ["alpha", "beta", "gamma"] file = _AutoMock('file')
If you're not going to use the result, don't build one. Building a list costs time and memory.
🎯 Your Turn: Pivot Table
Write pivot_table(rows, key, value) that groups a list of dicts by one field and collects another field's values into a list.
people = [
{"dept": "eng", "salary": 90},
{"dept": "sales", "salary": 60},
{"dept": "eng", "salary": 110},
{"dept": "sales", "salary": 80},
{"dept": "eng", "salary": 95},
]
pivot_table(people, key="dept", value="salary")
# {'eng': [90, 110, 95], 'sales': [60, 80]} setup added so this can run · defines pivot_table
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) def pivot_table(*_a, **_kw): print('-> pivot_table() called') return _AutoMock('pivot_table()')
Skeleton:
def pivot_table(rows, key, value): # TODO 1: for each row, append row[value] under the bucket row[key] # TODO 2: return the resulting dict ...
Hint 1: Grouping pattern
You can't easily write this as a single dict comprehension, because you need to append to existing lists as you go. Either build an empty dict and usedict.setdefault(k, []).append(v) inside a regular for, or import collections.defaultdict(list) and you won't need setdefault.
Hint 2: setdefault vs defaultdict
d.setdefault("eng", []).append(90) returns the existing list for "eng" (creating an empty one if missing), then appends to it. defaultdict(list) does the same thing automatically the first time you index a missing key. It is slightly cleaner if you use this pattern often.
Show full solution
Version A: setdefault, no imports:
def pivot_table(rows, key, value): result = {} for row in rows: result.setdefault(row[key], []).append(row[value]) return result
Version B: defaultdict, slightly cleaner:
from collections import defaultdict def pivot_table(rows, key, value): result = defaultdict(list) for row in rows: result[row[key]].append(row[value]) return dict(result) # turn it back into a plain dict for the caller
Version C: only comprehensions, using a set of the unique keys:
def pivot_table(rows, key, value): return {k: [r[value] for r in rows if r[key] == k] for k in {r[key] for r in rows}}
Version C uses only comprehensions, but it is O(n × k): for every unique key, it goes through the whole row list again. Versions A and B are O(n). On 10 rows it doesn't matter, but on 10 million it does. Also, a set has no fixed order, so Version C can return the keys in a different order from run to run (the dicts are still equal). With comprehensions, you often have to choose between short, clear and fast code.
Test it:
people = [
{"dept": "eng", "salary": 90},
{"dept": "sales", "salary": 60},
{"dept": "eng", "salary": 110},
]
print(pivot_table(people, "dept", "salary"))
# {'eng': [90, 110], 'sales': [60]}setup added so this can run · re-runs 3 earlier blocks on this page first
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) def pivot_table(*_a, **_kw): print('-> pivot_table() called') return _AutoMock('pivot_table()') # Earlier blocks on this page that this one builds on, re-run quietly first. try: from collections import defaultdict except Exception: pass _pm_earlier = [ 'def pivot_table(rows, key, value):\n result = {}\n for row in rows:\n result.setdefault(row[key], []).append(row[value])\n return result', 'from collections import defaultdict\n\ndef pivot_table(rows, key, value):\n result = defaultdict(list)\n for row in rows:\n result[row[key]].append(row[value])\n return dict(result) # turn it back into a plain dict for the caller', 'def pivot_table(rows, key, value):\n return {k: [r[value] for r in rows if r[key] == k]\n for k in {r[key] for r in rows}}', ] def _pm_run_earlier(_srcs, _g): import sys as _s, io as _io, warnings as _w for _src in _srcs: _o, _e = _s.stdout, _s.stderr _s.stdout = _s.stderr = _io.StringIO() try: with _w.catch_warnings(): _w.simplefilter('ignore') exec(compile(_src, '<earlier block>', 'exec'), _g) except (Exception, SystemExit): pass finally: _s.stdout, _s.stderr = _o, _e if 'matplotlib.pyplot' in _s.modules: _s.modules['matplotlib.pyplot'].close('all') # their plots belong to their own blocks _pm_run_earlier(_pm_earlier, globals()) del _pm_earlier, _pm_run_earlier
What You Learned
- Four forms (list
[ ], set{ }, dict{k: v}, generator( )) use the same syntax to build different containers. - Filter
ifgoes after thefor, but conditional-expressionifgoes before theforinside the result slot. - Nested comprehensions build matrices, and flat double-
forflattens them. - Dict comprehensions transform, invert, or rebuild dicts in one line.
- Generator expressions are lazy. Use them whenever a single-pass consumer (
sum,any,all,join...) is on the other end. - Stop using a comprehension when it no longer fits on one readable line. Clear code matters more than short code.
- Don't use comprehensions for side effects. Use a
forloop.
Next: Decorators: functions that wrap other functions to add the same behaviour (like logging or timing) to many functions.
Practice this
on practicepython.inShort exercises that run in your browser and tell you what your code did, as well as whether a test passed.
Found this useful? Share it with a friend who’s learning Python.