Performance: Profile, Then Optimise

Most Python performance work goes wrong before the first edit, because someone guessed where the slow part was. Eyeballing a 2000-line module and "knowing" the bottleneck is in parse_row is how you spend an afternoon rewriting parse_row in C and discover the real cost was a single SELECT * two functions up.

This lesson is the discipline of measuring first. The tools (timeit, cProfile, pstats, tracemalloc, dis), the wins that actually move the needle, and the wrong tools — the cases where pure Python is the wrong substrate and you should reach for numpy, numba, or multiprocessing.

1. The Rule: Measure First

"Premature optimisation is the root of all evil." — Knuth (full quote includes "about 97% of the time").

The 3% that matters is real. The other 97% is where engineers burn weeks rewriting code that wasn't slow to begin with. The only way to know which 3% you're in is to profile.

Three questions, in order, before you change a single line:

1. Is it actually slow? Wall-clock the workload. If a nightly job takes 8s and a human notices nothing, leave it.
2. Where is the time going? Profile. Find the function (or line) responsible for the bulk of the time.
3. What's the algorithm? A big-O improvement beats a 2× micro-optimisation every time.

Only after those three do you reach for the timing-loop refinements in Section 6.

2. timeit — Microbenchmarking a Snippet

timeit runs a small piece of code many times and reports the best time. The repetition smooths out OS jitter, GC pauses, and the cold-cache penalty of the first run.

import timeit

# Compare two ways to build a list of squares
t_loop = timeit.timeit("out = []\nfor x in range(1000):\n    out.append(x*x)",
                       number=10_000)
t_comp = timeit.timeit("[x*x for x in range(1000)]", number=10_000)

print(f"loop:        {t_loop:.3f}s")
print(f"comp:        {t_comp:.3f}s")
print(f"speedup:     {t_loop / t_comp:.2f}x")

In IPython or a Jupyter notebook, the same job is one line:

%timeit [x*x for x in range(1000)]
# 18.4 µs ± 120 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

%timeit auto-picks the loop count and reports mean + standard deviation — the right default for casual measurement. Reach for timeit.timeit (or timeit.repeat for a stable minimum) when you want to script the comparison.

One pitfall: timeit runs your snippet in its own namespace. Pass setup code via the setup= argument, not by relying on globals:

timeit.timeit("sorted(data)", setup="import random; data = [random.random() for _ in range(1000)]",
              number=1000)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

timeit = _AutoMock('timeit')

3. time.perf_counter() — Ad-Hoc Timing

For a one-off "how long did this block take," perf_counter from the datetime toolkit gives you the highest-resolution monotonic clock the OS provides.

import time

start = time.perf_counter()
result = expensive_pipeline(data)
elapsed = time.perf_counter() - start
print(f"pipeline: {elapsed:.3f}s")

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def expensive_pipeline(*_a, **_kw):
    print('-> expensive_pipeline() called')
    return _AutoMock('expensive_pipeline()')
data = _AutoMock('data')

Wrap it in a context manager and you've got a reusable timer:

from contextlib import contextmanager

@contextmanager
def timer(label):
    start = time.perf_counter()
    try:
        yield
    finally:
        print(f"{label}: {time.perf_counter() - start:.3f}s")

with timer("load"):
    rows = load_csv("big.csv")
with timer("transform"):
    out = transform(rows)

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def load_csv(*_a, **_kw):
    print('-> load_csv() called')
    return _AutoMock('load_csv()')
def transform(*_a, **_kw):
    print('-> transform() called')
    return _AutoMock('transform()')
time = _AutoMock('time')

perf_counter is for timing. Never use time.time() for measuring durations — it can jump backwards when NTP adjusts the wall clock.

4. cProfile — Function-Level Profiling

When the workload is bigger than a snippet, cProfile instruments every function call and tells you where the time goes. The cheapest path is the command line:

python -m cProfile -s tottime script.py

-s tottime sorts by total time spent inside the function (excluding sub-calls). Reading the output:

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   100000    1.842    0.000    2.103    0.000 parser.py:42(parse_row)
        1    0.215    0.215    2.318    2.318 script.py:10(run)
   100000    0.261    0.000    0.261    0.000 {method 'split' of 'str' objects}

• ncalls — how many times this function was called.
• tottime — seconds spent inside this function, not counting sub-calls. The hottest leaf is the one with the biggest tottime.
• cumtime — seconds spent in this function plus everything it called. The root of a slow tree has the biggest cumtime.

Rule of thumb: sort by tottime to find expensive primitives; sort by cumtime (-s cumulative) to find expensive call chains. Both views are useful.

For programmatic use, dump a profile to disk and analyse it later:

import cProfile
cProfile.run("run_pipeline()", filename="run.prof")

5. pstats — Picking the Profile Apart

pstats reads a .prof file and gives you a queryable interface — filter by filename, restrict to top N, sort by different columns without re-running.

import pstats

p = pstats.Stats("run.prof")
p.strip_dirs()                    # drop noisy absolute paths
p.sort_stats("cumulative")
p.print_stats(20)                 # top 20 by cumulative time

p.sort_stats("tottime")
p.print_stats("parser", 10)       # top 10 results whose path matches 'parser'

For deeper exploration, point a flame-graph viewer at the file: snakeviz run.prof opens an interactive browser view (third-party, pip install snakeviz) that makes the call tree visually obvious in seconds.

6. Line-Level Profiling — line_profiler

cProfile tells you which function is slow. When the function is 80 lines long, you still need to know which line. That's line_profiler (third-party):

pip install line_profiler

Decorate the function with @profile (the tool injects it) and run via kernprof:

@profile
def parse_row(line):
    parts = line.split(",")           # is this the slow line?
    parts = [p.strip() for p in parts]
    return {"name": parts[0], "value": int(parts[1])}

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

profile = _AutoMock('profile')

kernprof -l -v script.py

Output is per-line: % Time, Hits, Time. Once you've seen which line is eating the budget, you can fix exactly that line.

7. Memory Profiling — tracemalloc and memory_profiler

tracemalloc is in the standard library — no install needed. It tracks every allocation made by the Python heap.

import tracemalloc

tracemalloc.start()
result = build_huge_thing()
current, peak = tracemalloc.get_traced_memory()
print(f"current: {current / 1e6:.1f} MB, peak: {peak / 1e6:.1f} MB")

snapshot = tracemalloc.take_snapshot()
for stat in snapshot.statistics("lineno")[:10]:
    print(stat)                       # top 10 lines by allocation

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def build_huge_thing(*_a, **_kw):
    print('-> build_huge_thing() called')
    return _AutoMock('build_huge_thing()')

For decorated function-level memory profiling, memory_profiler (third-party, pip install memory_profiler) gives you a @profile decorator and a mprof run script.py CLI that produces a memory-over-time chart.

Memory bugs (leaks, runaway growth) usually show as growing peak over repeated runs. CPU profiling won't catch them — you need the allocation view.

8. dis.dis() — Reading the Bytecode

When two pieces of code look equivalent and timeit says they aren't, the bytecode shows you why.

import dis

def f(xs):
    out = []
    for x in xs:
        out.append(x * x)
    return out

def g(xs):
    return [x * x for x in xs]

dis.dis(f)
dis.dis(g)

The list-comp g compiles to a tighter loop using a LIST_APPEND opcode instead of LOAD_METHOD + CALL_METHOD on out.append. That single change is most of why comprehensions outperform manual for/append — and you'll only see it if you read the bytecode.

You don't read dis daily. You read it the moment a colleague claims "X is faster than Y" and you want to know whether it's actually true on your Python version.

9. Algorithms Beat Micro-Optimisations — Always

A function with O(n²) behaviour and a 2× constant-factor speedup is still O(n²). The real win is to change the algorithm.

# O(n²) — for each item, scan the rest of the list
def count_pairs_slow(items, target):
    n, total = len(items), 0
    for i in range(n):
        for j in range(i + 1, n):
            if items[i] + items[j] == target:
                total += 1
    return total

# O(n) — single pass, look up the complement in a dict
def count_pairs_fast(items, target):
    seen, total = {}, 0
    for x in items:
        total += seen.get(target - x, 0)
        seen[x] = seen.get(x, 0) + 1
    return total

For n = 10_000, the slow version is ~50 million comparisons; the fast version is 10,000 dict lookups. No amount of Cython on the slow version closes that gap.

The hierarchy of speedups, from most to least leverage:

1. Drop a whole call (memoise, cache, skip).
2. Replace O(n²) with O(n log n) or O(n).
3. Switch data structure (list → set/dict for membership).
4. Vectorise with numpy for numeric loops.
5. Move hot loops to C (numba, cython, native extension).
6. Micro-optimise pure-Python idioms (the Section 10 list).

You attack from the top. Don't fight a Section 10 battle if a Section 9 win is sitting upstream.

10. The Common 10× Wins in Pure Python

Once you've fixed the algorithm and you still want more:

Local variable lookups beat globals. Local names are an array index; globals are a dict lookup. Inside a hot loop, cache them:

def hot_loop(xs):
    _len = len               # rebind builtins as locals
    _append = result.append
    result = []
    for x in xs:
        if _len(x) > 3:
            _append(x)
    return result

Comprehensions beat manual for/append. Section 8 explains why at the bytecode level.

set/dict membership is O(1); list is O(n).

banned = {"alice", "bob", "carol"}   # set: O(1) lookup
if name in banned: ...

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

name = _AutoMock('name')

str.join(list) beats += on string in a loop. Strings are immutable; += allocates a new string each iteration — quadratic over a long loop.

# WRONG — O(n²)
out = ""
for chunk in chunks:
    out += chunk

# RIGHT — O(n)
out = "".join(chunks)

chunks = ["alpha", "beta", "gamma"]

Generators beat lists for one-pass iteration. No allocation of the intermediate. See generators.

functools.lru_cache for repeated calls with the same args:

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    return n if n < 2 else fib(n-1) + fib(n-2)

Memoisation turns exponential recursion into linear lookup. Only works for hashable arguments — pass a list and you get TypeError: unhashable type.

11. The Wrong-Tool Tells — When to Leave Pure Python

Some workloads are not Python-shaped. Symptoms:

• A hot inner loop over millions of numbers. Pure Python's per-iteration overhead is ~50ns. numpy vectorisation does the same work in C at ~1ns per element. 50× speedup, often more.

python # SLOW — Python loop total = sum(x * x for x in data) # 10M items: ~1.5s # FAST — numpy vectorisation import numpy as np arr = np.asarray(data) total = (arr * arr).sum() # 10M items: ~25ms

• Per-row iteration of a DataFrame. df.iterrows() is the cardinal sin of pandas performance. Use vectorised column operations (df["price"] * df["qty"]), df.apply(..., axis=1) for genuinely row-shaped work, or convert to numpy and process the array.

• A CPU-bound function you want to run in parallel. Threading buys you nothing — the GIL serialises CPU-bound Python. Reach for multiprocessing or concurrent.futures.ProcessPoolExecutor. See multiprocessing.

• Numerical kernels that even numpy can't vectorise. numba's @jit compiles a Python function to LLVM in seconds; cython lets you add type annotations and compile to a C extension. For a hand-written tight loop, either gives you near-C speed.

The pattern: pure Python is fast enough until it isn't, and when it isn't, the answer is usually "stop being pure Python."

12. Other Levers Worth Knowing

__slots__ declares a fixed set of attributes on a class. Saves the per-instance __dict__. For a class you'll create millions of, the memory saving is large:

class Point:
    __slots__ = ("x", "y")
    def __init__(self, x, y):
        self.x, self.y = x, y

Trade-offs: no dynamic attributes, no per-instance __dict__, and multiple inheritance gets fiddly. Use when you've measured a memory problem, not by default.

weakref lets you hold a reference without preventing garbage collection — useful for caches that should release entries when nothing else holds them.

Async vs multiprocessing — workload-dependent. I/O-bound (network, disk, DB) → asyncio. CPU-bound → multiprocessing. Mixing them up is the classic Python performance mistake: threading or async around a CPU-bound function is slower than running it sequentially, because of overhead and the GIL.

Common Mistakes

1. Optimising without profiling. 95% of your code doesn't matter. The hot 5% might not be where you think. Without a profile, you're rewriting on vibes.

2. Micro-optimising while an O(n²) algorithm sits upstream. The 2% you save on comprehensions is invisible next to the 99% you save by changing the algorithm. Fix the big-O first.

3. Premature numpy. For 100-element lists, pure Python is faster than numpy — array construction has fixed overhead that swamps the benefit on small data. numpy wins from thousands of elements upwards.

4. Threading CPU-bound code. The GIL serialises CPU-bound Python regardless of thread count. CPU-bound parallelism needs multiprocessing, numba, or a native extension that releases the GIL.

5. @lru_cache on a function with mutable arguments.

@lru_cache(maxsize=128)
def search(items, target):              # items is a list — unhashable
    ...

search([1, 2, 3], 2)                    # TypeError: unhashable type: 'list'

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def lru_cache(*_a, **_kw):
    print('-> lru_cache() called')
    return _AutoMock('lru_cache()')

Either pass a tuple instead, or frozenset for "membership without order," or don't cache.

6. print-based "did this run" debugging in a hot loop. print flushes to stdout, which is slow — easily 100µs per call. In a million-iteration loop that's 100 seconds of noise on top of the real work. Use logging with a level filter, or count + report once after the loop.

🎯 Your Turn — From O(n²) to O(n)

You're given a slow count_pairs(items, target) that uses nested loops. Your job:

1. Write a fast version using a dict to look up complements in one pass.
2. Benchmark both with timeit for n = 10_000 random integers.
3. Confirm they agree on the answer.

import random
from timeit import timeit

def count_pairs_slow(items, target):
    """O(n^2) — nested-loop reference implementation."""
    n, total = len(items), 0
    for i in range(n):
        for j in range(i + 1, n):
            if items[i] + items[j] == target:
                total += 1
    return total

def count_pairs_fast(items, target):
    # TODO 1: walk items once
    # TODO 2: for each x, count how many earlier items equal (target - x)
    # TODO 3: track counts of items seen so far in a dict
    ...

# Smoke test
data = [random.randint(0, 50) for _ in range(200)]
assert count_pairs_slow(data, 50) == count_pairs_fast(data, 50)

# Benchmark — 10_000 items
big = [random.randint(0, 1000) for _ in range(10_000)]
t_slow = timeit(lambda: count_pairs_slow(big, 500), number=1)
t_fast = timeit(lambda: count_pairs_fast(big, 500), number=1)
print(f"slow: {t_slow:.3f}s   fast: {t_fast:.3f}s   speedup: {t_slow/t_fast:.0f}x")

import random
from timeit import timeit
from collections import Counter

def count_pairs_slow(items, target):
    n, total = len(items), 0
    for i in range(n):
        for j in range(i + 1, n):
            if items[i] + items[j] == target:
                total += 1
    return total

def count_pairs_fast(items, target):
    """O(n) — for each x, the number of new pairs is the count of (target - x) seen so far."""
    seen = Counter()
    total = 0
    for x in items:
        total += seen[target - x]       # pair x with every earlier occurrence of the complement
        seen[x] += 1                    # then register x for future pairings
    return total

# Correctness — match the reference on a small case
random.seed(0)
small = [random.randint(0, 50) for _ in range(200)]
assert count_pairs_slow(small, 50) == count_pairs_fast(small, 50)

# Benchmark
big = [random.randint(0, 1000) for _ in range(10_000)]
t_slow = timeit(lambda: count_pairs_slow(big, 500), number=1)
t_fast = timeit(lambda: count_pairs_fast(big, 500), number=1)
print(f"slow: {t_slow:.3f}s   fast: {t_fast:.3f}s   speedup: {t_slow/t_fast:.0f}x")

# Typical output on a laptop:
# slow: 3.142s   fast: 0.002s   speedup: 1571x

What You Learned

• Profile first. timeit for snippets, cProfile + pstats for whole scripts, line_profiler for hot lines, tracemalloc for memory.
• time.perf_counter() for ad-hoc timing; never time.time() (NTP can jump backwards).
• dis.dis(fn) when two equivalent-looking pieces of code disagree on speed.
• Algorithms first. O(n²) → O(n) with a dict beats every micro-optimisation downstream.
• Common 10× wins: comprehensions over for/append, set/dict membership over list, str.join over +=, lru_cache for repeated calls, generators for one-pass streams.
• Wrong-tool tells: per-row DataFrame iteration, Python loops over millions of numbers, threading CPU-bound code. Reach for numpy, multiprocessing, numba, or cython.
• __slots__ for memory on classes you'll instantiate millions of. weakref for caches that shouldn't pin objects.
• Async vs multiprocessing is workload-dependent. I/O-bound → async. CPU-bound → multiprocessing.

Next: Design Patterns That Actually Fit Python — which GoF patterns survive in a language with first-class functions, and which dissolve into a one-liner.