Python for Data Analysis

Even before NumPy and pandas, Python's standard library can do a lot of analysis work. This lesson covers the idioms that data science code uses most (comprehensions, zip, Counter, generators) and the file formats you'll meet from the first day. By the end, you'll have written a small analysis script with no third-party imports.

1. Comprehensions Are Everywhere

A data analysis script transforms one collection into another. Comprehensions express that in one line. You met them in Comprehensions. Here's how DS code uses them.

prices = [9.99, 14.50, 7.25, 22.00, 3.99]

# Transform: add VAT to each price
with_vat = [round(p * 1.2, 2) for p in prices]
print(with_vat)                      # [11.99, 17.4, 8.7, 26.4, 4.79]

# Filter: keep only the items above £10
expensive = [p for p in prices if p > 10]
print(expensive)                     # [14.5, 22.0]

# Map to a dict
sku_to_price = {f"SKU-{i:03d}": p for i, p in enumerate(prices, start=1)}
print(sku_to_price)
# {'SKU-001': 9.99, 'SKU-002': 14.5, 'SKU-003': 7.25, 'SKU-004': 22.0, 'SKU-005': 3.99}

# Aggregate: the total revenue if we sold one of each item
total = sum(p for p in prices)
print(f"Total: £{total:.2f}")        # Total: £57.73

Memorise these forms: [expr for x in iter if cond], {k: v for ...}, sum(expr for ...). Until you start using pandas, they handle most cleanup tasks.

2. enumerate and zip: The Pair You Will Use Most

enumerate gives you (index, value) as you loop. zip loops over several iterables together, one item from each at a time. Between them, they replace most range(len(...))-style code.

products = ["Coffee", "Tea", "Cocoa"]
prices   = [3.50, 2.75, 4.00]
units    = [120,  340,   85]

# enumerate: number each line of output
for i, name in enumerate(products, start=1):
    print(f"{i}. {name}")

# zip: pair two columns together
revenue = {name: price * qty for name, price, qty in zip(products, prices, units)}
print(revenue)
# {'Coffee': 420.0, 'Tea': 935.0, 'Cocoa': 340.0}

zip stops at the end of the shortest iterable. Pass strict=True (Python 3.10+) if you want it to raise an error when the lengths are different.

3. collections.Counter: Counting in One Line

Whenever you'd write if key in d: d[key] += 1 else: d[key] = 1, use Counter instead.

from collections import Counter

orders = ["coffee", "tea", "coffee", "coffee", "cocoa", "tea", "coffee"]

tally = Counter(orders)
print(tally)                         # Counter({'coffee': 4, 'tea': 2, 'cocoa': 1})
print(tally.most_common(2))          # [('coffee', 4), ('tea', 2)]
print(tally["coffee"])               # 4
print(tally["water"])                # 0  (a missing key gives 0 instead of raising KeyError)

most_common() gives you the "top N categories" in one line, and you will need that very often. Counter also supports arithmetic: Counter(a) + Counter(b) adds two sets of counts together.

4. Loading CSV with the Standard Library

Most data for analysis comes as CSV. The standard library's csv module reads it, with nothing extra to install. (Recap and details in CSV & JSON.)

import csv
from io import StringIO

# A tiny CSV inline so we don't need a file
csv_text = """product,price,quantity
Coffee,3.50,120
Tea,2.75,340
Cocoa,4.00,85
"""

# DictReader gives each row as a dict of {column: value}
reader = csv.DictReader(StringIO(csv_text))
rows = list(reader)

for row in rows:
    print(row)
# {'product': 'Coffee', 'price': '3.50', 'quantity': '120'}
# ...

Note that the values are strings, because CSV has no types. You convert them yourself:

typed = [
    {"product": r["product"], "price": float(r["price"]), "quantity": int(r["quantity"])}
    for r in rows
]

rows = [{"product": "alpha", "price": "3", "quantity": "3"}, {"product": "beta", "price": "6", "quantity": "6"}, {"product": "gamma", "price": "9", "quantity": "9"}]

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import csv
    from io import StringIO
except Exception:
    pass
_pm_earlier = [
    'import csv\nfrom io import StringIO\n\n# A tiny CSV inline so we don\'t need a file\ncsv_text = """product,price,quantity\nCoffee,3.50,120\nTea,2.75,340\nCocoa,4.00,85\n"""\n\n# DictReader gives each row as a dict of {column: value}\nreader = csv.DictReader(StringIO(csv_text))\nrows = list(reader)\n\nfor row in rows:\n    print(row)\n# {\'product\': \'Coffee\', \'price\': \'3.50\', \'quantity\': \'120\'}\n# ...',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Converting types when you load the file is one of the routine jobs that pandas does for you. We'll meet it in Pandas.

5. Loading JSON

JSON maps directly to Python dict / list / scalars. No conversion needed.

import json

json_text = '''
{
    "store": "Camden",
    "date":  "2026-05-12",
    "items": [
        {"sku": "C1", "qty": 4},
        {"sku": "T2", "qty": 7}
    ]
}
'''

data = json.loads(json_text)         # the JSON string becomes a Python object
print(data["store"])                 # Camden
print(data["items"][0]["qty"])       # 4

# Write it back out, pretty-printed
print(json.dumps(data, indent=2))

For files, use json.load(f) / json.dump(obj, f). For working with file paths, see File I/O.

6. The Mental Shift: Row-by-Row → Vectorised

This is the most important idea in this lesson. Here is the same task written in three ways, using two different ways of thinking:

# Row by row: natural in Python, but slow on large data
prices = [3.50, 2.75, 4.00, 9.99, 14.50]
with_vat = []
for p in prices:
    with_vat.append(p * 1.2)

# Comprehension: the same idea in less code
with_vat = [p * 1.2 for p in prices]

# Vectorised: what NumPy lets you write (next lesson)
import numpy as np
arr = np.array(prices)
with_vat = arr * 1.2                 # one operation on the whole array

The first two loop in Python, which is slow. The third runs the loop in compiled C, which is fast. The change in thinking, "I am working on a whole column, not on each row", is the main difference between data science code and general Python code.

7. Mini-Example A: Word Count from Strings

from collections import Counter

reviews = [
    "great coffee fast service",
    "coffee was cold service slow",
    "great location great coffee",
]

words = [w for review in reviews for w in review.split()]
top = Counter(words).most_common(3)
print(top)
# [('great', 3), ('coffee', 3), ('service', 2)]

The nested comprehension turns the list of lists into one flat list of words. Counter counts them, and most_common sorts them by count.

8. Mini-Example B: Per-Category Averages

import csv
from io import StringIO
from collections import defaultdict
from statistics import mean

csv_text = """category,price
books,12.50
books,8.99
games,49.99
games,29.99
books,15.00
games,39.99
"""

reader = csv.DictReader(StringIO(csv_text))

by_cat = defaultdict(list)
for row in reader:
    by_cat[row["category"]].append(float(row["price"]))

avg_by_cat = {cat: round(mean(prices), 2) for cat, prices in by_cat.items()}
print(avg_by_cat)
# {'books': 12.16, 'games': 39.99}

This is groupby followed by mean, written by hand. Pandas does it in one line, but writing it yourself once shows you exactly what that line does.

9. Generators: Streaming Big Files

A list holds every row in memory. A generator yields one row at a time. For a 5 GB CSV, that decides whether the job fits in memory or not.

def stream_prices(rows):
    """Yield (sku, price) pairs without building a list."""
    for row in rows:
        yield row["sku"], float(row["price"])

# Loop over the rows without ever building the full list in memory
# for sku, price in stream_prices(reader):
#     ...

The pattern: build a chain of generators, then call sum, max or min on the last one. Memory use stays the same, whatever the file size.

10. Jupyter Notebooks: A One-Paragraph Intro

A Jupyter notebook is a .ipynb file containing alternating cells of code and markdown. You can run cells in any order, and each cell's output (numbers, charts, tables) appears directly below it. Notebooks are the most common tool for exploring data, because you can mix code, plots and notes, and variables stay in memory between cells, so you don't have to rerun everything. For finished pipelines, you move the code back to .py files. Use both: notebooks for exploration, scripts for production.

Common Mistakes

• Writing for i in range(len(x)) when you meant enumerate(x). Try enumerate or zip first.
• Nested for-loops that a comprehension would handle. Two levels is the limit for readable code. For three or more, write a normal loop.
• Loading a giant CSV into a list. If you only need a sum or a max, stream it with a generator.
• Forgetting CSV values are strings. row["quantity"] > 10 compares a string to an int, which raises a TypeError. Always convert types after DictReader.
• Reinventing Counter. If you find yourself writing d[k] = d.get(k, 0) + 1, stop and import Counter.

🎯 Your Turn: top_categories

Write a function that takes CSV-like rows and returns the top N categories by total value. Use only the standard library.

Skeleton:

from collections import defaultdict

def top_categories(rows, key_col, value_col, n=5):
    """Return [(category, total), ...] sorted desc, top N."""
    # TODO 1: accumulate totals per category
    # TODO 2: sort by total descending and return the top n
    ...

# Test
rows = [
    {"category": "books", "amount": "12.50"},
    {"category": "games", "amount": "49.99"},
    {"category": "books", "amount": "8.99"},
    {"category": "music", "amount": "5.00"},
    {"category": "games", "amount": "29.99"},
    {"category": "books", "amount": "15.00"},
]
print(top_categories(rows, "category", "amount", n=2))
# Expected: [('games', 79.98), ('books', 36.49)]

from collections import defaultdict

def top_categories(rows, key_col, value_col, n=5):
    totals = defaultdict(float)
    for row in rows:
        totals[row[key_col]] += float(row[value_col])
    ranked = sorted(totals.items(), key=lambda kv: kv[1], reverse=True)
    return [(cat, round(total, 2)) for cat, total in ranked[:n]]

rows = [
    {"category": "books", "amount": "12.50"},
    {"category": "games", "amount": "49.99"},
    {"category": "books", "amount": "8.99"},
    {"category": "music", "amount": "5.00"},
    {"category": "games", "amount": "29.99"},
    {"category": "books", "amount": "15.00"},
]
print(top_categories(rows, "category", "amount", n=2))
# [('games', 79.98), ('books', 36.49)]

What You Learned

• Comprehensions, enumerate, zip, Counter cover most preprocessing before pandas.
• csv.DictReader and json.loads load the two formats you'll meet every day.
• CSV values arrive as strings, so you must convert the types yourself.
• Vectorised thinking means one operation on the whole column. NumPy, in the next lesson, makes this possible.
• Generators keep memory use low and constant for huge files.
• Jupyter is for exploration, and scripts are for production.

Next: NumPy: Working with Numbers: replace the for-loop with operations on whole arrays.