Pandas: Working with Tables
1 · The lesson
readNumPy is great at numbers; real datasets have column names, mixed types, missing values, and indexes. Pandas adds all four on top of NumPy. If NumPy is the engine, pandas is the chassis — it's what you'll spend 70% of your DS career inside.
Think of pandas as "Excel for Python, scriptable". A DataFrame is a spreadsheet; a Series is a single column.
1. The Two Types
import pandas as pd # Series — 1D, like a NumPy array with a labelled index ages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"]) print(ages) # Alice 25 # Bob 32 # Carol 47 # Dan 19 # dtype: int64 # DataFrame — 2D table of Series sharing one index df = pd.DataFrame({ "name": ["Alice", "Bob", "Carol", "Dan"], "age": [25, 32, 47, 19], "city": ["London", "Paris", "Berlin", "London"], "salary": [55_000, 72_000, 91_000, 38_000], }) print(df)
A DataFrame is a dict of columns under the hood. Every column is a Series; every column shares the same row index (default: 0, 1, 2, ...).
The convention is import pandas as pd. Always.
2. Loading Data
One-line loaders for every format you'll meet:
# CSV — the workhorse df = pd.read_csv("sales.csv") # JSON df = pd.read_json("orders.json") # Excel — needs `openpyxl` installed df = pd.read_excel("report.xlsx", sheet_name="Q1") # Parquet — the format for big analytical data df = pd.read_parquet("events.parquet") # Even straight from a URL df = pd.read_csv("https://example.com/data.csv")
setup added so this can run · defines pd
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) pd = _AutoMock('pd')
For this lesson, we'll build DataFrames inline so every example runs immediately:
import pandas as pd df = pd.DataFrame({ "name": ["Alice", "Bob", "Carol", "Dan", "Eve"], "age": [25, 32, 47, 19, 28], "city": ["London", "Paris", "Berlin", "London", "Paris"], "salary": [55_000, 72_000, 91_000, 38_000, 64_000], })
3. Inspecting a DataFrame
The four commands you run every time you load a new dataset:
print(df.shape) # (5, 4) print(df.head(3)) # first 3 rows print(df.tail(2)) # last 2 rows print(df.info()) # column names, types, non-null counts print(df.describe()) # summary stats for numeric columns
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
describe() is the lazy DS's best friend — count, mean, std, min, 25%, 50%, 75%, max for every numeric column in one call. For categoricals, use describe(include="object").
print(df.dtypes) # name object # age int64 # city object # salary int64 print(df.columns.tolist()) # ['name', 'age', 'city', 'salary']
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
object dtype usually means strings.
4. Selecting Columns
# One column → Series print(df["name"]) # Multiple columns → DataFrame (note the double brackets) print(df[["name", "salary"]]) # Dot access — works only if the name has no spaces and isn't a reserved word print(df.age)
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
Stick to df["col"] and df[["col1", "col2"]]. The dot form looks tidy but breaks on column names like "first name" and silently shadows methods like .shape.
5. Filtering Rows — Boolean Masking
Identical to NumPy. The mask is a Series of booleans:
# One condition print(df[df["age"] > 30]) # Combined — use & and |, parenthesise every clause print(df[(df["age"] > 25) & (df["city"] == "London")]) # Membership print(df[df["city"].isin(["London", "Berlin"])]) # String methods print(df[df["name"].str.startswith("A")])
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
The .str accessor exposes Python string methods over a whole column. Same idea for .dt on date columns.
6. .loc vs .iloc
This is the chunk of pandas that confuses everyone once. Memorise it.
.loc[rows, cols]— label-based. Uses the row's index value and the column's name..iloc[rows, cols]— integer-position based. Uses 0-based integers like a Python list.
# Label-based — names and labels print(df.loc[0, "name"]) # Alice print(df.loc[0:2, "name"]) # rows 0..2 INCLUSIVE, name column print(df.loc[df["age"] > 30, ["name", "salary"]]) # Integer-position based print(df.iloc[0, 0]) # cell at row 0, col 0 → 'Alice' print(df.iloc[0:2, 0:3]) # rows 0..1 EXCLUSIVE, cols 0..2 EXCLUSIVE print(df.iloc[-1]) # last row
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
Two booby-traps worth flagging:
1. df.loc[0:2] includes row 2. df.iloc[0:2] does not. (Labels include the endpoint; integer slices don't.)
2. After filtering or sorting, the index is no longer 0, 1, 2, .... Reset it with df.reset_index(drop=True) if you need clean integers back.
7. Adding and Changing Columns
# New column from arithmetic — vectorised, NumPy-backed df["bonus"] = df["salary"] * 0.10 # Conditional column df["senior"] = df["age"] >= 30 # From a function applied to one column df["initial"] = df["name"].str[0] # From multiple columns df["total_comp"] = df["salary"] + df["bonus"] print(df)
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
For row-by-row logic over multiple columns, df.apply(func, axis=1) exists — but it's slow. Reach for vectorised expressions first, np.where second, apply last.
import numpy as np df["tier"] = np.where(df["salary"] >= 70_000, "high", "standard")
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
8. Grouping — The Killer Feature
groupby is what pulls hands away from Excel forever. It splits the DataFrame by a column, applies an aggregation, and combines the results.
# Mean salary per city print(df.groupby("city")["salary"].mean()) # city # Berlin 91000.0 # London 46500.0 # Paris 68000.0 # Multiple aggregations on multiple columns print(df.groupby("city").agg( avg_salary=("salary", "mean"), max_age=("age", "max"), headcount=("name", "count"), )) # avg_salary max_age headcount # city # Berlin 91000.0 47 1 # London 46500.0 25 2 # Paris 68000.0 32 2
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
The pattern — split, apply, combine — generalises to any aggregation function (built-in or custom) and any number of grouping columns: df.groupby(["city", "tier"]).
9. Sorting
df.sort_values("salary", ascending=False) # by one column df.sort_values(["city", "salary"], ascending=[True, False]) # multi-key df.nlargest(3, "salary") # top-N shortcut df.nsmallest(2, "age")
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
nlargest / nsmallest are faster than sort_values(...).head(n) when N is small.
10. Saving
df.to_csv("out.csv", index=False) # almost always pass index=False df.to_json("out.json", orient="records") df.to_parquet("out.parquet") # for analytical pipelines df.to_excel("out.xlsx", index=False)
setup added so this can run · defines df
# Lightweight mock for objects whose attributes/methods aren't critical class _AutoMock: def __init__(self, name='mock'): self._name = name def __getattr__(self, k): return _AutoMock(self._name + '.' + k) def __call__(self, *a, **kw): print('-> ' + self._name + '() called') return _AutoMock(self._name + '()') def __repr__(self): return '<mock ' + self._name + '>' def __str__(self): return '<mock ' + self._name + '>' def __bool__(self): return True def __iter__(self): return iter([]) def __len__(self): return 0 def __getitem__(self, k): return _AutoMock(self._name + '[...]') def __setitem__(self, k, v): pass def __enter__(self): return self def __exit__(self, *a): return False async def __aenter__(self): return self async def __aexit__(self, *a): return False def __add__(self, o): return self def __radd__(self, o): return self def __sub__(self, o): return self def __mul__(self, o): return self def __rmul__(self, o): return self def __truediv__(self, o): return self def __eq__(self, o): return isinstance(o, _AutoMock) def __hash__(self): return hash(self._name) def __lt__(self, o): return True def __le__(self, o): return True def __gt__(self, o): return False def __ge__(self, o): return False def __mro_entries__(self, bases): return (object,) df = _AutoMock('df')
Default CSV behaviour writes the row index as an unnamed first column — fine for round-tripping pandas, annoying for everyone else. index=False is the right default for handoffs.
11. Putting It Together
import pandas as pd df = pd.DataFrame({ "product": ["A", "B", "A", "C", "B", "A"], "price": [10.0, 25.0, 10.0, 15.0, 25.0, 10.0], "quantity": [3, 1, 2, 4, 2, 5], }) # Add revenue per row df["revenue"] = df["price"] * df["quantity"] # Total revenue per product revenue_by_product = df.groupby("product")["revenue"].sum().sort_values(ascending=False) print(revenue_by_product) # product # A 100.0 # B 75.0 # C 60.0
Five lines from raw transactions to a ranked answer. That's the pandas value proposition.
Common Mistakes
- Chained assignment:
df[df["x"] > 5]["y"] = 10may or may not work and pandas warns about it. Usedf.loc[df["x"] > 5, "y"] = 10. ==vs.equals()for DataFrames:df1 == df2returns an elementwise boolean DataFrame. To check whole-frame equality, usedf1.equals(df2).- Forgetting
index=Falseonto_csv. Your colleague opens the file in Excel and sees a mystery integer column. - Using
.iterrows(). It builds a Series for each row — orders of magnitude slower than vectorised ops. Use it only as a last resort. and/orin masks. Same trap as NumPy. Use&and|with parentheses:(df["a"] > 0) & (df["b"] < 5).- Confusing
.locand.iloc. Labels vs integer positions. Pick one mental model per cell and check the slice endpoint behaviour.
🎯 Your Turn — Revenue per Product
Build a DataFrame inline, compute revenue per row, then return total revenue per product via groupby.
Skeleton:
import pandas as pd def revenue_report(df): """Add a 'total' column, return total revenue per product as a Series.""" # TODO 1: add df["total"] = price * quantity # TODO 2: groupby product, sum the total, sort descending ... # Test df = pd.DataFrame({ "product": ["Pen", "Pad", "Pen", "Mug", "Pad", "Pen"], "price": [1.50, 3.00, 1.50, 8.00, 3.00, 1.50], "quantity": [10, 5, 8, 3, 6, 12], }) print(revenue_report(df)) # Expected (order): # Pen 45.0 # Pad 33.0 # Mug 24.0
Hint 1 — Adding the column
df["total"] = df["price"] * df["quantity"] creates the new column from elementwise multiplication. No loop needed.
Hint 2 — Grouping
df.groupby("product")["total"].sum() returns a Series indexed by product. Chain .sort_values(ascending=False) to rank it.
Show full solution
import pandas as pd def revenue_report(df): df = df.copy() df["total"] = df["price"] * df["quantity"] return ( df.groupby("product")["total"] .sum() .sort_values(ascending=False) ) df = pd.DataFrame({ "product": ["Pen", "Pad", "Pen", "Mug", "Pad", "Pen"], "price": [1.50, 3.00, 1.50, 8.00, 3.00, 1.50], "quantity": [10, 5, 8, 3, 6, 12], }) print(revenue_report(df)) # product # Pen 45.0 # Pad 33.0 # Mug 24.0 # Name: total, dtype: float64
Two transformations and a sort. The df.copy() keeps the caller's DataFrame untouched — a habit worth forming for any function that adds columns.
What You Learned
Seriesis 1D with an index,DataFrameis 2D — a dict of aligned Series.head,tail,info,describeare the four "look at the data" commands you'll run every day.- Filter rows with boolean masks; select columns with
df["col"]ordf[["a", "b"]]. .locis label-based (inclusive slices);.ilocis integer-position based (exclusive slices).- Add columns with vectorised arithmetic; reserve
applyanditerrowsfor last resorts. groupbyis the split-apply-combine engine — the single most useful pandas verb.- Always pass
index=Falsetoto_csvwhen handing off.
Next: Your First Data Visualisation — turning these tables into pictures people can read.
Practice this
on practicepython.inShort exercises that run in your browser and tell you what your code actually did, not just whether a test passed.