Pandas: Working with Tables

NumPy is great at numbers, but real datasets also have column names, mixed types, missing values and indexes. Pandas adds all four on top of NumPy. Most of your day-to-day data science work will happen in pandas.

You can think of pandas as Excel that you control with Python code. A DataFrame is like a spreadsheet, and a Series is like a single column.

1. The Two Types

python

import pandas as pd

# Series: 1D, like a NumPy array with a labelled index
ages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])
print(ages)
# Alice    25
# Bob      32
# Carol    47
# Dan      19
# dtype: int64

# DataFrame: a 2D table of Series that share one index
df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol", "Dan"],
    "age":    [25, 32, 47, 19],
    "city":   ["London", "Paris", "Berlin", "London"],
    "salary": [55_000, 72_000, 91_000, 38_000],
})

print(df)

You can think of a DataFrame as a dict of columns. Every column is a Series, and every column shares the same row index (default: 0, 1, 2, ...).

The convention is import pandas as pd, and almost all pandas code uses it.

2. Loading Data

There is a one-line loader for each common format:

python

# CSV: the format you will use most
df = pd.read_csv("sales.csv")

# JSON
df = pd.read_json("orders.json")

# Excel: needs the `openpyxl` package installed
df = pd.read_excel("report.xlsx", sheet_name="Q1")

# Parquet: the usual format for large analytical data
df = pd.read_parquet("events.parquet")

# Even straight from a URL
df = pd.read_csv("https://example.com/data.csv")

+ setup added so this can run · defines pd

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

pd = _AutoMock('pd')

For this lesson, we'll build DataFrames inline so every example runs immediately:

python

import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],
    "age":    [25, 32, 47, 19, 28],
    "city":   ["London", "Paris", "Berlin", "London", "Paris"],
    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],
})

3. Inspecting a DataFrame

The four commands to run every time you load a new dataset:

python

print(df.shape)                       # (5, 4)
print(df.head(3))                     # first 3 rows
print(df.tail(2))                     # last 2 rows
print(df.info())                      # column names, types, non-null counts
print(df.describe())                  # summary stats for numeric columns

+ setup added so this can run · re-runs 3 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

describe() gives you a lot of information in one call: count, mean, std, min, 25%, 50%, 75% and max for every numeric column. For categoricals, use describe(include="object").

python

print(df.dtypes)
# name      object
# age        int64
# city      object
# salary     int64

print(df.columns.tolist())            # ['name', 'age', 'city', 'salary']

+ setup added so this can run · re-runs 4 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

object dtype usually means strings.

4. Selecting Columns

python

# One column gives a Series
print(df["name"])

# Several columns give a DataFrame (note the double brackets)
print(df[["name", "salary"]])

# Dot access: works only if the name has no spaces and is not a reserved word
print(df.age)

+ setup added so this can run · re-runs 5 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Stick to df["col"] and df[["col1", "col2"]]. The dot form looks tidy, but it doesn't work for column names like "first name", and it gives you the method instead of the column when a column has the same name as a DataFrame method or attribute, such as .count or .shape.

5. Filtering Rows: Boolean Masking

This works the same way as in NumPy. The mask is a Series of booleans:

python

# One condition
print(df[df["age"] > 30])

# Combined: use & and |, and put each condition in parentheses
print(df[(df["age"] > 25) & (df["city"] == "London")])

# Membership
print(df[df["city"].isin(["London", "Berlin"])])

# String methods
print(df[df["name"].str.startswith("A")])

+ setup added so this can run · re-runs 5 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

The .str accessor exposes Python string methods over a whole column. .dt does the same for date columns.

6. `.loc` vs `.iloc`

Almost everyone is confused by this part of pandas at first. Memorise it.

.loc[rows, cols] is label-based. Uses the row's index value and the column's name.
.iloc[rows, cols] is based on integer positions. Uses 0-based integers like a Python list.

python

# Label-based: uses index labels and column names
print(df.loc[0, "name"])              # Alice
print(df.loc[0:2, "name"])            # rows 0 to 2 (2 is INCLUDED), name column
print(df.loc[df["age"] > 30, ["name", "salary"]])

# Integer-position based
print(df.iloc[0, 0])                  # cell at row 0, column 0, which is 'Alice'
print(df.iloc[0:2, 0:3])              # rows 0 and 1, columns 0 to 2 (slice ends are EXCLUDED)
print(df.iloc[-1])                    # last row

+ setup added so this can run · re-runs 6 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
    '# One condition\nprint(df[df["age"] > 30])\n\n# Combined: use & and |, and put each condition in parentheses\nprint(df[(df["age"] > 25) & (df["city"] == "London")])\n\n# Membership\nprint(df[df["city"].isin(["London", "Berlin"])])\n\n# String methods\nprint(df[df["name"].str.startswith("A")])',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Two traps to watch for:

1. df.loc[0:2] includes row 2. df.iloc[0:2] does not. (Labels include the endpoint, but integer slices don't.)
2. After filtering or sorting, the index is no longer 0, 1, 2, .... Reset it with df.reset_index(drop=True) if you need clean integers back.

7. Adding and Changing Columns

python

# New column from arithmetic (vectorised, so NumPy does the work)
df["bonus"] = df["salary"] * 0.10

# Conditional column
df["senior"] = df["age"] >= 30

# From a function applied to one column
df["initial"] = df["name"].str[0]

# From multiple columns
df["total_comp"] = df["salary"] + df["bonus"]

print(df)

+ setup added so this can run · re-runs 6 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
    '# One condition\nprint(df[df["age"] > 30])\n\n# Combined: use & and |, and put each condition in parentheses\nprint(df[(df["age"] > 25) & (df["city"] == "London")])\n\n# Membership\nprint(df[df["city"].isin(["London", "Berlin"])])\n\n# String methods\nprint(df[df["name"].str.startswith("A")])',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

For row-by-row logic over multiple columns, you can use df.apply(func, axis=1), but it is slow. Try vectorised expressions first, np.where second, and apply last.

python

import numpy as np

df["tier"] = np.where(df["salary"] >= 70_000, "high", "standard")

+ setup added so this can run · re-runs 7 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
    '# One condition\nprint(df[df["age"] > 30])\n\n# Combined: use & and |, and put each condition in parentheses\nprint(df[(df["age"] > 25) & (df["city"] == "London")])\n\n# Membership\nprint(df[df["city"].isin(["London", "Berlin"])])\n\n# String methods\nprint(df[df["name"].str.startswith("A")])',
    '# New column from arithmetic (vectorised, so NumPy does the work)\ndf["bonus"] = df["salary"] * 0.10\n\n# Conditional column\ndf["senior"] = df["age"] >= 30\n\n# From a function applied to one column\ndf["initial"] = df["name"].str[0]\n\n# From multiple columns\ndf["total_comp"] = df["salary"] + df["bonus"]\n\nprint(df)',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

8. Grouping: The Most Useful Feature

groupby is often the reason people move from Excel to pandas. It splits the DataFrame by a column, applies an aggregation to each group, and combines the results.

python

# Mean salary per city
print(df.groupby("city")["salary"].mean())
# city
# Berlin     91000.0
# London     46500.0
# Paris      68000.0

# Multiple aggregations on multiple columns
print(df.groupby("city").agg(
    avg_salary=("salary", "mean"),
    max_age=("age", "max"),
    headcount=("name", "count"),
))
#         avg_salary  max_age  headcount
# city
# Berlin     91000.0       47          1
# London     46500.0       25          2
# Paris      68000.0       32          2

+ setup added so this can run · re-runs 8 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
    '# One condition\nprint(df[df["age"] > 30])\n\n# Combined: use & and |, and put each condition in parentheses\nprint(df[(df["age"] > 25) & (df["city"] == "London")])\n\n# Membership\nprint(df[df["city"].isin(["London", "Berlin"])])\n\n# String methods\nprint(df[df["name"].str.startswith("A")])',
    '# New column from arithmetic (vectorised, so NumPy does the work)\ndf["bonus"] = df["salary"] * 0.10\n\n# Conditional column\ndf["senior"] = df["age"] >= 30\n\n# From a function applied to one column\ndf["initial"] = df["name"].str[0]\n\n# From multiple columns\ndf["total_comp"] = df["salary"] + df["bonus"]\n\nprint(df)',
    'import numpy as np\n\ndf["tier"] = np.where(df["salary"] >= 70_000, "high", "standard")',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

This pattern, split, apply, combine, works with any aggregation function (built-in or custom) and any number of grouping columns: df.groupby(["city", "tier"]).

9. Sorting

python

df.sort_values("salary", ascending=False)               # by one column
df.sort_values(["city", "salary"], ascending=[True, False])  # sort by two columns
df.nlargest(3, "salary")                                # shortcut for the top N rows
df.nsmallest(2, "age")

+ setup added so this can run · re-runs 9 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')
pd = _AutoMock('pd')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\n\n# Series: 1D, like a NumPy array with a labelled index\nages = pd.Series([25, 32, 47, 19], index=["Alice", "Bob", "Carol", "Dan"])\nprint(ages)\n# Alice    25\n# Bob      32\n# Carol    47\n# Dan      19\n# dtype: int64\n\n# DataFrame: a 2D table of Series that share one index\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan"],\n    "age":    [25, 32, 47, 19],\n    "city":   ["London", "Paris", "Berlin", "London"],\n    "salary": [55_000, 72_000, 91_000, 38_000],\n})\n\nprint(df)',
    '# CSV: the format you will use most\ndf = pd.read_csv("sales.csv")\n\n# JSON\ndf = pd.read_json("orders.json")\n\n# Excel: needs the `openpyxl` package installed\ndf = pd.read_excel("report.xlsx", sheet_name="Q1")\n\n# Parquet: the usual format for large analytical data\ndf = pd.read_parquet("events.parquet")\n\n# Even straight from a URL\ndf = pd.read_csv("https://example.com/data.csv")',
    'import pandas as pd\n\ndf = pd.DataFrame({\n    "name":   ["Alice", "Bob", "Carol", "Dan", "Eve"],\n    "age":    [25, 32, 47, 19, 28],\n    "city":   ["London", "Paris", "Berlin", "London", "Paris"],\n    "salary": [55_000, 72_000, 91_000, 38_000, 64_000],\n})',
    'print(df.shape)                       # (5, 4)\nprint(df.head(3))                     # first 3 rows\nprint(df.tail(2))                     # last 2 rows\nprint(df.info())                      # column names, types, non-null counts\nprint(df.describe())                  # summary stats for numeric columns',
    "print(df.dtypes)\n# name      object\n# age        int64\n# city      object\n# salary     int64\n\nprint(df.columns.tolist())            # ['name', 'age', 'city', 'salary']",
    '# One condition\nprint(df[df["age"] > 30])\n\n# Combined: use & and |, and put each condition in parentheses\nprint(df[(df["age"] > 25) & (df["city"] == "London")])\n\n# Membership\nprint(df[df["city"].isin(["London", "Berlin"])])\n\n# String methods\nprint(df[df["name"].str.startswith("A")])',
    '# New column from arithmetic (vectorised, so NumPy does the work)\ndf["bonus"] = df["salary"] * 0.10\n\n# Conditional column\ndf["senior"] = df["age"] >= 30\n\n# From a function applied to one column\ndf["initial"] = df["name"].str[0]\n\n# From multiple columns\ndf["total_comp"] = df["salary"] + df["bonus"]\n\nprint(df)',
    'import numpy as np\n\ndf["tier"] = np.where(df["salary"] >= 70_000, "high", "standard")',
    '# Mean salary per city\nprint(df.groupby("city")["salary"].mean())\n# city\n# Berlin     91000.0\n# London     46500.0\n# Paris      68000.0\n\n# Multiple aggregations on multiple columns\nprint(df.groupby("city").agg(\n    avg_salary=("salary", "mean"),\n    max_age=("age", "max"),\n    headcount=("name", "count"),\n))\n#         avg_salary  max_age  headcount\n# city\n# Berlin     91000.0       47          1\n# London     46500.0       25          2\n# Paris      68000.0       32          2',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

nlargest / nsmallest are faster than sort_values(...).head(n) when N is small.

10. Saving

python

df.to_csv("out.csv", index=False)         # almost always pass index=False
df.to_json("out.json", orient="records")
df.to_parquet("out.parquet")              # for analytical pipelines
df.to_excel("out.xlsx", index=False)

+ setup added so this can run · defines df

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

By default, to_csv writes the row index as an unnamed first column. That is fine if you only read the file back into pandas, but confusing for anyone else. Use index=False when you give the file to other people.

11. Putting It Together

python

import pandas as pd

df = pd.DataFrame({
    "product":  ["A", "B", "A", "C", "B", "A"],
    "price":    [10.0, 25.0, 10.0, 15.0, 25.0, 10.0],
    "quantity": [3, 1, 2, 4, 2, 5],
})

# Add revenue per row
df["revenue"] = df["price"] * df["quantity"]

# Total revenue per product
revenue_by_product = df.groupby("product")["revenue"].sum().sort_values(ascending=False)
print(revenue_by_product)
# product
# A    100.0
# B     75.0
# C     60.0

It takes five lines to go from raw transactions to a ranked answer. This is why people use pandas.

Common Mistakes

Chained assignment: df[df["x"] > 5]["y"] = 10 does not reliably change df, and pandas warns about it. Use df.loc[df["x"] > 5, "y"] = 10.
== vs .equals() for DataFrames: df1 == df2 returns an elementwise boolean DataFrame. To check whole-frame equality, use df1.equals(df2).
Forgetting index=False on to_csv. Your colleague opens the file in Excel and sees an extra integer column with no name.
Using .iterrows(). It builds a Series for each row, which makes it many times slower than vectorised operations. Use it only as a last resort.
and / or in masks. The same trap as in NumPy. Use & and | with parentheses: (df["a"] > 0) & (df["b"] < 5).
Confusing .loc and .iloc. Labels vs integer positions. Use one of them consistently in each cell, and check whether the slice includes the endpoint.

🎯 Your Turn: Revenue per Product

Build a DataFrame inline, compute revenue per row, then return total revenue per product via groupby.

Skeleton:

python

import pandas as pd

def revenue_report(df):
    """Add a 'total' column, return total revenue per product as a Series."""
    # TODO 1: add df["total"] = price * quantity
    # TODO 2: groupby product, sum the total, sort descending
    ...

# Test
df = pd.DataFrame({
    "product":  ["Pen", "Pad", "Pen", "Mug", "Pad", "Pen"],
    "price":    [1.50, 3.00, 1.50, 8.00, 3.00, 1.50],
    "quantity": [10,    5,    8,    3,    6,    12],
})
print(revenue_report(df))
# Expected (order):
# Pen    45.0
# Pad    33.0
# Mug    24.0

Hint 1: Adding the column

df["total"] = df["price"] * df["quantity"] creates the new column from elementwise multiplication. No loop needed.

Hint 2: Grouping

df.groupby("product")["total"].sum() returns a Series indexed by product. Chain .sort_values(ascending=False) to rank it.

Show full solution

python

import pandas as pd

def revenue_report(df):
    df = df.copy()
    df["total"] = df["price"] * df["quantity"]
    return (
        df.groupby("product")["total"]
          .sum()
          .sort_values(ascending=False)
    )

df = pd.DataFrame({
    "product":  ["Pen", "Pad", "Pen", "Mug", "Pad", "Pen"],
    "price":    [1.50, 3.00, 1.50, 8.00, 3.00, 1.50],
    "quantity": [10,    5,    8,    3,    6,    12],
})
print(revenue_report(df))
# product
# Pen    45.0
# Pad    33.0
# Mug    24.0
# Name: total, dtype: float64

Two transformations and a sort. The df.copy() leaves the caller's DataFrame unchanged. Do this in any function that adds columns.

What You Learned

Series is 1D with an index, DataFrame is 2D: a dict of Series that share one index.
head, tail, info, describe are the four "look at the data" commands you'll run every day.
Filter rows with boolean masks, and select columns with df["col"] or df[["a", "b"]].
.loc is label-based (inclusive slices), and .iloc is integer-position based (exclusive slices).
Add columns with vectorised arithmetic. Use apply and iterrows only as a last resort.
groupby does split-apply-combine, and it is one of the most useful pandas methods.
Always pass index=False to to_csv when you give the file to someone else.

Next: Your First Data Visualisation: turn these tables into charts that people can read.

1 · The lesson

1. The Two Types

2. Loading Data

3. Inspecting a DataFrame

4. Selecting Columns

5. Filtering Rows: Boolean Masking

6. `.loc` vs `.iloc`

7. Adding and Changing Columns

8. Grouping: The Most Useful Feature

9. Sorting

10. Saving

11. Putting It Together

Common Mistakes

🎯 Your Turn: Revenue per Product

What You Learned

Practice this

Pandas: Working with Tables

1. The Two Types

2. Loading Data

3. Inspecting a DataFrame

4. Selecting Columns

5. Filtering Rows: Boolean Masking

6. .loc vs .iloc

7. Adding and Changing Columns

8. Grouping: The Most Useful Feature

9. Sorting

10. Saving

11. Putting It Together

Common Mistakes

🎯 Your Turn: Revenue per Product

What You Learned

6. `.loc` vs `.iloc`