Exploratory Data Analysis

Before you train a model, build a dashboard or quote a number, look at the data carefully, and don't assume it is correct. John Tukey called this exploratory data analysis (EDA) in his 1977 book. It's still the cheapest way to avoid a wrong answer.

The baseball player Yogi Berra said it simply: "You can observe a lot by watching." A model trained on a column that was half NaN, or on prices typed in two different currencies, will run without any error and give you wrong results. Ten minutes of looking would have caught both. And plenty of useful findings come straight out of EDA, before any model exists.

Looking at and cleaning the data often takes more of a project than the modelling does. Plan for that from the start.

1. The Standard EDA Flow

Six steps, in this order:

text

1. Shape & types     "What am I looking at?"
2. Missing values    "What's not there?"
3. Distributions     "What does each column look like alone?"
4. Categoricals      "What are the levels and counts?"
5. Relationships     "How do columns relate to each other?"
6. Outliers          "What's unusually far from the rest?"

Run all six every time. The order matters: you can't judge relationships before you know the dtypes, and you can't trust a correlation when half the column is missing.

2. A Toy Dataset

We'll explore a small DataFrame inline so every step runs:

python

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "customer_id": range(1, 13),
    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],
    "city":        ["London", "Paris", "Berlin", "London", "Paris",
                    "London", "Berlin", "Paris", "London", "Berlin",
                    "Paris", "London"],
    "tier":        ["free", "pro", "pro", "free", "free",
                    "free", "enterprise", "pro", "free", "pro",
                    "free", "enterprise"],
    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,
                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],
})

print(df)

Real datasets have thousands of rows. With twelve rows, every output is small enough to read on screen.

3. Step 1: Shape & Types

python

print(df.shape)                       # (12, 5)
print(df.dtypes)
# customer_id      int64
# age            float64       (float because the column has NaNs)
# city            object
# tier            object
# spend          float64

df.info()
# Non-null counts, dtypes and memory usage, all in one call

+ setup added so this can run · re-runs 1 earlier block on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Look for:

Unexpected dtypes. A column you expected to be numeric showing as object usually means it has stray strings ("N/A", "-") mixed in.
float instead of int. Usually missing values forced the upcast (NumPy ints can't hold NaN).
Row count. Does it match what you were told to expect?

4. Step 2: Missing Values

python

print(df.isnull().sum())
# customer_id    0
# age            2
# city           0
# tier           0
# spend          0

+ setup added so this can run · re-runs 2 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

.isnull() returns a boolean DataFrame, and .sum() counts the Trues per column. Add .sort_values(ascending=False) to surface the worst offenders first.

Percentages are more useful than raw counts on real data:

python

print((df.isnull().mean() * 100).round(2))
# age            16.67

+ setup added so this can run · re-runs 3 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
    'print(df.isnull().sum())\n# customer_id    0\n# age            2\n# city           0\n# tier           0\n# spend          0',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Four ways to respond, chosen column by column:

Strategy	When to use
Drop rows (`df.dropna()`)	Few rows missing, plenty of data
Drop column	>50% missing and not critical
Fill / impute (`df.fillna(value)`)	Few rows missing but every row matters
Leave & flag	Algorithm can handle NaNs (e.g. xgboost)

python

df["age"] = df["age"].fillna(df["age"].median())     # fill missing ages with the median

+ setup added so this can run · re-runs 4 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
    'print(df.isnull().sum())\n# customer_id    0\n# age            2\n# city           0\n# tier           0\n# spend          0',
    'print((df.isnull().mean() * 100).round(2))\n# age            16.67',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

The fuller treatment is in the data cleaning lesson.

5. Step 3: Distributions of Numerics

python

print(df.describe())
#        customer_id        age        spend
# count        12.00      12.00        12.00   <- 12 ages: the two NaNs were filled above
# mean          6.50      34.33       216.21
# std           3.61       9.77       388.24   <- very large, which is suspicious
# min           1.00      19.00         5.00
# 25%           3.75      28.75        16.62
# 50%           6.50      32.50        52.50
# 75%           9.25      38.75       118.75
# max          12.00      55.00      1100.00   <- far above the 75th percentile

+ setup added so this can run · re-runs 5 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
    'print(df.isnull().sum())\n# customer_id    0\n# age            2\n# city           0\n# tier           0\n# spend          0',
    'print((df.isnull().mean() * 100).round(2))\n# age            16.67',
    'df["age"] = df["age"].fillna(df["age"].median())     # fill missing ages with the median',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

In describe(), look first at the gap between the 75th percentile and the max. When max is far above 75%, you have outliers: here the maximum spend is 1100, while the median is 52.50.

For shape, plot a histogram of every numeric column:

python

# df.hist(figsize=(10, 6), bins=20)
# plt.tight_layout()
# plt.savefig("dists.png")
# Chart preview: one histogram per numeric column laid out in a grid.

df.hist() draws a histogram for every numeric column in one line. The charts are simple, but they are usually enough.

6. Step 4: Categoricals

For object / category columns, describe(include="object") and value_counts do the work:

python

print(df.describe(include="object"))
#         city  tier
# count     12    12
# unique     3     3
# top   London  free
# freq       5     6

print(df["tier"].value_counts())
# free          6
# pro           4
# enterprise    2

print(df["tier"].value_counts(normalize=True).round(2))
# free          0.50
# pro           0.33
# enterprise    0.17

+ setup added so this can run · re-runs 6 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
    'print(df.isnull().sum())\n# customer_id    0\n# age            2\n# city           0\n# tier           0\n# spend          0',
    'print((df.isnull().mean() * 100).round(2))\n# age            16.67',
    'df["age"] = df["age"].fillna(df["age"].median())     # fill missing ages with the median',
    'print(df.describe())\n#        customer_id        age        spend\n# count        12.00      12.00        12.00   <- 12 ages: the two NaNs were filled above\n# mean          6.50      34.33       216.21\n# std           3.61       9.77       388.24   <- very large, which is suspicious\n# min           1.00      19.00         5.00\n# 25%           3.75      28.75        16.62\n# 50%           6.50      32.50        52.50\n# 75%           9.25      38.75       118.75\n# max          12.00      55.00      1100.00   <- far above the 75th percentile',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Look for:

Cardinality. 3 unique cities is fine. A column with 10,000 unique values is an identifier, not a category.
Class imbalance. If 95% of your target is "no", accuracy tells you nothing: a model that always says "no" scores 95%.
Typos and dirty levels. "London", "london " and "LONDON" should be one category, not three. Fix them in cleaning.

7. Step 5: Relationships

For numeric pairs, .corr() gives a matrix of Pearson correlations between -1 and +1:

python

print(df[["age", "spend"]].corr().round(2))
#         age  spend
# age    1.00   0.62
# spend  0.62   1.00

+ setup added so this can run · re-runs 7 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
    'print(df.isnull().sum())\n# customer_id    0\n# age            2\n# city           0\n# tier           0\n# spend          0',
    'print((df.isnull().mean() * 100).round(2))\n# age            16.67',
    'df["age"] = df["age"].fillna(df["age"].median())     # fill missing ages with the median',
    'print(df.describe())\n#        customer_id        age        spend\n# count        12.00      12.00        12.00   <- 12 ages: the two NaNs were filled above\n# mean          6.50      34.33       216.21\n# std           3.61       9.77       388.24   <- very large, which is suspicious\n# min           1.00      19.00         5.00\n# 25%           3.75      28.75        16.62\n# 50%           6.50      32.50        52.50\n# 75%           9.25      38.75       118.75\n# max          12.00      55.00      1100.00   <- far above the 75th percentile',
    'print(df.describe(include="object"))\n#         city  tier\n# count     12    12\n# unique     3     3\n# top   London  free\n# freq       5     6\n\nprint(df["tier"].value_counts())\n# free          6\n# pro           4\n# enterprise    2\n\nprint(df["tier"].value_counts(normalize=True).round(2))\n# free          0.50\n# pro           0.33\n# enterprise    0.17',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

Then plot it. A correlation is a single number. It can't show you a curve, a group of points, or one extreme value that changes the result:

python

# import matplotlib.pyplot as plt
# plt.scatter(df["age"], df["spend"], alpha=0.7)
# plt.xlabel("Age"); plt.ylabel("Spend")
# plt.savefig("scatter.png")
# Chart preview: a noisy rising cloud of points (older customers tend to spend more).

For numeric × categorical, groupby plus a boxplot:

python

print(df.groupby("tier")["spend"].agg(["mean", "median", "count"]))
#              mean  median  count
# tier
# enterprise  1040.0  1040.0      2
# free         15.92   15.25      6
# pro          104.75   99.5      4

+ setup added so this can run · re-runs 8 earlier blocks on this page first

# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

df = _AutoMock('df')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    import pandas as pd
    import numpy as np
except Exception:
    pass
_pm_earlier = [
    'import pandas as pd\nimport numpy as np\n\ndf = pd.DataFrame({\n    "customer_id": range(1, 13),\n    "age":         [25, 32, 47, np.nan, 28, 19, 55, 41, 33, 29, np.nan, 38],\n    "city":        ["London", "Paris", "Berlin", "London", "Paris",\n                    "London", "Berlin", "Paris", "London", "Berlin",\n                    "Paris", "London"],\n    "tier":        ["free", "pro", "pro", "free", "free",\n                    "free", "enterprise", "pro", "free", "pro",\n                    "free", "enterprise"],\n    "spend":       [12.5, 89.0, 145.0, 8.0, 22.0, 5.0,\n                    980.0, 110.0, 30.0, 75.0, 18.0, 1100.0],\n})\n\nprint(df)',
    'print(df.shape)                       # (12, 5)\nprint(df.dtypes)\n# customer_id      int64\n# age            float64       (float because the column has NaNs)\n# city            object\n# tier            object\n# spend          float64\n\ndf.info()\n# Non-null counts, dtypes and memory usage, all in one call',
    'print(df.isnull().sum())\n# customer_id    0\n# age            2\n# city           0\n# tier           0\n# spend          0',
    'print((df.isnull().mean() * 100).round(2))\n# age            16.67',
    'df["age"] = df["age"].fillna(df["age"].median())     # fill missing ages with the median',
    'print(df.describe())\n#        customer_id        age        spend\n# count        12.00      12.00        12.00   <- 12 ages: the two NaNs were filled above\n# mean          6.50      34.33       216.21\n# std           3.61       9.77       388.24   <- very large, which is suspicious\n# min           1.00      19.00         5.00\n# 25%           3.75      28.75        16.62\n# 50%           6.50      32.50        52.50\n# 75%           9.25      38.75       118.75\n# max          12.00      55.00      1100.00   <- far above the 75th percentile',
    'print(df.describe(include="object"))\n#         city  tier\n# count     12    12\n# unique     3     3\n# top   London  free\n# freq       5     6\n\nprint(df["tier"].value_counts())\n# free          6\n# pro           4\n# enterprise    2\n\nprint(df["tier"].value_counts(normalize=True).round(2))\n# free          0.50\n# pro           0.33\n# enterprise    0.17',
    'print(df[["age", "spend"]].corr().round(2))\n#         age  spend\n# age    1.00   0.62\n# spend  0.62   1.00',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

For all numerics at once, a heatmap of the correlation matrix shows which columns move together. Seaborn's heatmap does this in one line.

8. Step 6: Outliers

The boxplot is the diagnostic. The whiskers reach to 1.5×IQR, and anything beyond is flagged:

python

# df.boxplot(column="spend")
# plt.savefig("box.png")
# Chart preview: a tight box near 0-120 with two distant flier points near 980 and 1100.

For each suspect, investigate before "fixing":

Real but rare: keep them. A customer who spent £1100 is unusual, but still a real customer.
Data entry error: fix or drop. An age of 250 is a typo.
Different population: split the dataset. Enterprise and free-tier users behave so differently that one model for both groups will work badly for each of them.

Never delete outliers without a note saying why. They're often the most interesting rows you have.

9. EDA Heuristics: When You See X, Suspect Y

Symptom	Likely cause	First move
Huge skew, long right tail	Power-law variable	Try `np.log1p` and replot
One column with many `NaN`s	Optional field or join failure	Check upstream pipeline
`dtype=object` in a numeric column	Strings mixed in (`"N/A"`, `"-"`)	`pd.to_numeric(col, errors="coerce")`
95/5 class imbalance	Imbalanced target	Use a different metric (F1, AUC), not accuracy
Correlation 0.99 between two features	Duplicate or trivially-derived	Drop one
`max` >> `99th percentile`	Outliers or a data cap	Boxplot the column
Bimodal histogram	Two subpopulations mixed	Group by a categorical, replot

None of these are rules. Treat each one as a reason to look closer.

10. Write a Data Dictionary

Before modelling, write a short markdown table with one row per column: name, type, allowed values, unit and source. It takes five minutes and answers the questions your reviewer (or you, three months from now) will ask.

markdown

| column      | type    | unit      | notes                          |
|-------------|---------|-----------|--------------------------------|
| customer_id | int     | -         | unique, primary key            |
| age         | float   | years     | 16.7% missing, imputed median  |
| city        | string  | -         | 3 levels: London/Paris/Berlin  |
| tier        | string  | -         | 3 levels: free/pro/enterprise  |
| spend       | float   | GBP       | last 30 days, includes outliers|

11. A Note on Automated EDA

Tools like ydata-profiling (formerly pandas-profiling) and sweetviz generate a full HTML EDA report from a single call:

python

# from ydata_profiling import ProfileReport
# report = ProfileReport(df, title="Customer EDA")
# report.to_file("eda.html")

They're a good first scan of a dataset you've never seen. They show you problems, like a column that is 40% empty. You still have to find out why.

Common Mistakes

Skipping EDA and jumping to a model. The model will run, but it will also be wrong, and you won't know why.
Ignoring missing values. They spread through pandas operations without a warning (NaN + 1 is NaN) and skew your totals and averages.
Not looking at the target's distribution. Imbalanced classes, a heavy tail, or zero-inflation completely change which model is appropriate.
Treating describe() as enough. Mean and std hide bimodality, skew, and outliers. Plot the histogram.
Trusting correlation without a scatter. Anscombe's quartet is four small datasets with the same correlation and completely different shapes. It's worth looking up.
Cleaning outliers without understanding them. Sometimes the outlier is the finding.

🎯 Your Turn: `quick_eda()`

Write a reusable function that prints the first-pass EDA summary you'll want on every dataset: shape, dtypes, missing counts, and describe().

Skeleton:

python

import pandas as pd

def quick_eda(df):
    """Print a first-pass EDA summary of a DataFrame."""
    # TODO 1: shape and dtype counts
    # TODO 2: missing counts per column (only if > 0)
    # TODO 3: describe() for the numeric and the text (object) columns
    ...

# Test
import numpy as np
df = pd.DataFrame({
    "id":     range(1, 8),
    "age":    [25, 32, np.nan, 41, 28, 55, np.nan],
    "city":   ["London", "Paris", "London", "Berlin", "London", "Paris", "Berlin"],
    "spend":  [50, 120, 35, 200, 85, 410, 60],
})
quick_eda(df)

Hint 1: Filtering the missing report

Compute missing = df.isnull().sum(), then print only the columns where missing > 0. Use missing[missing > 0].

Hint 2: Describe for both kinds

df.describe() covers numerics. df.describe(include="object") covers strings. Wrap the second in try/except in case the DataFrame has no object columns.

Show full solution

python

import pandas as pd

def quick_eda(df):
    print("=" * 50)
    print(f"Shape: {df.shape[0]:,} rows x {df.shape[1]} columns")
    print("\nDtypes:")
    print(df.dtypes.value_counts())

    missing = df.isnull().sum()
    missing = missing[missing > 0]
    if not missing.empty:
        pct = (missing / len(df) * 100).round(2)
        report = pd.DataFrame({"missing": missing, "pct": pct})
        print("\nMissing values:")
        print(report.sort_values("missing", ascending=False))
    else:
        print("\nMissing values: none.")

    print("\nNumeric summary:")
    print(df.describe().round(2))

    try:
        print("\nCategorical summary:")
        print(df.describe(include="object"))
    except ValueError:
        pass

    print("=" * 50)


import numpy as np
df = pd.DataFrame({
    "id":     range(1, 8),
    "age":    [25, 32, np.nan, 41, 28, 55, np.nan],
    "city":   ["London", "Paris", "London", "Berlin", "London", "Paris", "Berlin"],
    "spend":  [50, 120, 35, 200, 85, 410, 60],
})
quick_eda(df)

Paste this into the first cell of every new notebook and you can see the basic facts about any dataset with one function call.

What You Learned

EDA follows a fixed six-step flow: shape → missing → distributions → categoricals → relationships → outliers.
.info(), .describe(), .isnull().sum(), .value_counts() are the four commands you'll run on every dataset.
Distributions matter more than means. Plot a histogram before you trust describe().
.corr() gives you one number, but a scatter plot shows you the shape behind it. Look at both.
Outliers need investigating before deleting.
Write a short data dictionary before modelling.
Tools like ydata-profiling give a quick first report, but they don't replace looking yourself.

That's the end of the data science fundamentals path: questions, Python idioms, NumPy, pandas, visualisation and EDA. The next paths take you deeper: Statistics for inference, Regression for your first models, and the full Data Science track for production-grade tooling.

1. The Standard EDA Flow

2. A Toy Dataset

3. Step 1: Shape & Types

4. Step 2: Missing Values

5. Step 3: Distributions of Numerics

6. Step 4: Categoricals

7. Step 5: Relationships

8. Step 6: Outliers

9. EDA Heuristics: When You See X, Suspect Y

10. Write a Data Dictionary

11. A Note on Automated EDA

Common Mistakes

🎯 Your Turn: quick_eda()

What You Learned

🎯 Your Turn: `quick_eda()`