PythonMastery
intermediate 18 min read · lesson 8 of 13 in Python Intermediate

File I/O, Pathlib, and Large Files

1 · The lesson

read

You already know with open(...) as f, the read/write modes, and that for line in f is better than f.read() for anything but small files. This lesson shows how file code looks in professional Python: pathlib instead of paths built from strings, streaming for huge files, atomic writes that survive crashes, and the encoding problem that sooner or later affects every program that runs on more than one operating system.

If you still use os.path.join, this lesson shows you the modern replacement.

1. Pathlib: Stop Treating Paths as Strings

pathlib.Path is the object-oriented path API. A Path knows its parent, its suffix and its parts. It can also read and write the file, search with glob patterns, and get file information (stat). You will rarely need raw strings or os.path again.

python
from pathlib import Path

p = Path("data") / "logs" / "app.log"        # the / operator builds paths
print(p)                                     # data/logs/app.log (or data\logs\app.log on Windows)

print(p.name)                                # app.log
print(p.stem)                                # app
print(p.suffix)                              # .log
print(p.parent)                              # data/logs
print(p.parts)                               # ('data', 'logs', 'app.log')

Path defines its own meaning for the / operator. It uses the correct separator on every platform, so you never need os.sep.

2. The Five Pathlib Methods You'll Use Daily

python
from pathlib import Path

root = Path("project")

root.mkdir(parents=True, exist_ok=True)      # creates parent folders too, no error if it exists
(root / "config.yaml").exists()              # True/False
(root / "config.yaml").is_file()             # True only if it exists AND is a file
(root / "src").is_dir()                      # the same check, for a directory

# Read or write in one call. Short files need no `with` block.
(root / "README.md").write_text("# Hi\n", encoding="utf-8")
content = (root / "README.md").read_text(encoding="utf-8")

# Binary equivalents
(root / "thumb.png").write_bytes(b"\x89PNG...")
blob = (root / "thumb.png").read_bytes()

A few more that are very useful:

python
# Globbing: find every file that matches a pattern
for csv in Path("data").glob("*.csv"):
    print(csv)

# Recursive glob: ** also looks inside subfolders
for py in Path("src").rglob("*.py"):
    print(py)

# Change the file extension, common when converting files
src = Path("notes.md")
out = src.with_suffix(".html")               # notes.html

# Loop over the items directly inside a folder
for child in Path(".").iterdir():
    print(child, "(dir)" if child.is_dir() else "(file)")
+ setup added so this can run · re-runs 2 earlier blocks on this page first
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def Path(*_a, **_kw):
    print('-> Path() called')
    return _AutoMock('Path()')

# Earlier blocks on this page that this one builds on, re-run quietly first.
try:
    from pathlib import Path
except Exception:
    pass
_pm_earlier = [
    'from pathlib import Path\n\np = Path("data") / "logs" / "app.log"        # the / operator builds paths\nprint(p)                                     # data/logs/app.log (or data\\logs\\app.log on Windows)\n\nprint(p.name)                                # app.log\nprint(p.stem)                                # app\nprint(p.suffix)                              # .log\nprint(p.parent)                              # data/logs\nprint(p.parts)                               # (\'data\', \'logs\', \'app.log\')',
    'from pathlib import Path\n\nroot = Path("project")\n\nroot.mkdir(parents=True, exist_ok=True)      # creates parent folders too, no error if it exists\n(root / "config.yaml").exists()              # True/False\n(root / "config.yaml").is_file()             # True only if it exists AND is a file\n(root / "src").is_dir()                      # the same check, for a directory\n\n# Read or write in one call. Short files need no `with` block.\n(root / "README.md").write_text("# Hi\\n", encoding="utf-8")\ncontent = (root / "README.md").read_text(encoding="utf-8")\n\n# Binary equivalents\n(root / "thumb.png").write_bytes(b"\\x89PNG...")\nblob = (root / "thumb.png").read_bytes()',
]
def _pm_run_earlier(_srcs, _g):
    import sys as _s, io as _io, warnings as _w
    for _src in _srcs:
        _o, _e = _s.stdout, _s.stderr
        _s.stdout = _s.stderr = _io.StringIO()
        try:
            with _w.catch_warnings():
                _w.simplefilter('ignore')
                exec(compile(_src, '<earlier block>', 'exec'), _g)
        except (Exception, SystemExit):
            pass
        finally:
            _s.stdout, _s.stderr = _o, _e
    if 'matplotlib.pyplot' in _s.modules:
        _s.modules['matplotlib.pyplot'].close('all')   # their plots belong to their own blocks
_pm_run_earlier(_pm_earlier, globals())
del _pm_earlier, _pm_run_earlier

write_text and read_text are fine for small files, when you really want the whole file in memory. For anything that could grow, keep using with open(...) and stream it (see Section 5).

3. Text vs Binary Mode: When Bytes Matter

Text mode ("r", "w", "a") decodes bytes to str on read and encodes on write. Binary mode ("rb", "wb", "ab") gives you raw bytes, with no decoding and no newline translation.

Use binary mode for anything that isn't human-readable text:

python
# Hashing a file: read bytes only, never decode them
import hashlib

def file_sha256(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()

Common binary cases: images, audio, video, pickle files, executables, PDFs, and anything that gave you a UnicodeDecodeError when you read it as text. If you read a JPEG in text mode, Python tries to decode the bytes with a text encoding, usually fails on some byte, and raises an error.

On Windows, text mode also translates \n to \r\n on write. You don't notice this most of the time, but it gives wrong results when you compute a hash or compare file sizes. When in doubt, use binary mode.

4. Encoding: The "Works on My Machine" Problem

Text files are bytes plus an encoding. The world has standardised on UTF-8. Most operating systems default to UTF-8. On Windows, Python's open() has long used the system's locale encoding by default, which is often cp1252: a Western European code page that can't represent most non-ASCII text.

python
# Danger: the encoding defaults to the operating system's choice
with open("notes.txt", "w") as f:
    f.write("café — 北京 — 🐍")           # crashes on Windows cp1252

# Correct: always give the encoding, every time
with open("notes.txt", "w", encoding="utf-8") as f:
    f.write("café — 北京 — 🐍")

This bug is hard to catch. The code works on your Mac and in CI on Linux. Then a Windows user runs it and gets UnicodeEncodeError: 'charmap' codec can't encode character. Or, worse, the program writes garbled text (called mojibake) without any error.

Rules:

  • Reading text? Pass encoding="utf-8".
  • Writing text? Pass encoding="utf-8".
  • Reading text from a source that really uses a different encoding (legacy logs, a CSV from a Windows export)? Pass that encoding explicitly.
  • Never rely on the default. Writing the encoding costs you a few extra characters.

When the source might contain a few invalid bytes, you can replace them instead of failing:

python
with open("legacy.txt", "r", encoding="utf-8", errors="replace") as f:
    text = f.read()        # bad bytes become � instead of crashing

5. Reading Huge Files: Iterate, Don't Load Everything

A very common bug with large files is loading a 4 GB log into memory all at once. The fix is the iteration pattern you already know, but it helps to see both versions side by side.

python
# Bad: loads the entire file into a list of strings
with open("server.log", encoding="utf-8") as f:
    lines = f.readlines()              # a 4 GB file needs more than 4 GB of RAM
    for line in lines:
        process(line)

# Good: reads one line at a time, so memory use stays small
with open("server.log", encoding="utf-8") as f:
    for line in f:                     # file objects are iterators
        process(line)
+ setup added so this can run · defines process
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def process(*_a, **_kw):
    print('-> process() called')
    return _AutoMock('process()')

The file object is its own iterator. Each next() call reads up to the next \n and returns the line (including the trailing \n). Memory usage stays at one buffer's worth, no matter how big the file is.

Need only the first line? f.readline(). Need only the last ten? Use collections.deque with a max length. It keeps only the most recent items as you read:

python
from collections import deque

with open("server.log", encoding="utf-8") as f:
    last_ten = deque(f, maxlen=10)     # reads every line but keeps only the last 10

for line in last_ten:
    print(line, end="")

6. Streaming Binary Chunks

For binary files, you read in fixed-size chunks instead of lines. The standard loop uses the walrus operator (:=), Python's assignment expression:

python
with open("video.mp4", "rb") as f:
    while chunk := f.read(8192):       # read 8 KB at a time, stop when read returns b""
        process(chunk)
+ setup added so this can run · defines process, chunk
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def process(*_a, **_kw):
    print('-> process() called')
    return _AutoMock('process()')
chunk = _AutoMock('chunk')

chunk := f.read(8192) assigns the value and returns it. The loop ends when f.read() returns b"" (empty bytes), which is the EOF signal in binary mode. 8 KB is a fine default, but 64 KB to 1 MB is better for high-throughput work on SSDs.

The version without the walrus operator is longer, but it also works on Python 3.7 and earlier:

python
with open("video.mp4", "rb") as f:
    for chunk in iter(lambda: f.read(8192), b""):
        process(chunk)
+ setup added so this can run · defines process
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def process(*_a, **_kw):
    print('-> process() called')
    return _AutoMock('process()')

iter(callable, sentinel) keeps calling callable() until it returns the sentinel. Both forms are idiomatic.

7. Atomic Writes: Surviving Crashes

With the simple pattern, there is a moment when the file on disk is only half-written:

python
# Bad: if Python crashes while writing, config.json is left broken
with open("config.json", "w", encoding="utf-8") as f:
    f.write(serialize(big_object))     # a crash here leaves an empty or half-written file
+ setup added so this can run · defines serialize, big_object
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

def serialize(*_a, **_kw):
    print('-> serialize() called')
    return _AutoMock('serialize()')
big_object = _AutoMock('big_object')

The professional pattern is to write to a temp file, then rename it:

python
import os
from pathlib import Path

def atomic_write_text(path, text, encoding="utf-8"):
    """Write text to `path` atomically. Readers see either the old file or the full new one."""
    path = Path(path)
    tmp = path.with_suffix(path.suffix + ".tmp")
    tmp.write_text(text, encoding=encoding)
    os.replace(tmp, path)              # atomic on POSIX (see below)

On POSIX systems (Linux, macOS), the Python docs guarantee that os.replace() is atomic: the rename either fully happens or doesn't happen at all. Readers of the target file never see a half-written file. On Windows the docs don't promise this, but the pattern is still much safer than writing the file in place. Configuration files, sqlite-style databases, and anything that other processes might read while you're writing should use this pattern.

For extra safety against a power failure or an OS crash, call tmp_fh.flush() and os.fsync(tmp_fh.fileno()) before closing the temp file. That makes the OS write the bytes to disk, not just keep them in its memory cache. Most apps don't need this, but databases do.

8. Temporary Files and Directories

The tempfile module gives you safe, OS-chosen temp paths that get cleaned up automatically.

python
import tempfile
from pathlib import Path

# A one-off temp FILE, deleted when the block exits (delete_on_close needs Python 3.12+)
with tempfile.NamedTemporaryFile(mode="w", encoding="utf-8", suffix=".json", delete_on_close=False) as tmp:
    tmp.write('{"hello": "world"}')
    tmp.flush()
    print("Temp path:", tmp.name)
    # use tmp.name as a real path here, for example give it to another tool
# the file is deleted here

# A whole temp DIRECTORY, removed with all its contents when the block exits
with tempfile.TemporaryDirectory() as tmpdir:
    workdir = Path(tmpdir)
    (workdir / "input.txt").write_text("hi", encoding="utf-8")
    # ... do work in workdir ...
# the folder and everything in it are gone

Use NamedTemporaryFile when you need a real file path to give to a subprocess or another library. Use TemporaryDirectory for test fixtures, build sandboxes, and any case where you need to write many files and then delete them all.

Both clean up when the with block ends. The Advanced path shows how to build your own context managers.

9. shutil Highlights

shutil is the standard library's module for high-level file operations. These are the ones you'll use most:

python
import shutil
from pathlib import Path

shutil.copy("src.txt", "dst.txt")              # copy one file (shutil.copy2 also copies metadata)
shutil.copytree("src_dir", "dst_dir")          # copy a directory tree
shutil.move("from.txt", "to.txt")              # rename or move, even to another drive
shutil.rmtree("scratch_dir")                   # delete a folder and all its contents. No undo, no Trash.

usage = shutil.disk_usage(Path.home())
print(f"{usage.free / 1e9:.1f} GB free")

shutil.rmtree deletes everything under the path, and there's no undo. Check the path twice, and keep the default ignore_errors=False so you see an error when something fails.

10. CSV and JSON: A Short Introduction

You'll often want structured text rather than raw lines. The standard library has both:

python
import csv, json

# CSV: read the rows as dicts
with open("people.csv", encoding="utf-8", newline="") as f:
    for row in csv.DictReader(f):
        print(row["name"], row["email"])

# JSON: read a whole file into a Python object
with open("config.json", encoding="utf-8") as f:
    config = json.load(f)

# JSON: write a Python object to a file
with open("config.json", "w", encoding="utf-8") as f:
    json.dump(config, f, indent=2)

Always pass the newline="" argument for CSV files. The csv module handles newlines itself, and the default text-mode translation can break quoted fields that contain newlines. Both modules have a full lesson's worth of options (dialects, custom encoders, streaming). The CSV & JSON How-To covers them.

11. Common Mistakes

1. Forgetting encoding=
The code works on your machine and breaks on other machines. Adding encoding="utf-8" is the cheapest way to make your code more reliable. Every text open() should have it.

2. Holding file handles open longer than needed
Use with for every file you open. A file that stays open longer than needed uses up OS file descriptors, locks the file on Windows, and risks data loss if your process is killed before the buffer flushes.

3. Reading the whole file when you only need part
Calling f.read() on a 4 GB file when you only wanted the first line is a beginner's mistake. Use f.readline() for the first line, collections.deque(f, maxlen=N) for the last N, and for line in f for everything in between.

4. Comparing paths as strings
"data/log.txt" == "data\\log.txt" is False. Path("data/log.txt") == Path("data\\log.txt") is True on Windows. Once you have Path objects, compare and hash them as paths. Don't convert them with str() just to use ==.

5. The check-then-act race

python
# Race condition: another process could create the file between these two lines
if not path.exists():
    path.write_text("default")
+ setup added so this can run · defines path
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

path = _AutoMock('path')

Between the exists() and the write_text(), another process or thread could create the file. Either use exclusive-create mode (open(path, "x") raises FileExistsError if it exists), use Path.write_text(...) unconditionally if you're happy to overwrite, or catch the exception:

python
try:
    with open(path, "x", encoding="utf-8") as f:
        f.write("default")
except FileExistsError:
    pass        # another process created it first, which is fine
+ setup added so this can run · defines path
# Lightweight mock for objects whose attributes/methods aren't critical
class _AutoMock:
    def __init__(self, name='mock'): self._name = name
    def __getattr__(self, k): return _AutoMock(self._name + '.' + k)
    def __call__(self, *a, **kw):
        print('-> ' + self._name + '() called')
        return _AutoMock(self._name + '()')
    def __repr__(self): return '<mock ' + self._name + '>'
    def __str__(self): return '<mock ' + self._name + '>'
    def __bool__(self): return True
    def __iter__(self): return iter([])
    def __len__(self): return 0
    def __getitem__(self, k): return _AutoMock(self._name + '[...]')
    def __setitem__(self, k, v): pass
    def __enter__(self): return self
    def __exit__(self, *a): return False
    async def __aenter__(self): return self
    async def __aexit__(self, *a): return False
    def __add__(self, o): return self
    def __radd__(self, o): return self
    def __sub__(self, o): return self
    def __mul__(self, o): return self
    def __rmul__(self, o): return self
    def __truediv__(self, o): return self
    def __eq__(self, o): return isinstance(o, _AutoMock)
    def __hash__(self): return hash(self._name)
    def __lt__(self, o): return True
    def __le__(self, o): return True
    def __gt__(self, o): return False
    def __ge__(self, o): return False
    def __mro_entries__(self, bases): return (object,)

path = _AutoMock('path')

This is the EAFP principle: "easier to ask forgiveness than permission". We'll cover it more in Exceptions.

🎯 Your Turn: Remove Duplicate Lines From a Huge File

Write dedupe_lines(input_path, output_path) that reads a (potentially huge) text file and writes the unique lines, in first-seen order, to the output file. The constraint: memory must be O(unique line count), not O(file size). You can't load the whole file with readlines().

python
# Input file (one line per row):
# alpha
# beta
# alpha
# gamma
# beta
# delta

# Output file:
# alpha
# beta
# gamma
# delta

Skeleton:

python
from pathlib import Path

def dedupe_lines(input_path, output_path):
    # TODO 1: open input for reading and output for writing, both as UTF-8
    # TODO 2: keep a set of lines you've already seen
    # TODO 3: stream input line-by-line; if a line is new, write it and add to the set
    ...
Hint 1: Read line by line Iterate the file object directly: for line in f:. Each iteration reads one line. Don't call readlines(), because that loads everything.
Hint 2: Memory-efficient deduplication A set gives you O(1) membership checks. You store each unique line in it the first time you see it. The set grows with the number of distinct lines, not the total number of lines. That is what the task asks for.
Show full solution
python
from pathlib import Path

def dedupe_lines(input_path, output_path):
    """Copy each line only the first time it appears. Memory grows with unique lines."""
    seen = set()
    in_path = Path(input_path)
    out_path = Path(output_path)

    with in_path.open("r", encoding="utf-8") as src, \
         out_path.open("w", encoding="utf-8") as dst:
        for line in src:
            if line not in seen:
                seen.add(line)
                dst.write(line)


# Demo
import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmpdir:
    tmp = Path(tmpdir)
    src = tmp / "in.txt"
    dst = tmp / "out.txt"

    src.write_text("alpha\nbeta\nalpha\ngamma\nbeta\ndelta\n", encoding="utf-8")
    dedupe_lines(src, dst)
    print(dst.read_text(encoding="utf-8"))
    # alpha
    # beta
    # gamma
    # delta

What you did:


  • Streamed the input: for line in src reads one line at a time and never loads the whole file.

  • Used a set for membership: O(1) check, O(unique) storage.

  • Wrote each new line straight to the output, with no intermediate list.

  • Wrapped both files in one with so they close even if writing fails halfway.

If the input had 10 million lines but only 100 unique, your memory usage is 100 entries, not 10 million. That's the difference between a script that finishes and a script that the system stops because it ran out of memory.

For huge inputs where even the set of unique lines won't fit in memory, you need a method that uses the disk (sort + uniq), or a Bloom filter, which removes duplicates with a small chance of error. That's outside this lesson, but those methods also build on the streaming pattern.

What You Learned

  • Pathlib replaces os.path. Build paths with /, check them with .exists() / .is_file() / .glob(), change the extension with .with_suffix(), create with .mkdir(parents=True, exist_ok=True).
  • Binary mode ("rb", "wb") for anything that isn't human-readable text: hashes, images, pickles. Text mode can fail to decode such data, and it changes newlines.
  • Always pass encoding="utf-8". The default encoding causes bugs on Windows.
  • Stream, don't load the whole file. for line in f for text, while chunk := f.read(8192) for binary. O(1) memory.
  • Atomic writes: write to *.tmp, then os.replace(tmp, final). This survives crashes.
  • tempfile.NamedTemporaryFile and TemporaryDirectory clean up after themselves.
  • shutil for copy/move/rmtree/disk_usage. rmtree is permanent, so check the path twice.
  • csv and json for structured text, covered in full in the CSV & JSON How-To.
  • Avoid check-then-act races: use open(path, "x") or catch the exception (EAFP).

Next: Exceptions: Patterns Beyond the Basics: custom exception hierarchies, chaining, ExceptionGroup, and the patterns used in production code.

Practice this

on practicepython.in

Short exercises that run in your browser and tell you what your code did, as well as whether a test passed.

Found this useful? Share it with a friend who’s learning Python.