OSError: [Errno 24] Too many open files

What this error means

Your process hit the operating system's limit on open file descriptors. A descriptor is not only a file. Every socket, pipe and database connection uses one too, so a program that never opens a file can still reach the limit with network connections alone.

The error almost always means a leak. Something is being opened in a loop and never closed.

When you see it

Traceback (most recent call last):
  File "worker.py", line 31, in fetch_all
    r = requests.get(url)
OSError: [Errno 24] Too many open files

It rarely appears where the bug is. The leak grows without any warning, and the error appears at whichever line asked for the descriptor that went over the limit. This is often minutes or hours after the program started.

Why it happens

• Files opened without with. f = open(path) keeps the descriptor open until the object is garbage collected. That may be much later, or, if the object is in a reference cycle, it may not happen for a long time.
• A new requests.Session per call. Each session holds its own connection pool. Creating one per request in a loop leaks more sockets with every request.
• Sockets left in CLOSE_WAIT because the code stopped reading but never closed.
• A database connection per request instead of a pool, with connections never returned.
• A limit that is too low for the workload. The default is often 1024, and a busy service with many concurrent connections can reach it without any leak.

How to fix it

Count the open descriptors, and watch how the number changes. This is the fastest way to tell a leak from a limit that is too low:

ls /proc/$(pgrep -f worker.py)/fd | wc -l     # Linux
lsof -p $(pgrep -f worker.py) | wc -l         # macOS

Run it a few times a minute apart. If the number keeps going up, you have a leak. If it stays flat but high, the limit really is too small.

From inside the process:

import resource

soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print(f"file descriptor limit: soft={soft} hard={hard}")

Always use with for files. It closes the file when the block ends, including when an exception is raised:

with open("data.csv", encoding="utf-8") as f:
    rows = f.readlines()
# the file is always closed by this point

Reuse one session for many requests. A new session per request is the most common cause of this leak in Python services:

import requests

session = requests.Session()          # create it once, at module or app level
for url in urls:
    r = session.get(url, timeout=(3.05, 10))

urls = ["alpha", "beta", "gamma"]

Pool your database connections rather than opening one per request, and return them:

from psycopg_pool import ConnectionPool

pool = ConnectionPool("postgresql://db/shop", min_size=2, max_size=10)

with pool.connection() as conn:        # borrowed here, returned at the end of the block
    conn.execute("SELECT 1")

Raise the limit only once you know it is not a leak. Raising it on a leaking process only delays the error and hides the cause:

# docker-compose.yml
services:
  worker:
    ulimits:
      nofile: { soft: 65535, hard: 65535 }

When you'd see this in real code

• A crawler creating a Session inside the loop instead of outside it. It works for ten URLs and fails at ten thousand.
• A log handler opened per message rather than once at startup.
• An image pipeline using Image.open() without with, keeping every file open until garbage collection runs.
• A service that runs fine for days and fails on the busiest afternoon, because the leak needed enough traffic to reach the limit.

Related errors

• MemoryError: the same kind of bug, but a different resource runs out.
• [ConnectionRefusedError: [Errno 111] Connection refused](../error-connection-refused/): what clients see once you can no longer accept connections.
• [PermissionError: [Errno 13] Permission denied](../error-permissionerror/)

See Also

• All Python errors: the full index, by type and by when it happens.
• File I/O, Pathlib, and Large Files: why with exists.
• Databases in production: pooling.