PythonMastery
reference 3 min read · lesson 28 of 60 in Errors

UnicodeDecodeError: 'utf-8' codec can't decode byte

1 · The lesson

read

What this error means

You opened a file (or decoded a bytes object) telling Python it was UTF-8, but the bytes don't form valid UTF-8. The codec found a byte sequence that has no meaning in UTF-8, and it refused to guess. UnicodeDecodeError is a subclass of ValueError.

When you see it

text
Traceback (most recent call last):
  File "load.py", line 2, in <module>
    text = open("export.csv").read()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 142:
invalid start byte

The byte and the position help you find the cause. 0xa3 is the pound sign in Windows-1252 / Latin-1. 0xff 0xfe at position 0 is a UTF-16 BOM. 0xef 0xbb 0xbf is a UTF-8 BOM, which some tools don't handle.

Why it happens

Python 3's open() uses your platform's text encoding by default. This is UTF-8 on modern Linux and macOS, but on Windows it is usually a legacy code page such as cp1252. A file produced on Windows by Excel ("Save as CSV") is usually cp1252 or has a UTF-8 BOM. A file scraped from an old website may be latin-1. The bytes are valid in their original encoding, but they aren't valid UTF-8.

How to fix it

Option 1: pass the right encoding. This is the only real fix.

python
# Excel CSV, Western Europe / US English locale
text = open("export.csv", encoding="cp1252").read()

# Older web pages, plain text from legacy systems
text = open("scrape.html", encoding="latin-1").read()

# UTF-8 with a leading BOM (), which Excel adds
text = open("export.csv", encoding="utf-8-sig").read()

latin-1 is also useful as a fallback, because every single byte is a valid latin-1 character, so it never raises an error. But non-Western text may come out as unreadable characters.

Option 2: detect the encoding. This is useful when you have many files and don't know where they came from.

python
import chardet            # third-party: python -m pip install chardet

raw = open("export.csv", "rb").read()
guess = chardet.detect(raw)
print(guess)              # {'encoding': 'Windows-1252', 'confidence': 0.73}
text = raw.decode(guess["encoding"])

For UTF-8 vs UTF-16 only, the stdlib codecs module's BOM constants are enough.

Option 3: accept bad bytes. Use this only as a last resort, when losing some data is acceptable.

python
text = open("messy.txt", encoding="utf-8", errors="replace").read()
# bad bytes become U+FFFD (the replacement character)

text = open("messy.txt", encoding="utf-8", errors="ignore").read()
# bad bytes are dropped without any warning

errors="surrogateescape" is the right choice if you want to write the bytes back out unchanged. It stores each invalid byte as a special code point, and turns it back into the same byte when you encode the text again.

Option 4: re-export the source as UTF-8. When you control the program that creates the file (Excel, a database export), set its output encoding to UTF-8 once, and the problem goes away.

When you'd see this in real code

  • "It worked on my machine": a colleague's CSV, exported from Excel on Windows, opens fine for them (cp1252 default) and fails on your Linux server (UTF-8 default).
  • Scraping a page that is encoded as ISO-8859-1 and calling response.content.decode("utf-8") on it.
  • Reading log files from a 20-year-old appliance that emits Latin-1.
  • UnicodeEncodeError: the reverse direction: you have a string and tried to write it in an encoding that can't represent some character.
  • LookupError: unknown encoding: ...: a typo in the encoding name.

See Also

Found this useful? Share it with a friend who’s learning Python.