Introduction

UnicodeDecodeError: 'utf8' codec can't decode byte 0x9c means Python was told to interpret some bytes as UTF-8, but the byte sequence does not follow UTF-8 rules. In practice, the data is usually encoded with a different character set, often an older Windows or Latin-based encoding.

What the Error Really Means

Python stores text as Unicode, but files and network payloads are bytes. Decoding is the step that turns bytes into text. If you use the wrong codec, Python cannot map the raw bytes correctly.

A UTF-8 decoder expects bytes to follow valid UTF-8 patterns. The byte 0x9c by itself is not valid in UTF-8, so decoding fails immediately.

That does not mean the file is corrupted. It often means the file is encoded in something like cp1252 instead of UTF-8.

Reproducing the Problem

This small example shows the same failure:

raw = b"Price list: \x9c100"
print(raw.decode("utf-8"))

Running it raises:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 12: invalid start byte

If the data was actually encoded in Windows-1252, decoding with the right codec succeeds:

raw = b"Price list: \x9c100"
print(raw.decode("cp1252"))

The First Fix: Use the Correct Encoding

If you know the source encoding, decode with that encoding explicitly.

with open("report.txt", "r", encoding="cp1252") as file:
    text = file.read()
    print(text)

That is the best fix because it preserves the intended characters. Guessing less and decoding correctly is always better than suppressing the error.

Common encodings that show up in old files:

• 'cp1252 for older Windows text,'
• 'latin-1 for permissive single-byte decoding,'
• 'utf-8-sig for UTF-8 files with a byte-order mark,'
• 'iso-8859-1 in legacy export pipelines.'

How to Investigate Unknown Data

If you are not sure what the input encoding is, inspect the bytes first.

with open("report.txt", "rb") as file:
    raw = file.read(40)
    print(raw)

Reading in binary mode keeps Python from guessing. Once you have the raw bytes, you can test likely encodings.

A simple helper can try several candidates:

1def try_decodings(raw: bytes) -> None:
2    for encoding in ["utf-8", "utf-8-sig", "cp1252", "latin-1"]:
3        try:
4            print(encoding, "=>", raw.decode(encoding))
5        except UnicodeDecodeError:
6            print(encoding, "=> failed")
7
8
9try_decodings(b"Price list: \x9c100")

latin-1 is useful diagnostically because it can decode any byte value, but that does not mean it is the right answer semantically.

When You Must Keep Going

Sometimes you only need to process a messy file without crashing. In those cases, Python lets you choose an error strategy.

with open("report.txt", "r", encoding="utf-8", errors="replace") as file:
    text = file.read()
    print(text)

Useful options include:

• 'errors="replace" to substitute invalid bytes with a replacement character,'
• 'errors="ignore" to drop invalid bytes,'
• 'errors="backslashreplace" to preserve a visible escaped form.'

These are recovery tools, not ideal fixes. They let you continue, but they may lose information.

Prefer Consistent UTF-8 at Boundaries

If you control the file generation step, the long-term solution is to write UTF-8 consistently and document it clearly.

with open("output.txt", "w", encoding="utf-8") as file:
    file.write("Price list: EUR 100")

Encoding bugs often come from hidden assumptions between systems. One program writes CP1252, another assumes UTF-8, and the failure only appears later in a pipeline.

Common Pitfalls

The most common mistake is assuming every text file is UTF-8. Many older exports from spreadsheets, desktop tools, and Windows programs are not.

Another mistake is using latin-1 as a permanent fix without checking the actual source encoding. It avoids exceptions, but it can silently produce the wrong characters.

Developers also forget to open files in binary mode while debugging. If you need to understand the real bytes, start with rb, not text mode.

Summary

• The error means the bytes are not valid UTF-8 for the way Python is decoding them.
• '0x9c often points to a legacy single-byte encoding such as cp1252.'
• The best fix is to decode using the real source encoding.
• Use binary reads and small decoding tests when the encoding is unknown.
• 'errors="replace" and similar options help with recovery, but they are not a substitute for correct encoding.'