Introduction

The character  in a Python string is a Byte Order Mark (BOM). It appears when you read a file that was saved with UTF-8 BOM encoding, and Python decodes it using standard utf-8 instead of utf-8-sig. The fix is to use encoding="utf-8-sig" when opening the file, which strips the BOM automatically. If the string is already loaded, strip the leading  character manually before processing.

What is the Byte Order Mark?

BOM (U+FEFF) is a Unicode character originally designed to indicate the byte order of a text stream. In UTF-16 and UTF-32, byte order matters because multi-byte integers can be stored in big-endian or little-endian format. BOM at the start of the stream tells the decoder which order to expect.

In UTF-8, byte order is irrelevant because UTF-8 encodes each byte independently. However, some tools still prepend a BOM to UTF-8 files as a signature to indicate that the file is UTF-8 encoded. Microsoft Notepad, Excel's CSV export, and many Windows applications do this by default.

The BOM in UTF-8 is the three-byte sequence EF BB BF. When Python decodes these bytes with the utf-8 codec, they become the single character  in the resulting string.

1# The raw bytes of a UTF-8 BOM
2bom_bytes = b'\xef\xbb\xbf'
3print(bom_bytes.decode('utf-8'))       # Prints nothing visible
4print(repr(bom_bytes.decode('utf-8'))) # ''

Why BOM Causes Bugs

BOM is invisible in most displays but is a real character in the string. This breaks string comparisons, dictionary key lookups, CSV header parsing, and JSON processing.

1# Simulating a file with BOM
2text = "name,age,city"
3
4# This comparison fails because the first character is not 'n'
5print(text == "name,age,city")        # False
6print(text.startswith("name"))        # False
7print(repr(text[0]))                  # ''
8print(len(text))                      # 14, not 13

In CSV processing, the BOM becomes part of the first column header. A column named name does not match "name" in a dictionary lookup, causing KeyError exceptions that are extremely difficult to debug because the header looks correct when printed.

Fix 1: Read Files with utf-8-sig Encoding

The utf-8-sig codec is specifically designed to handle BOM. It strips a leading BOM during decoding and is otherwise identical to utf-8. If no BOM is present, it behaves exactly like utf-8.

1# Writing a file with BOM (simulating what Excel or Notepad produces)
2from pathlib import Path
3
4test_file = Path("data.csv")
5test_file.write_bytes(b"\xef\xbb\xbfname,age\nAlice,30\nBob,25\n")
6
7# BAD: Reading with utf-8 preserves the BOM
8with open(test_file, "r", encoding="utf-8") as f:
9    header = f.readline().strip()
10print(repr(header))  # 'name,age'
11
12# GOOD: Reading with utf-8-sig removes the BOM
13with open(test_file, "r", encoding="utf-8-sig") as f:
14    header = f.readline().strip()
15print(repr(header))  # 'name,age'

This is the cleanest solution. Apply utf-8-sig at the point where the file is first opened, and all downstream code receives clean text.

Fix 2: Strip BOM from In-Memory Strings

When you receive text from an API, message queue, or database where you cannot control the encoding parameter, strip the BOM after the fact:

1def strip_bom(text: str) -> str:
2    """Remove leading BOM if present."""
3    if text.startswith(""):
4        return text[1:]
5    return text
6
7# Usage
8raw = "id,name,email"
9clean = strip_bom(raw)
10print(repr(clean))  # 'id,name,email'

Only strip from the start of the string. Using .replace("", "") globally can corrupt data in edge cases where U+FEFF appears as a zero-width no-break space within the text body (its secondary Unicode function).

Fix 3: Handle BOM in CSV Processing

CSV files from Excel are one of the most common sources of BOM issues. Here is a robust CSV reading pattern:

1import csv
2from pathlib import Path
3
4def read_csv_safe(filepath: str) -> list[dict]:
5    """Read a CSV file, handling BOM transparently."""
6    with open(filepath, "r", encoding="utf-8-sig", newline="") as f:
7        reader = csv.DictReader(f)
8        return list(reader)
9
10# Works regardless of whether the file has BOM
11rows = read_csv_safe("export.csv")
12for row in rows:
13    print(row["name"], row["age"])

For pandas users, read_csv accepts an encoding parameter:

1import pandas as pd
2
3# Option 1: Use utf-8-sig encoding
4df = pd.read_csv("export.csv", encoding="utf-8-sig")
5
6# Option 2: If the file is already loaded with BOM in headers
7df = pd.read_csv("export.csv", encoding="utf-8")
8df.columns = [col.lstrip("") for col in df.columns]

Fix 4: Handle BOM in JSON Processing

JSON files with BOM cause json.loads to fail in some Python versions:

1import json
2
3# Bytes with BOM
4raw_bytes = b'\xef\xbb\xbf{"key": "value"}'
5
6# Python 3.x json.loads handles BOM in strings but not always in bytes
7text = raw_bytes.decode("utf-8-sig")
8data = json.loads(text)
9print(data)  # {'key': 'value'}

The safest approach is always decoding bytes with utf-8-sig before passing the string to json.loads.

Detecting BOM in Files

Sometimes you need to check whether a file contains BOM before deciding how to process it:

1from pathlib import Path
2
3def has_bom(filepath: str) -> bool:
4    """Check if a file starts with UTF-8 BOM."""
5    with open(filepath, "rb") as f:
6        return f.read(3) == b"\xef\xbb\xbf"
7
8# Check multiple files
9for path in Path(".").glob("*.csv"):
10    if has_bom(str(path)):
11        print(f"BOM detected: {path}")

This is useful in data pipelines where files come from multiple sources with inconsistent encoding practices.

Removing BOM from Files on Disk

If you want to permanently remove BOM from a file:

1from pathlib import Path
2
3def remove_bom_from_file(filepath: str) -> bool:
4    """Remove UTF-8 BOM from a file if present. Returns True if BOM was removed."""
5    path = Path(filepath)
6    content = path.read_bytes()
7    if content.startswith(b"\xef\xbb\xbf"):
8        path.write_bytes(content[3:])
9        return True
10    return False

For a command-line approach:

1# Check for BOM
2hexdump -C data.csv | head -1
3# If BOM is present, the first three bytes will be: ef bb bf
4
5# Remove BOM using sed (macOS)
6sed -i '' '1s/^\xef\xbb\xbf//' data.csv
7
8# Remove BOM using sed (Linux)
9sed -i '1s/^\xef\xbb\xbf//' data.csv

Writing Files With or Without BOM

If you need to produce files with BOM for consumers that require it (such as Excel reading UTF-8 CSV files correctly on Windows):

1# Writing WITH BOM (for Excel compatibility)
2with open("for_excel.csv", "w", encoding="utf-8-sig", newline="") as f:
3    writer = csv.writer(f)
4    writer.writerow(["name", "city"])
5    writer.writerow(["Alice", "Tokyo"])
6
7# Writing WITHOUT BOM (standard UTF-8)
8with open("for_api.csv", "w", encoding="utf-8", newline="") as f:
9    writer = csv.writer(f)
10    writer.writerow(["name", "city"])
11    writer.writerow(["Alice", "Tokyo"])

Note that utf-8-sig also works for writing. It prepends the BOM bytes to the output.

Common Pitfalls

Using .replace("", "") globally. This removes every occurrence of U+FEFF in the string, not just the leading BOM. In Unicode, U+FEFF also serves as a zero-width no-break space, which can legitimately appear within text. Strip only the leading character.

Decoding bytes twice. If bytes pass through two decoding steps with different codecs, hidden characters can reappear. Decode once at the boundary, then work with strings throughout.

Assuming BOM only appears in files. Clipboard data, HTTP response bodies, database exports, and API payloads can all contain BOM. Any text crossing a system boundary should be treated as potentially BOM-prefixed.

Not including BOM in test fixtures. BOM bugs are invisible in development because developers typically create clean UTF-8 files. Add at least one test fixture that starts with b"\xef\xbb\xbf" to catch regressions in CSV, JSON, and configuration file parsers.

Forgetting the newline="" parameter with csv module. When opening files for the csv module, always pass newline="" to prevent double newline translation. This is unrelated to BOM but frequently co-occurs in CSV handling bugs.

Summary

•  is a Byte Order Mark (BOM) character. In UTF-8, it appears as the three-byte sequence EF BB BF.
• The primary fix is reading files with encoding="utf-8-sig", which strips leading BOM automatically.
• For in-memory strings, strip only the leading  character, never replace it globally.
• CSV files from Excel and Windows tools are the most common source of BOM issues in Python.
• Use utf-8-sig for writing when you need BOM output (for example, CSV files that Excel must open correctly on Windows).
• Add BOM-containing test fixtures to your test suite to catch encoding regressions early.