Python
JSON
UTF-8
Encoding
Programming

Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Pass ensure_ascii=False to json.dumps() to keep non-ASCII characters as readable Unicode instead of \u escape sequences. When writing that output to a file, also specify encoding="utf-8" in the open() call. These are two separate steps: one controls the JSON text representation, the other controls the byte encoding on disk.

Why json.dumps() Escapes Non-ASCII by Default

By default, json.dumps() converts every non-ASCII character to a \uXXXX escape sequence. This is not a bug. The JSON specification (RFC 8259) allows both escaped and unescaped Unicode, and the Python standard library defaults to the escaped form because it produces output that is safe to transmit through systems that only support ASCII.

python
1import json
2
3data = {"greeting": "こんにちは", "city": "Munchen"}
4print(json.dumps(data))

Output:

text
{"greeting": "こんにちは", "city": "München"}

The data is not corrupted. Any conforming JSON parser will decode こんにちは back to the original Japanese text. But the escaped form is difficult for humans to read and inflates file size.

The Fix: ensure_ascii=False

Setting ensure_ascii=False tells the JSON encoder to emit actual Unicode characters instead of escape sequences.

python
1import json
2
3data = {"greeting": "こんにちは", "city": "Munchen"}
4print(json.dumps(data, ensure_ascii=False))

Output:

text
{"greeting": "こんにちは", "city": "München"}

The output is still valid JSON. The only difference is representation. Both forms decode to identical Python objects.

Writing Readable JSON to a File

json.dumps() returns a Python str object, not bytes. When you write that string to a file, you need to specify the encoding. If you do not, Python uses the system default encoding, which may not be UTF-8 on all platforms (notably Windows, where the default is often cp1252).

python
1import json
2
3data = {
4    "product": "Kaffee",
5    "origin": "Sao Paulo",
6    "notes": "Excellent quality roast"
7}
8
9# Correct: both ensure_ascii and encoding specified
10with open("products.json", "w", encoding="utf-8") as f:
11    json.dump(data, f, ensure_ascii=False, indent=2)

Note the use of json.dump() (no s) when writing directly to a file handle. This is equivalent to calling json.dumps() and then f.write(), but slightly more efficient because it avoids creating the entire string in memory for large objects.

What Happens When You Forget encoding="utf-8"

python
1# On Windows with default cp1252 encoding:
2with open("data.json", "w") as f:
3    json.dump({"text": "日本語"}, f, ensure_ascii=False)
4# Raises UnicodeEncodeError: 'charmap' codec can't encode characters

This error only manifests on systems where the default encoding is not UTF-8. On Linux and macOS (which default to UTF-8), the code works without the explicit encoding parameter, but adding it is still the right practice for portable code.

Reading the File Back

When reading a UTF-8 JSON file, specify the same encoding:

python
1import json
2
3with open("products.json", "r", encoding="utf-8") as f:
4    data = json.load(f)
5
6print(data["product"])  # Kaffee
7print(type(data["product"]))  # <class 'str'>

If the file was written with ensure_ascii=True (the default), you do not need encoding="utf-8" to read it because the file contains only ASCII characters. But specifying UTF-8 on both read and write is a safe default that handles both cases.

Sending JSON as Bytes Over the Network

When transmitting JSON over HTTP, WebSockets, or message queues, you typically need bytes, not a Python string. Encode the string explicitly:

python
1import json
2
3payload = {
4    "currency": "EUR",
5    "symbol": "€",  # Euro sign in source code
6    "description": "European currency"
7}
8
9# Step 1: JSON string with readable Unicode
10text = json.dumps(payload, ensure_ascii=False)
11print(text)
12# {"currency": "EUR", "symbol": "€", "description": "European currency"}
13
14# Step 2: Encode to UTF-8 bytes for transport
15body = text.encode("utf-8")
16print(body)
17# b'{"currency": "EUR", "symbol": "\xe2\x82\xac", ...}'
18print(len(body))  # Byte length, not character length

Many HTTP libraries (like requests) handle this encoding automatically when you pass a dict or string. But when working with lower-level APIs (sockets, raw WSGI), you need to manage the encoding yourself.

Combining with Other json.dumps() Options

ensure_ascii=False works with all other json.dumps() parameters:

python
1import json
2
3data = {
4    "users": [
5        {"name": "Amelie", "city": "Paris"},
6        {"name": "Takeshi", "city": "Tokyo"},
7        {"name": "Hans", "city": "Zurich"}
8    ]
9}
10
11output = json.dumps(
12    data,
13    ensure_ascii=False,
14    indent=2,
15    sort_keys=True,
16    separators=(",", ": ")
17)
18print(output)

Output:

text
1{
2  "users": [
3    {"city": "Paris","name": "Amelie"},
4    {"city": "Tokyo","name": "Takeshi"},
5    {"city": "Zurich","name": "Hans"}
6  ]
7}

Comparison Table

Scenarioensure_asciiFile encodingResult
Default behaviorTrue (default)System defaultASCII-safe JSON, \u escapes for non-ASCII
Readable file outputFalse"utf-8"Human-readable Unicode, portable across platforms
Network transportFalseN/A (use .encode("utf-8"))Compact UTF-8 bytes
Legacy system compatibilityTrue (default)AnySafe for ASCII-only systems
Windows file output without encodingFalseSystem default (cp1252)UnicodeEncodeError for CJK/emoji characters

Handling Special Cases

Emoji and Characters Outside the BMP

Characters outside the Basic Multilingual Plane (like emoji) require surrogate pairs in the escaped form. ensure_ascii=False avoids this complexity entirely:

python
1import json
2
3data = {"status": "All good! \U0001f44d"}
4
5escaped = json.dumps(data)
6readable = json.dumps(data, ensure_ascii=False)
7
8print(escaped)
9# {"status": "All good! 👍"}
10
11print(readable)
12# {"status": "All good! 👍"}

Mixed Content with Already-Escaped Strings

If your input data already contains literal \u sequences as part of the string content (not as Unicode characters), ensure_ascii=False does not affect them. The backslash is a regular character in the string and will be preserved:

python
1import json
2
3# The string literally contains backslash-u, not a Unicode escape
4data = {"regex": "\\u[0-9a-f]{4}"}
5print(json.dumps(data, ensure_ascii=False))
6# {"regex": "\\u[0-9a-f]{4}"}

Custom Encoders

If you use a custom JSONEncoder subclass, ensure_ascii=False still applies to the final output:

python
1import json
2from datetime import datetime
3
4class DateEncoder(json.JSONEncoder):
5    def default(self, obj):
6        if isinstance(obj, datetime):
7            return obj.isoformat()
8        return super().default(obj)
9
10data = {"event": "Concert debut", "date": datetime(2025, 3, 15)}
11print(json.dumps(data, cls=DateEncoder, ensure_ascii=False))
12# {"event": "Concert début", ... }  -- wait, that's wrong
13# Actually: {"event": "Concert début", "date": "2025-03-15T00:00:00"}

Common Pitfalls

  • Setting ensure_ascii=False but forgetting encoding="utf-8" in the open() call. On Windows, this causes UnicodeEncodeError for any character outside the system code page.
  • Assuming \u escape sequences in JSON output mean the data is corrupted. Both the escaped and unescaped forms are valid JSON and decode to identical values.
  • Using ensure_ascii=False when writing JSON that will be consumed by an ASCII-only system. In that case, the default escaped form is the correct choice.
  • Confusing json.dumps() (returns a string) with json.dump() (writes to a file). Use dump() when writing to a file handle to avoid building the entire string in memory.
  • Calling .encode("utf-8") on the result of json.dumps() and then writing it to a text-mode file. Either write bytes to a binary-mode file ("wb") or write the string to a text-mode file ("w") with the encoding parameter. Do not mix the two.
  • Pretty-printing with indent but leaving ensure_ascii at the default. The file is readable in structure but not in content, which defeats the purpose of pretty-printing.

Summary

  • json.dumps(data, ensure_ascii=False) keeps Unicode characters readable instead of escaping them as \uXXXX.
  • Always pair ensure_ascii=False with encoding="utf-8" when writing to files for cross-platform compatibility.
  • Use json.dump() (without s) to write directly to a file handle instead of building the full string in memory.
  • Use .encode("utf-8") when you need bytes for network transport.
  • Both escaped and unescaped JSON are valid and decode to identical data. The choice is about readability and compatibility, not correctness.

Course illustration
Course illustration

All Rights Reserved.