Introduction

In TensorFlow input pipelines, text often enters the graph as UTF-8 encoded byte strings. If you need actual Unicode code points inside the graph, the correct tool is not decode_raw; it is TensorFlow’s Unicode string ops such as tf.strings.unicode_decode, which understand text encodings instead of treating the bytes as arbitrary binary data.

Understand the Difference Between Bytes and Text

A TensorFlow string tensor stores byte sequences. That is fine for filenames, raw text records, or serialized examples, but it is not the same thing as a tensor of characters or code points.

For example, the text "hé" in UTF-8 is more than two plain ASCII bytes. If you decode it incorrectly as raw binary, you will get byte values rather than Unicode characters.

That is why this distinction matters:

• byte decoding is about binary representation
• Unicode decoding is about text encoding rules

When the question is "how do I decode a Unicode string inside the graph," you want the second category.

Use tf.strings.unicode_decode

The standard TensorFlow op for this job is tf.strings.unicode_decode. It takes string tensors and returns Unicode code points.

Example in TensorFlow:

1import tensorflow as tf
2
3text = tf.constant(["héllo", "世界"])
4codepoints = tf.strings.unicode_decode(text, input_encoding="UTF-8")
5
6print(codepoints)

The result is a ragged tensor because different strings can have different numbers of characters. That is expected. Unicode-aware text operations often produce variable-length results.

If you want one string split into characters rather than integer code points, TensorFlow also provides tf.strings.unicode_split:

1import tensorflow as tf
2
3text = tf.constant(["héllo", "世界"])
4chars = tf.strings.unicode_split(text, "UTF-8")
5
6print(chars)

This is often more convenient for tokenization-style pipelines where character strings matter more than numeric code points.

Use the Op Inside a tf.data Pipeline

A common real use case is decoding text as part of a dataset pipeline:

1import tensorflow as tf
2
3dataset = tf.data.Dataset.from_tensor_slices(
4    tf.constant(["café", "naïve", "東京"])
5)
6
7dataset = dataset.map(
8    lambda x: tf.strings.unicode_decode(x, input_encoding="UTF-8")
9)
10
11for item in dataset:
12    print(item)

This keeps the decoding inside the TensorFlow data pipeline instead of bouncing text back into Python for preprocessing.

If you are maintaining older graph-style code with tf.compat.v1.Session, the same idea still applies. The op is built into the graph and evaluated later:

1import tensorflow as tf
2
3tf.compat.v1.disable_eager_execution()
4
5text = tf.constant(["café"])
6decoded = tf.strings.unicode_decode(text, input_encoding="UTF-8")
7
8with tf.compat.v1.Session() as sess:
9    print(sess.run(decoded))

The graph build phase defines the operation. The session run phase produces the values.

Do Not Use decode_raw for Text Decoding

tf.io.decode_raw is for interpreting bytes as fixed-width numeric values. It is useful for binary records, not for Unicode text.

For example:

1import tensorflow as tf
2
3raw = tf.constant(["ABC"])
4bytes_tensor = tf.io.decode_raw(raw, tf.uint8)
5print(bytes_tensor)

This returns byte values, which is fine for binary formats but wrong if your goal is text-aware character decoding. Text encoding rules such as UTF-8 require Unicode-aware ops.

That is the core mistake behind many Unicode pipeline bugs: using a binary decoder where a text decoder is required.

Common Pitfalls

The biggest mistake is assuming TensorFlow string tensors already behave like arrays of characters. They do not. They are byte strings until you explicitly decode or split them.

Another issue is using decode_raw for Unicode data. That gives you raw byte values, not characters or code points.

Developers also sometimes forget that decoded Unicode sequences are often ragged because string lengths vary. If you need dense tensors later, you may need padding or additional processing.

Finally, make sure the declared encoding matches the real data. If your pipeline says "UTF-8" but the source bytes use a different encoding, the decode step will fail or produce incorrect output.

Summary

• TensorFlow string tensors store byte sequences, not decoded characters.
• Use tf.strings.unicode_decode to turn UTF-8 or other encoded text into Unicode code points.
• Use tf.strings.unicode_split when you want character strings instead of integer code points.
• Keep Unicode decoding inside tf.data or the graph when possible.
• Avoid decode_raw for text, because it is a binary decoder rather than a Unicode-aware text decoder.