Introduction

This TensorFlow error happens when tf.io.decode_csv expects a different number of columns than your input row actually contains. The message often flips between expected three and expected five while debugging, which indicates inconsistent parsing assumptions between files, defaults, or delimiters. The fix is to make schema, delimiters, and selected columns fully explicit.

How decode_csv Decides Expected Field Count

record_defaults defines expected column count unless you also use select_cols. Each default corresponds to one decoded output tensor. If your raw line has a different count after splitting by delimiter rules, TensorFlow raises an error.

1import tensorflow as tf
2
3row = tf.constant("1,apple,2.5,true,2026-03-04")
4
5defaults = [
6    tf.constant([], dtype=tf.int32),
7    tf.constant([], dtype=tf.string),
8    tf.constant([], dtype=tf.float32),
9    tf.constant([], dtype=tf.bool),
10    tf.constant([], dtype=tf.string),
11]
12
13fields = tf.io.decode_csv(row, record_defaults=defaults)
14print([f.numpy() for f in fields])

In this valid example, row and defaults both represent five fields.

Why the Error Oscillates Between Three and Five

You usually see oscillating errors when one of these changes between attempts:

• A header line is being parsed as data in one run and skipped in another.
• A different delimiter is used in one stage.
• select_cols is set while defaults still assume full row size.
• Data has malformed rows with missing or extra separators.

For mixed-quality CSV files, inspect raw lines directly before decoding.

1import tensorflow as tf
2
3dataset = tf.data.TextLineDataset("sample.csv")
4for line in dataset.take(5):
5    print(line.numpy())

This quickly reveals hidden delimiters, quotes, or embedded commas.

Correct Use of select_cols

When reading only specific columns, defaults must match selected columns, not total raw columns.

1import tensorflow as tf
2
3row = tf.constant("1,apple,2.5,true,2026-03-04")
4
5# pick columns 0, 2, 4
6fields = tf.io.decode_csv(
7    row,
8    record_defaults=[
9        tf.constant([], dtype=tf.int32),
10        tf.constant([], dtype=tf.float32),
11        tf.constant([], dtype=tf.string),
12    ],
13    select_cols=[0, 2, 4],
14)
15
16print([f.numpy() for f in fields])

If select_cols length is three, defaults must also be three.

Building a Safe Input Pipeline

A practical production pipeline should:

• Skip header consistently.
• Validate minimum field count before decode.
• Log or route malformed lines.
• Keep one authoritative schema definition.

1import tensorflow as tf
2
3DEFAULTS = [
4    tf.constant([], dtype=tf.int32),
5    tf.constant([], dtype=tf.string),
6    tf.constant([], dtype=tf.float32),
7]
8
9
10def parse_line(line: tf.Tensor):
11    parts = tf.strings.split(line, sep=",")
12    tf.debugging.assert_equal(tf.size(parts), 3, message="bad csv field count")
13    return tf.io.decode_csv(line, record_defaults=DEFAULTS)
14
15
16ds = tf.data.Dataset.from_tensor_slices([
17    "1,banana,3.2",
18    "2,orange,4.1",
19])
20
21ds = ds.map(parse_line)
22for item in ds:
23    print(item)

Explicit checks make failures easier to diagnose than late training-time crashes.

Delimiter and Quoting Considerations

If your data contains quoted commas, verify CSV generator behavior and TensorFlow parsing options. Sometimes upstream export tools switch delimiters or escape styles between environments.

Normalize CSV generation in data engineering pipelines whenever possible. Machine learning preprocessing is much easier when file format contracts are strict and versioned.

When migrating schemas, roll out changes with explicit version markers in file names or metadata columns. Then branch parsing logic by version during transition windows. This avoids brittle parser heuristics and allows safe backfills when historical data still follows the old schema.

Common Pitfalls

A common pitfall is changing record_defaults count without checking all input files and all preprocessing branches. The pipeline may work for one file and fail on another.

Another issue is parsing the header row as numeric data. Skip headers intentionally with dataset transforms before decoding.

Developers also mix select_cols and full-row defaults incorrectly. Remember that selected-column decoding expects defaults only for selected outputs.

Finally, malformed rows are often ignored during local tests and then fail at scale. Add row validation and monitoring early in the pipeline.

Summary

• decode_csv field count is controlled by defaults and selected columns.
• Mismatch errors usually come from schema drift or delimiter inconsistencies.
• When using select_cols, defaults length must match selected column count.
• Validate and inspect raw lines before model preprocessing.
• Keep one strict, versioned CSV schema contract across environments.