Introduction

When TensorFlow fails below Python, the Python traceback usually stops at the wrapper layer and does not explain what the native code was doing. The practical debugging workflow is to shrink the problem to a tiny Python reproduction, run that Python process under a native debugger, and use a build with symbols so the C or C++ stack frames are readable.

Start with the Smallest Python Repro

Do not begin with a full training pipeline. If the crash takes minutes to reproduce, every debugger cycle becomes painful. The first job is to reduce the failure to the smallest Python script that still enters the same native path.

1import tensorflow as tf
2
3@tf.function
4def step(x):
5    return tf.linalg.matmul(x, x)
6
7x = tf.ones((2, 2))
8print(step(x))

Even if your real bug is deeper than this toy example, the principle is the same: make the repro minimal, deterministic, and fast.

Use a Build with Symbols

If you are debugging TensorFlow from source or a custom op loaded by TensorFlow, symbols matter. Optimized builds inline functions, remove variable information, and make backtraces harder to read.

A debug-friendly TensorFlow build from source typically looks like this:

bazel build -c dbg --strip=never //tensorflow/tools/pip_package:build_pip_package

If the issue is in your own native extension or custom op, apply the same rule there: compile with debug symbols and avoid aggressive optimization while diagnosing the fault.

Run the Python Process Under a Native Debugger

The Python process is the process executing the native code, so that is what you need to debug.

With gdb:

gdb --args python repro.py

Useful initial commands:

1set breakpoint pending on
2catch throw
3run
4bt

With lldb:

lldb -- python repro.py

Then:

run
bt

Once the crash or thrown exception happens, the native stack trace becomes much more informative than the Python traceback alone.

Break in the Native Code That Matters

A common waste of time is stepping through wrapper glue when the real bug is in one kernel, allocator, or custom op implementation. If you know the likely native function, break there directly.

For example, with a custom op:

break MyCustomOp::Compute
run

If you are debugging a lower-level TensorFlow path and do not yet know the exact failing function, breaking on a higher native entry point can still help narrow the path.

The goal is not to understand every wrapper layer. The goal is to get to the code that owns the bug.

Use Python to Provide Context

Python is still useful during this process. Before entering the suspected native path, log the operation name, input shapes, and any device placement assumptions. That gives you enough context to connect a friendly Python call to the native frames you see in the debugger.

The division of labor is:

• Python creates the repro and context
• native tooling finds the actual fault

That mindset is much more effective than expecting the wrapper layer to provide complete diagnostics.

Logging Can Narrow the Search

Debugger work is easier if you already know roughly which operation fails. Native TensorFlow logging can help narrow that down.

export TF_CPP_MIN_LOG_LEVEL=0
python repro.py

This is not a substitute for breakpoints, but it often tells you the last significant operation before the process aborts or throws.

Custom Ops Need the Same Discipline

If the failure is in a custom operation, treat it like any other native extension problem. Build the custom op with symbols, load only that op in a tiny Python script, and break in the kernel method or helper function you own.

That is usually much faster than trying to step from Python all the way through unrelated TensorFlow internals.

Debugging Strategy Matters More Than SWIG Details

People often focus too much on the wrapper technology itself. Whether the interface layer is SWIG-based or not, the operational debugging process is mostly the same:

1. reduce the Python repro
2. use a debug build with symbols
3. run Python under a native debugger
4. break in the native code path that likely owns the bug
5. inspect the native stack and local state

That is the repeatable workflow.

Common Pitfalls

• Trying to debug the full training job instead of a tiny deterministic repro.
• Using an optimized build and expecting meaningful native stack traces.
• Attaching the debugger too late, after the interesting context is gone.
• Spending too much time stepping through wrapper glue instead of breaking in the relevant native function.
• Relying on Python tracebacks alone for a crash that is actually native.

Summary

• To debug TensorFlow below Python, run the Python process under a native debugger.
• Reduce the failure to the smallest reproducible Python script you can.
• Use a build with symbols so backtraces and breakpoints are useful.
• Break in the native code that actually matters, especially custom ops you control.
• Let Python define the repro, but use native tools to find the real root cause.