Introduction

Capturing stack traces from a live Python process is one of the fastest ways to diagnose hangs, deadlocks, and slow requests. The best method depends on whether you can change code and how much production risk you can tolerate. A strong operational setup includes both in-process and external no-code-change options.

Use faulthandler with Signals

faulthandler is built into Python and is usually the safest first option on Unix-like systems.

1import faulthandler
2import signal
3import time
4
5faulthandler.enable()
6faulthandler.register(signal.SIGUSR1, all_threads=True)
7
8print("Running. Send SIGUSR1 to dump all thread stacks.")
9while True:
10    time.sleep(1)

Trigger from another shell:

kill -USR1 <pid>

This prints stack traces without stopping the process.

Custom Stack Dumps with sys._current_frames

When you need custom formatting or routing to logs, use a signal handler with thread frame introspection.

1import os
2import signal
3import sys
4import threading
5import traceback
6
7
8def dump_stacks(signum, frame):
9    print(f"\n=== stack dump from pid {os.getpid()} ===")
10    thread_names = {t.ident: t.name for t in threading.enumerate()}
11
12    for tid, stack in sys._current_frames().items():
13        print(f"\n--- {thread_names.get(tid, tid)} ({tid}) ---")
14        traceback.print_stack(stack)
15
16
17signal.signal(signal.SIGUSR2, dump_stacks)

This is useful when incident tooling expects structured diagnostic output.

External Inspection with py-spy

If code changes are not possible, py-spy can inspect a running process externally.

py-spy dump --pid <pid>
py-spy top --pid <pid>
py-spy record -o flame.svg --pid <pid> --duration 30

Advantages:

• No code modification.
• Useful in production and containers.
• Can show both call stacks and CPU hotspots.

This is often the fastest path during urgent incidents.

Multi-Thread and Deadlock Diagnosis

For deadlock-like behavior, one stack dump is often insufficient. Capture repeated dumps over time and compare frames.

If the same lock-waiting frames repeat for many seconds, you likely have contention or deadlock. If frames change, you may be facing slow I/O or starvation rather than hard deadlock.

Practical strategy:

• Capture three to five dumps with timestamps.
• Correlate with request latency metrics.
• Map blocked threads to application subsystems.

Async Application Considerations

For async services, stack traces should be combined with event-loop diagnostics. A blocked loop thread can freeze many requests while worker threads appear idle.

Add periodic slow-task logging and loop-latency metrics so stack traces have context. Stack dumps alone can show where code is, but not always why throughput dropped.

Operational Workflow

A reliable production workflow:

1. Capture process metadata and version.
2. Dump stacks multiple times.
3. Preserve logs in incident artifact storage.
4. Annotate suspected blocking points.
5. Reproduce in staging if possible.

Version and deployment metadata are critical for later root-cause analysis.

Security and Privacy

Stack traces can expose secrets, file paths, and internal architecture details. Treat stack dumps as sensitive data:

• Restrict who can trigger dumps.
• Store dumps in protected log channels.
• Redact before sharing broadly.

Diagnostics should not bypass data-governance requirements.

Low-Risk Preparedness Step

Add one documented runbook command per service for stack dumping. During incidents, operators should not invent commands under pressure. A pre-approved procedure with signal name, expected output location, and rollback notes reduces diagnostic delay significantly.

Common Pitfalls

• Taking only one stack snapshot and drawing strong conclusions.
• Enabling overly verbose tracing permanently in production.
• Forgetting thread context and analyzing only main-thread frames.
• Capturing traces without deployment version metadata.
• Relying on one tool only, leaving no fallback when constraints change.

Summary

• Use faulthandler plus signals as the default low-friction live stack dump approach.
• Use custom handlers when structured or enriched diagnostics are needed.
• Use py-spy when code changes are not feasible.
• Capture repeated dumps with timestamps for deadlock and hang analysis.
• Treat stack traces as sensitive operational data and control access accordingly.