Get Output From the logging Module in IPython Notebook
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
Python logging inside a Jupyter or IPython notebook often behaves differently from a normal script because the kernel stays alive across multiple cell runs. That means handlers can accumulate, basicConfig() may appear to do nothing, and messages can vanish or duplicate. A good notebook setup creates one explicit logger and one explicit stream handler and then reuses them.
Why Logging Feels Strange in Notebooks
In a normal script, the process starts fresh each run. In a notebook, the kernel persists. If you configure logging in one cell, then run another cell that configures it again, you may end up with:
- duplicate log lines from multiple handlers
- no visible change because
basicConfig()only configures once - mixed formatting from old state left in memory
That is why notebook logging should be more explicit than script logging.
A Clean Minimal Setup
This pattern works well in notebooks:
Important choices here:
- '
handlers.clear()prevents duplicate output when the cell is rerun' - '
sys.stdoutsends the output into the notebook cell stream' - '
propagate = Falseavoids double logging through ancestor loggers'
Why basicConfig() Often Disappoints
People often try:
This can work once, but later notebook runs may not change anything because basicConfig() is intentionally conservative. If handlers already exist, it does very little.
In modern Python, you can force it:
force=True is helpful in notebooks because it resets previous logging setup.
Capturing Logs from Your Own Modules
If you are testing a module inside a notebook, create a named logger in that module:
Then in the notebook, configure the root logger or the specific namespace you care about:
That lets imported code send logs into notebook output cleanly.
Logging to a File as Well
For longer notebook sessions, file logging is useful because cell output is easy to lose.
You can combine a StreamHandler and a FileHandler if you want both interactive output and a persistent log file.
Common Notebook Debugging Pattern
For exploratory work, this setup is practical:
- clear handlers
- attach a stream handler to
sys.stdout - use a readable short formatter
- keep
INFOfor normal work and switch toDEBUGonly when needed
That keeps notebook output readable without turning every cell into a wall of framework noise.
Common Pitfalls
- Re-running logging setup cells without clearing old handlers.
- Expecting
basicConfig()to reconfigure logging after the first run. - Logging to
stderrand then wondering why notebook output ordering looks odd. - Letting messages propagate to the root logger and getting duplicates.
- Mixing library logger configuration and root logger configuration without a plan.
Summary
- Notebook kernels keep logging state alive across cell runs.
- Use explicit loggers and handlers instead of relying blindly on
basicConfig(). - Clear handlers before reconfiguring to avoid duplicate output.
- Send logs to
sys.stdoutfor predictable notebook display. - Use
force=TruewithbasicConfig()when you really want a reset.
Related reading
- Get pandas.read_csv to read empty values as empty string instead of nan
- Get statistics for each group such as count, mean, etc using pandas GroupBy?
- Get the label mappings from label encoder
- Get the rows which have the max value in groups using groupby
- Get replica set of the deployment
- Get Service selectors with K8s Python client
- Get the current git hash in a Python script
- Get the data received in a Flask request

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.