Tensorboard
Event File
Data Management
Machine Learning
Log Optimization

Tensorboard Event File Is Large and Growing

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Introduction

TensorBoard event files are append-only log files, so seeing them grow during training is normal. The real question is not whether growth is happening, but whether you are logging more data than you need for analysis, debugging, or reproducibility.

What Is Stored in an Event File

An event file contains serialized records written by TensorFlow summary APIs and related tooling. Depending on what you log, those records may include:

  • scalar metrics such as loss and accuracy
  • histograms of weights or activations
  • images, audio, text, or embeddings
  • graph metadata and profiler information

Scalars are cheap. Images and histograms are much more expensive. Profiling data can also grow quickly if it is captured repeatedly.

That is why two runs with the same model can produce very different event-file sizes even when the training duration is similar.

Why the File Keeps Growing

The file grows because a summary writer keeps appending new events as training progresses. If you log every batch, every histogram, and every sample image, the file will grow rapidly. Nothing is "wrong" in that behavior; the file is doing exactly what the writer told it to do.

In practice, the largest causes are usually:

  • logging too frequently
  • logging high-volume summary types such as images or histograms
  • training for a long time in one run directory
  • enabling profiler output more often than necessary

A useful rule is to ask whether a given summary answers a real monitoring question. If not, do not log it.

Reduce Logging Frequency in Keras

If you are using the Keras TensorBoard callback, the easiest control is logging cadence.

python
1import tensorflow as tf
2from tensorflow import keras
3
4model = keras.Sequential([
5    keras.layers.Input(shape=(10,)),
6    keras.layers.Dense(32, activation="relu"),
7    keras.layers.Dense(1)
8])
9model.compile(optimizer="adam", loss="mse")
10
11callback = keras.callbacks.TensorBoard(
12    log_dir="logs/run-001",
13    update_freq="epoch",
14    histogram_freq=0,
15    profile_batch=0,
16    write_graph=False
17)

This configuration does a few important things:

  • logs once per epoch instead of once per batch
  • disables histogram logging
  • disables profiler capture
  • skips graph writing if you do not need it

Those settings alone can reduce log size dramatically.

Manual Summary Writing Example

If you use tf.summary directly, you have even finer control. Log only every N steps instead of every step.

python
1import tensorflow as tf
2
3writer = tf.summary.create_file_writer("logs/manual-run")
4
5for step in range(1000):
6    loss = 1.0 / (step + 1)
7
8    if step % 50 == 0:
9        with writer.as_default():
10            tf.summary.scalar("loss", loss, step=step)
11            writer.flush()

For many experiments, logging every 50 or 100 steps is enough to show trend lines clearly without bloating the event file.

Split Runs Instead of Reusing One Directory Forever

Another common cause of oversized logs is reusing the same log_dir across many training sessions. TensorBoard can read that directory, but the history becomes noisy and the files grow unnecessarily.

A cleaner pattern is to create one run directory per experiment, checkpoint, or date. That makes TensorBoard easier to navigate and keeps each event file closer to the scope of a single run.

For example, use directories like logs/2026-03-11-run-a and logs/2026-03-11-run-b instead of appending to one permanent folder.

What Not to Do

Do not try to edit the binary event file by hand. If the file is too large, change your logging strategy and start a fresh run directory.

Also avoid logging image summaries or large tensors at high frequency unless you are actively debugging that exact signal. Those summary types are useful, but they are expensive.

If you need detailed debugging for one short run, enable rich summaries temporarily. For everyday training, keep the default logs lean.

Common Pitfalls

Assuming that growth means corruption is a common mistake. Event files are expected to grow while summaries are being written.

Leaving update_freq at a very chatty setting for long training jobs can produce unnecessary log volume.

Keeping histogram or profiler logging enabled after the debugging phase also wastes disk space.

Finally, reusing the same run directory for unrelated experiments makes the logs both larger and harder to interpret.

Summary

  • TensorBoard event files are append-only and normal training will make them grow
  • large files usually come from high logging frequency or heavy summary types such as images and histograms
  • reduce size by logging less often and disabling summaries you do not need
  • use separate run directories instead of writing every experiment into one log folder
  • if a file is already too large, change the logging policy and start a new run rather than editing the binary log

Related reading
Course
Intermediate
27 lessons
15 hours
DSA Fundamentals

Master algorithmic patterns and data structures through hands-on LeetCode-style problems - from arrays and hashing to dynamic programming and advanced graphs.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

All Rights Reserved.