Keras flowFromDirectory get file names as they are being generated
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.
Introduction
When flow_from_directory yields batches, it returns image tensors and labels, but it does not return the file names as a third value by default. The useful detail is that the iterator already knows every path it will load, so you can recover batch file names if you understand how the generator indexes its samples.
What flow_from_directory Returns
ImageDataGenerator.flow_from_directory(...) creates a DirectoryIterator. That iterator scans the directory tree once, builds a stable list of samples, and then yields batches of images and labels.
A basic setup looks like this:
The important point is that the iterator already stores metadata about all discovered files. In many Keras versions, you can inspect:
- '
generator.filenames' - '
generator.filepaths' - '
generator.classes'
If you only want the full list once, that is enough:
Getting File Names for the Current Batch
The trickier case is matching file names to the batch you just pulled with next(generator). The simplest way is to disable shuffling and derive the slice from the batch position.
This works because the generator iterates deterministically when shuffle=False.
What Happens When shuffle=True
Once shuffling is enabled, the simple slice approach is no longer enough. The batch order is driven by an index array rather than the original file order.
In that case, a practical solution is to subclass the iterator so you can capture the exact index_array used for each batch:
That gives you the actual file paths associated with the batch currently being generated.
When You Only Need Predictions Paired with Names
A very common use case is inference rather than training. In that situation, the cleanest answer is often:
- set
shuffle=False - run prediction
- pair outputs with
generator.filepaths
That is usually better than trying to print file names during every training step.
A Note on Current Keras Practice
flow_from_directory is still common in older codebases, but many newer Keras workflows prefer image_dataset_from_directory or custom tf.data pipelines. Those APIs can be easier to extend when you need sample metadata such as file paths. Still, if you are already using flow_from_directory, the iterator metadata is enough for most filename-tracking needs.
Common Pitfalls
- Forgetting that
shuffle=Truebreaks the assumption that a batch maps to a straight slice offilepaths. - Using
generator.filenameswhen you actually need full paths fromgenerator.filepaths. - Relying on
batch_indexmath without testing what happens at epoch boundaries. - Logging source file names during heavily augmented training without remembering that the tensor is transformed while the path is not.
- Adding per-batch filename plumbing when a simple
shuffle=Falseprediction pass would solve the real problem more cleanly.
Summary
- '
flow_from_directorydoes not yield file names by default, but the iterator stores them internally' - Use
generator.filepathsorgenerator.filenameswhen you only need the sample list - With
shuffle=False, you can map a batch to a slice of file paths - With
shuffle=True, a subclass or wrapper is the safest way to capture the current batch paths - For prediction workflows, pairing
model.predict(...)withgenerator.filepathsis often the cleanest design
Related reading
- Keras GaussianNoise layer no effect?
- Keras gives nan when training categorical LSTM sequence-to-sequence model
- Keras How should I prepare input data for RNN?
- Keras How to feed input directly into other hidden layers of the neural net than the first?
- Keras get labels name of pre-trained models on imagenet
- Keras history not accessible for loss or accuracy
- Keras How to get layer shapes in a Sequential model
- Keras how to get tensor dimensions inside custom loss?
.png&w=3840&q=75)
Tackling System Design Interview Problems
A short course that equips you with the skills to approach system design interviews methodically.
Start the free courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
ML System Design practice on Codemia
Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.