darknet
machine learning
training issues
neural networks
troubleshooting

Training darknet finishes immediately

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

Introduction

When Darknet training exits almost immediately, the framework is usually not "done learning." It is usually telling you that the run is misconfigured before the first real training loop begins. The fastest way to debug it is to stop thinking about the model first and verify the training inputs, iteration limits, and checkpoint state.

Check Whether max_batches Has Already Been Reached

One of the most common reasons Darknet exits immediately is that you resumed from weights whose iteration count is already at or above max_batches in the .cfg file.

For example, if the config says:

ini
1[net]
2batch=64
3subdivisions=16
4max_batches=4000
5steps=3200,3600

and you start from yolo-obj_last.weights that was saved at iteration 4000 or later, Darknet has no more training to do and stops.

That is not a crash. It is a normal exit.

The fix is either:

  • raise max_batches if training should continue
  • start from earlier pretrained weights instead of the final checkpoint
  • verify that you are not accidentally resuming from the wrong file

A typical command looks like this:

bash
./darknet detector train data/obj.data cfg/yolo-obj.cfg backup/yolo-obj_last.weights

If that file is already at the configured limit, the run ends immediately.

Verify the Dataset Paths in .data

Darknet relies on the .data file to find class names, training lists, validation lists, and backup paths. If those paths are wrong, training can stop before useful work begins.

A minimal example:

ini
1classes=2
2train=data/train.txt
3valid=data/valid.txt
4names=data/obj.names
5backup=backup/

Check each path directly:

bash
1cat data/train.txt | head
2cat data/valid.txt | head
3cat data/obj.names
4ls -ld backup

If train.txt points to missing images or the backup directory does not exist or is not writable, the training run can terminate very early.

Make Sure the Training List Is Not Empty

An empty or nearly empty train.txt file is another classic cause. Darknet expects a list of image paths, one per line.

Count the entries:

bash
wc -l data/train.txt

Then verify that at least a few paths are valid:

bash
head -n 3 data/train.txt | while read -r img; do ls "$img"; done

If the files do not exist at those paths, Darknet cannot build batches correctly.

Validate Label Files and Class Counts

Even when images exist, label problems can break training quickly. For YOLO-style training, each image should have a matching .txt label file in the expected format.

The config must also match the dataset:

  • 'classes in each [yolo] layer must equal your class count'
  • the preceding [convolutional] layer must use filters=(classes + 5) * anchors_per_scale

For a two-class model with three anchors on a YOLO head, the filters should be:

text
(2 + 5) * 3 = 21

A mismatch here often produces confusing behavior later, including broken training setup or unusable checkpoints.

Test with Pretrained Convolutional Weights First

If you are unsure whether the pipeline works, start from a standard convolutional backbone rather than from a partially trained custom checkpoint.

Example:

bash
./darknet detector train data/obj.data cfg/yolo-obj.cfg darknet53.conv.74

This removes one variable from the problem. If training starts normally from pretrained backbone weights but exits when you use your custom checkpoint, the issue is probably checkpoint state rather than dataset formatting.

Watch the Console Output Closely

Darknet usually prints the reason for the early exit if you read the log carefully. Look for clues such as:

  • cannot open image or label file
  • zero classes or zero training images
  • no more iterations left
  • invalid backup path
  • CUDA or GPU initialization failure

A run that "finishes immediately" is often just a run whose real error scrolled by too quickly.

You can capture the log for inspection:

bash
./darknet detector train data/obj.data cfg/yolo-obj.cfg darknet53.conv.74 2>&1 | tee train.log

Then search for file and config errors:

bash
rg 'error|cannot|failed|open|batch|max_batches' train.log

GPU and Build Checks Still Matter

If Darknet was compiled without the expected CUDA, cuDNN, or OpenCV support, startup can fail before training begins. Verify the build flags in the Makefile and rebuild if necessary.

Typical settings include:

ini
GPU=1
CUDNN=1
OPENCV=1

Then rebuild:

bash
make clean
make

This is less common than path or checkpoint mistakes, but it is worth checking if the binary exits before any dataset parsing occurs.

Common Pitfalls

The biggest mistake is assuming an immediate exit means a mysterious Darknet bug. Most often it is a plain configuration problem such as a finished checkpoint, bad paths, or empty training data.

Another mistake is editing classes and forgetting to update the corresponding filters values in the detection heads.

Teams also often trust train.txt without verifying the paths it contains. Relative-path mistakes are extremely common when datasets are moved between machines.

Finally, do not debug by changing everything at once. Start with one known-good config, one pretrained backbone, and a small verified dataset slice.

Summary

  • Immediate Darknet exit usually means configuration or dataset state, not successful training completion.
  • First check whether the checkpoint has already reached max_batches.
  • Verify the .data file, train.txt, image paths, labels, and backup directory.
  • Make sure classes and YOLO filters values match.
  • Capture the log and search it before assuming the framework failed silently.

Related reading
Free course
Beginner
7 lessons
2 hours
Tackling System Design Interview Problems

A short course that equips you with the skills to approach system design interviews methodically.

Start the free course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

ML System Design practice on Codemia

Design recommenders, ranking systems and training pipelines the way ML interviews actually ask for them, with worked solutions.

Practice ML system design

All Rights Reserved.