Introduction

Quantization reduces the precision of weights and, in some modes, activations so a model is smaller and faster on supported hardware. In the current TensorFlow workflow, quantizing a Keras model usually means converting it for TensorFlow Lite, then choosing between post-training quantization and quantization-aware training based on how much accuracy you can afford to lose.

Start with Post-Training Quantization

The easiest path is to train a normal Keras model, save it, and quantize during TFLite conversion. Dynamic-range quantization is the lightest-weight option because it mainly quantizes weights and requires no retraining.

1import tensorflow as tf
2from tensorflow import keras
3
4model = keras.Sequential([
5    keras.layers.Input(shape=(16,)),
6    keras.layers.Dense(32, activation="relu"),
7    keras.layers.Dense(1, activation="sigmoid"),
8])
9
10model.compile(optimizer="adam", loss="binary_crossentropy")
11model.save("saved_model.keras")
12
13converter = tf.lite.TFLiteConverter.from_keras_model(model)
14converter.optimizations = [tf.lite.Optimize.DEFAULT]
15quantized_tflite_model = converter.convert()
16
17with open("model_dynamic.tflite", "wb") as f:
18    f.write(quantized_tflite_model)

This is the lowest-friction way to shrink model size. It often gives a useful improvement with little code, but it is not the strongest option for integer-only inference.

Full Integer Quantization Needs a Representative Dataset

If you want both weights and activations quantized for better CPU efficiency or integer-only accelerators, provide a representative dataset during conversion.

1import numpy as np
2import tensorflow as tf
3
4
5def representative_dataset():
6    for _ in range(100):
7        sample = np.random.rand(1, 16).astype("float32")
8        yield [sample]
9
10
11converter = tf.lite.TFLiteConverter.from_keras_model(model)
12converter.optimizations = [tf.lite.Optimize.DEFAULT]
13converter.representative_dataset = representative_dataset
14converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
15converter.inference_input_type = tf.int8
16converter.inference_output_type = tf.int8
17
18quantized_tflite_model = converter.convert()

The representative dataset does not train the model again. It calibrates activation ranges so the converter can choose sensible integer scales.

Float16 Quantization Is a Good Middle Ground

If you target hardware that handles half precision well, float16 quantization can shrink model size with minimal accuracy loss.

1converter = tf.lite.TFLiteConverter.from_keras_model(model)
2converter.optimizations = [tf.lite.Optimize.DEFAULT]
3converter.target_spec.supported_types = [tf.float16]
4
5float16_tflite_model = converter.convert()

This is often a good choice when model size matters more than pure integer execution.

Use Quantization-Aware Training When Accuracy Drops Too Much

If post-training quantization causes too much degradation, train the model with fake quantization effects in the graph. TensorFlow Model Optimization Toolkit provides this workflow.

1import tensorflow_model_optimization as tfmot
2from tensorflow import keras
3
4base_model = keras.Sequential([
5    keras.layers.Input(shape=(16,)),
6    keras.layers.Dense(32, activation="relu"),
7    keras.layers.Dense(1, activation="sigmoid"),
8])
9
10qat_model = tfmot.quantization.keras.quantize_model(base_model)
11qat_model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

After fine-tuning the quantization-aware model, convert it to TFLite as usual. This requires more work, but it often preserves accuracy better on sensitive models.

Evaluate the Quantized Model Instead of Assuming Success

Quantization is always a tradeoff. Measure:

• model size on disk
• inference latency on target hardware
• accuracy or task-specific metrics after conversion
• operator compatibility in the target runtime

A smaller file is not enough by itself. If the quantized model breaks on unsupported ops or misses your accuracy target, it is not ready to ship.

Quantization Usually Targets Deployment, Not Training

People sometimes say they want to quantize a Keras model and then keep training it normally. That is usually the wrong mental model. The common flow is:

1. train a float model
2. quantize for deployment
3. optionally retrain with quantization-aware training if needed

Deployment format and training format are related, but they are not the same artifact.

Common Pitfalls

One common mistake is skipping the representative dataset for full integer quantization and then being surprised by poor results. Another is assuming every layer and op in the model has the same quantization support on the target runtime.

Developers also often benchmark only on a desktop machine. Quantization decisions should be validated on the actual device or hardware class where the model will run.

Finally, do not treat quantization as a guaranteed accuracy-preserving compression step. Some models tolerate it well; others need quantization-aware training.

Summary

• Quantization reduces model size and can improve inference efficiency.
• Post-training dynamic-range quantization is the easiest starting point.
• Full integer quantization requires a representative dataset.
• Float16 quantization is a useful compromise for compatible hardware.
• When accuracy drops too much, use quantization-aware training and evaluate on the real target environment.