Introduction

Running TensorFlow inference with an input queue pipeline is mainly about throughput and stability, not just prediction correctness. A model can be accurate but still underperform in production if data input stalls the compute device. This guide shows how to build a queue-based inference pipeline with tf.data, plus practical checks for correctness and latency.

Core Topic Sections

Clarify inference pipeline stages

A production inference pipeline usually has these stages:

1. Source read.
2. Parse and decode.
3. Transform and normalize.
4. Batch and prefetch.
5. Model prediction.

Treat each stage as separate and measurable. If one stage is slow, end-to-end throughput drops regardless of model speed.

Build dataset from files

For file-based inputs, tf.data.Dataset.list_files and mapping functions create a scalable queue-like pipeline.

1import tensorflow as tf
2
3IMAGE_SIZE = (224, 224)
4BATCH_SIZE = 32
5
6def parse_image(path):
7    raw = tf.io.read_file(path)
8    img = tf.image.decode_jpeg(raw, channels=3)
9    img = tf.image.resize(img, IMAGE_SIZE)
10    img = tf.cast(img, tf.float32) / 255.0
11    return img
12
13paths = tf.data.Dataset.list_files("./images/*.jpg", shuffle=False)
14ds = paths.map(parse_image, num_parallel_calls=tf.data.AUTOTUNE)
15ds = ds.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)

This queue-like dataflow overlaps I/O and compute efficiently.

Load model and run batched inference

1model = tf.keras.models.load_model("./saved_model")
2
3predictions = []
4for batch in ds:
5    p = model(batch, training=False)
6    predictions.append(p)
7
8pred_tensor = tf.concat(predictions, axis=0)
9print(pred_tensor.shape)

Batching usually improves device utilization and reduces per-item overhead.

Use model-specific preprocessing

If model was trained with specific preprocessing, reuse exactly the same transformation in inference path.

1from tensorflow.keras.applications.mobilenet_v2 import preprocess_input
2
3def parse_for_mobilenet(path):
4    raw = tf.io.read_file(path)
5    img = tf.image.decode_jpeg(raw, channels=3)
6    img = tf.image.resize(img, IMAGE_SIZE)
7    return preprocess_input(img)

Mismatch between training and inference preprocessing is a common reason for degraded predictions.

Queue sizing and latency tuning

Key controls:

1. Batch size.
2. num_parallel_calls.
3. prefetch depth.
4. Source storage latency.

Larger batch can improve throughput but increase latency. Tune based on service-level objectives.

For online inference, lower batch and tighter latency may be better. For batch inference jobs, maximize throughput.

Add deterministic ordering when needed

If output must align one-to-one with input order, disable random shuffling and keep identifiers with each record.

1def parse_with_id(path):
2    img = parse_image(path)
3    return path, img
4
5ds_ordered = paths.map(parse_with_id).batch(BATCH_SIZE)

Preserving IDs avoids mismatches when writing results downstream.

Instrument and validate pipeline performance

Track both model latency and input latency. If model time is low but end-to-end time is high, input stage is bottleneck.

You can measure per-batch duration:

1import time
2
3for batch in ds.take(5):
4    t0 = time.time()
5    _ = model(batch, training=False)
6    print("batch_ms", (time.time() - t0) * 1000)

Combine this with system metrics for I/O and CPU utilization.

Reliability considerations in production

Add guards for corrupt files and decode failures so one bad record does not kill full inference run.

1def safe_parse(path):
2    try:
3        return parse_image(path)
4    except Exception:
5        return tf.zeros((224, 224, 3), dtype=tf.float32)

In strict pipelines, you may prefer logging and skipping bad records rather than filling defaults.

Serving at scale with warm-up and batching control

For high-throughput services, warm model runtime before accepting traffic and tune dynamic batching windows carefully. Too much batching can hurt latency, while too little batching wastes accelerator utilization.

A practical rollout pattern is to start with conservative batch size, record latency percentiles, and then increase batch size gradually while watching timeout error rate.

Common Pitfalls

• Treating inference as model-only and ignoring input pipeline bottlenecks.
• Using preprocessing that differs from training data transformations.
• Enabling shuffle when deterministic output ordering is required.
• Choosing batch size based only on throughput while violating latency targets.
• Failing the entire job on one corrupt input record without fallback policy.

Summary

• Queue-based inference pipelines should optimize both correctness and throughput.
• Use tf.data with parallel map, batching, and prefetch for efficient input flow.
• Keep preprocessing identical to training pipeline.
• Tune batch and parallelism based on latency versus throughput goals.
• Add robust error handling and instrumentation for production reliability.