Introduction

Serving TensorFlow predictions through an API requires more than loading a model and exposing one route. Reliable deployment also needs stable preprocessing, versioned artifacts, health checks, and predictable response formats. This guide shows a practical path using SavedModel plus a FastAPI service.

Export a Stable TensorFlow Model

Train your model and export it in SavedModel format so serving code can load it consistently.

1import tensorflow as tf
2
3# simple demo model
4model = tf.keras.Sequential([
5    tf.keras.layers.Input(shape=(4,)),
6    tf.keras.layers.Dense(16, activation='relu'),
7    tf.keras.layers.Dense(3, activation='softmax'),
8])
9
10model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
11
12# mock training data
13x = tf.random.normal((200, 4))
14y = tf.random.uniform((200,), minval=0, maxval=3, dtype=tf.int32)
15model.fit(x, y, epochs=2, verbose=0)
16
17model.save('exported/model/1')

Version directories like model/1, model/2, and so on make rollbacks and staged rollout easier.

Build a Prediction API with FastAPI

Load the model once at startup, validate input schema, and return structured outputs.

1from fastapi import FastAPI, HTTPException
2from pydantic import BaseModel, Field
3import numpy as np
4import tensorflow as tf
5
6app = FastAPI()
7model = tf.keras.models.load_model('exported/model/1')
8
9class PredictRequest(BaseModel):
10    features: list[float] = Field(min_length=4, max_length=4)
11
12class PredictResponse(BaseModel):
13    predicted_class: int
14    probabilities: list[float]
15
16@app.get('/health')
17def health():
18    return {'status': 'ok'}
19
20@app.post('/predict', response_model=PredictResponse)
21def predict(req: PredictRequest):
22    try:
23        arr = np.array([req.features], dtype=np.float32)
24        probs = model(arr, training=False).numpy()[0]
25        pred = int(np.argmax(probs))
26        return PredictResponse(predicted_class=pred, probabilities=probs.tolist())
27    except Exception as exc:
28        raise HTTPException(status_code=400, detail=str(exc))

Run locally:

uvicorn app:app --host 0.0.0.0 --port 8000

Containerize for Repeatable Deployment

A container image ensures your model server runs with consistent dependencies.

1FROM python:3.11-slim
2WORKDIR /app
3COPY requirements.txt .
4RUN pip install --no-cache-dir -r requirements.txt
5COPY . .
6CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Example requirements.txt:

1tensorflow==2.15.1
2fastapi==0.110.0
3uvicorn==0.27.1
4numpy==1.26.4
5pydantic==2.6.3

Build and run:

docker build -t tf-api:latest .
docker run -p 8000:8000 tf-api:latest

Operational Hardening for Production

Production serving needs guardrails beyond correctness.

• enforce strict request schemas and size limits
• add request logging with latency and model version
• export metrics for success rate and p95 latency
• include readiness checks that verify model is loaded
• keep model preprocessing identical to training

Consider loading model metadata from a config file so deployments can switch versions without code edits.

MODEL_PATH = 'exported/model/1'
MODEL_VERSION = '1'

Return MODEL_VERSION in prediction responses for traceability.

Alternative: TensorFlow Serving Endpoint

If you need high throughput and model lifecycle tooling, TensorFlow Serving is a strong option. Your API gateway can call its REST endpoint and keep business logic in a separate service layer.

docker run -p 8501:8501   --mount type=bind,source=$(pwd)/exported/model,target=/models/model   -e MODEL_NAME=model tensorflow/serving

Then call:

curl -X POST http://localhost:8501/v1/models/model:predict   -H 'Content-Type: application/json'   -d '{"instances": [[0.2, -0.1, 0.4, 1.2]]}'

Common Pitfalls

A common mistake is running training preprocessing and serving preprocessing differently. Even small mismatch can degrade prediction quality dramatically.

Another issue is reloading model for each request. Always load once on startup to avoid high latency and memory churn.

A third issue is no timeout and retry policy in upstream clients. Model APIs can spike during deployments and need client side resilience.

Finally, avoid unversioned model artifacts in production. Versioning is essential for rollback, auditability, and incident response.

Automate deployment checks so schema, model path, and health probes stay synchronized.

Summary

• Export TensorFlow models as versioned SavedModel artifacts
• Serve predictions through validated API contracts with clear response schemas
• Containerize for reproducible deployment across environments
• Add health checks, metrics, and model version traceability
• Keep preprocessing consistent and versioned to maintain prediction quality