Introduction

TensorFlow Serving can host multiple models in one server process, which is useful for microservices, model families, and staged rollouts. The setup is stable when each model has a clean version directory and the server is driven by an explicit model config file. Good production behavior depends on naming conventions, resource limits, and observability.

Organize Model Directories Correctly

TensorFlow Serving expects SavedModel exports in versioned folders. A common layout looks like this:

1/models/
2  fraud_model/
3    1/
4    2/
5  recommendation_model/
6    1/

Each numeric directory is a version. You can keep one version for static models or several versions for controlled upgrades.

Create a Multi-Model Config File

Use model_config_list to tell TensorFlow Serving which models to load.

1model_config_list {
2  config {
3    name: "fraud_model"
4    base_path: "/models/fraud_model"
5    model_platform: "tensorflow"
6  }
7  config {
8    name: "recommendation_model"
9    base_path: "/models/recommendation_model"
10    model_platform: "tensorflow"
11  }
12}

Save this file as models.config. This approach is cleaner than relying on one default model flag.

Run TensorFlow Serving with the Config

Start the server with mounted model path and config file.

1docker run --rm -p 8501:8501 -p 8500:8500 \
2  -v "$PWD/models:/models" \
3  -v "$PWD/models.config:/models.config" \
4  tensorflow/serving:latest \
5  --model_config_file=/models.config \
6  --model_config_file_poll_wait_seconds=30

The polling flag allows config reload checks every thirty seconds, which supports non-disruptive updates.

Send Requests to Specific Models

With REST API, specify the model name in the URL path.

curl -X POST http://localhost:8501/v1/models/fraud_model:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [[0.1, 0.8, 0.3]]}'

Second model example:

curl -X POST http://localhost:8501/v1/models/recommendation_model:predict \
  -H "Content-Type: application/json" \
  -d '{"instances": [[42, 7, 3]]}'

Consistent model names are critical because API consumers depend on them.

Manage Versions Safely

When deploying a new model version, write it to a new numbered directory instead of overwriting an old version. TensorFlow Serving can shift to latest version automatically if policy allows.

If you need strict control, define version policy in config:

1model_config_list {
2  config {
3    name: "fraud_model"
4    base_path: "/models/fraud_model"
5    model_platform: "tensorflow"
6    model_version_policy {
7      specific { versions: 2 }
8    }
9  }
10}

This prevents accidental promotion of unvalidated versions.

Resource Planning for Multiple Models

Serving many models in one process can create CPU and memory contention. Start with baseline profiling for each model and set explicit host limits. If one model has high latency or memory spikes, split it to a separate serving instance.

Practical strategy:

• group models with similar latency profiles
• isolate large models from lightweight models
• monitor per-model request volume and tail latency

Multi-model serving is convenient, but isolation is often better for noisy workloads.

Health and Metadata Endpoints

Use metadata endpoints to verify loaded models and signatures.

curl http://localhost:8501/v1/models/fraud_model
curl http://localhost:8501/v1/models/recommendation_model/metadata

These endpoints are useful in deployment checks before routing live traffic.

Update Workflow in Production

A safe rollout workflow:

1. export new model as a new version directory
2. run shadow or canary traffic tests
3. update version policy or config to promote
4. monitor errors, latency, and prediction drift
5. keep rollback version available

This avoids downtime and reduces risk from bad exports.

Common Pitfalls

• Using non-versioned model folders and overwriting active artifacts.
• Running many heavy models in one instance without capacity planning.
• Exposing model names that change frequently and breaking clients.
• Promoting latest version automatically without validation gates.
• Deploying without latency and error monitoring per model.

Summary

• Use versioned SavedModel directories and explicit model config files.
• Run TensorFlow Serving with multi-model configuration for clarity and control.
• Route requests by model name through REST or gRPC endpoints.
• Manage versions with clear promotion and rollback policy.
• Monitor per-model resource usage and split workloads when contention appears.