Introduction

TensorFlow Serving can host multiple model versions from one model directory using standard Docker images. The key is directory structure and version policy configuration. With the right layout, you can run canary and stable versions side by side while controlling which versions stay loaded.

Core Sections

Required Model Directory Layout

TensorFlow Serving expects numbered subdirectories under each model name.

1/models/my_model/
2  1/
3    saved_model.pb
4    variables/
5  2/
6    saved_model.pb
7    variables/
8  3/
9    saved_model.pb
10    variables/

Each numeric folder represents one servable version.

Run Container with Model Mount

Mount the model root and pass model name environment variable.

docker run -p 8501:8501   --mount type=bind,source=/abs/path/models/my_model,target=/models/my_model   -e MODEL_NAME=my_model   tensorflow/serving

By default, serving loads all available versions unless policy limits are set.

Control Version Loading with Config File

Use model config to choose latest N versions or specific version sets.

1model_config_list {
2  config {
3    name: 'my_model'
4    base_path: '/models/my_model'
5    model_platform: 'tensorflow'
6    model_version_policy {
7      latest { num_versions: 2 }
8    }
9  }
10}

Start container with config file:

docker run -p 8501:8501   --mount type=bind,source=/abs/path/models,target=/models   --mount type=bind,source=/abs/path/models.config,target=/models/models.config   tensorflow/serving   --model_config_file=/models/models.config

Request a Specific Version

Client requests can target explicit version IDs through REST URL path.

curl -X POST http://localhost:8501/v1/models/my_model/versions/2:predict   -H "Content-Type: application/json"   -d '{"instances": [[1.0, 2.0, 3.0]]}'

For gRPC, version selection is done in model spec fields.

Operational Best Practices

Store versions immutably and avoid overwriting existing version directories. Promote new versions by adding new numeric folders and updating policy if needed. Keep model signatures consistent across versions where possible to simplify client compatibility.

Monitor load latency and memory footprint when multiple versions are active. Version policies that keep too many versions loaded can increase startup time and resource usage.

Blue-green and Canary Approach

A practical deployment style is keeping current stable plus one candidate version loaded. Route a small share of traffic to candidate version, compare metrics, then promote. If regressions appear, route back without rebuilding container images.

Multi-model and Multi-version Configuration

Production stacks often serve several models, each with its own version policy. The standard TensorFlow Serving config supports this directly.

1model_config_list {
2  config {
3    name: 'ranking_model'
4    base_path: '/models/ranking_model'
5    model_platform: 'tensorflow'
6    model_version_policy { latest { num_versions: 2 } }
7  }
8  config {
9    name: 'fraud_model'
10    base_path: '/models/fraud_model'
11    model_platform: 'tensorflow'
12    model_version_policy { specific { versions: 7 versions: 8 } }
13  }
14}

This allows independent rollout cadence per model. Keep monitoring segmented by model name and version to detect regressions accurately.

Also validate file permissions inside mounted volumes. Containers that cannot read version folders will silently fail to load expected versions.

Automated smoke tests per version endpoint are valuable before shifting production traffic. Validate model load status and basic prediction responses for each active version.

Clear model lifecycle rules also simplify rollback and incident response procedures.

Version-aware alerting and dashboards are essential for safe canary analysis in live environments.

Operational discipline around versions improves reliability and rollback speed.

Documented runbooks make emergency rollback decisions much faster.

This operational rigor improves reliability during high-pressure production events.

Common Pitfalls

• Using non-numeric version folder names, which TensorFlow Serving ignores.
• Overwriting an existing version directory and breaking immutability assumptions.
• Forgetting model config file when custom version policies are needed.
• Serving too many versions simultaneously and exhausting memory.
• Changing signature inputs between versions without client migration planning.

Summary

• Organize model versions as numeric subdirectories under one model name.
• Use standard TensorFlow Serving image with bind-mounted model paths.
• Configure version policy to load latest or specific versions intentionally.
• Target version-specific endpoints for canary or rollback workflows.
• Treat model versions as immutable deployment artifacts.