Introduction

TensorFlow Serving can host more than one model in a single server process, which is useful when you want separate endpoints for different tasks or versions. The usual setup is to store each model in its own directory and point TensorFlow Serving at a configuration file that lists all the models it should load.

How multi-model serving is organized

TensorFlow Serving expects models in the SavedModel directory format:

1/models/
2  sentiment/
3    1/
4      saved_model.pb
5      variables/
6  recommender/
7    1/
8      saved_model.pb
9      variables/

Each top-level directory is a model name, and each numbered subdirectory is a version. The version numbers let you roll forward without replacing files in place.

Use a model config file

For multiple models, you usually do not start the server with just one --model_name and --model_base_path. Instead, use --model_config_file.

Example models.config:

1model_config_list {
2  config {
3    name: "sentiment"
4    base_path: "/models/sentiment"
5    model_platform: "tensorflow"
6  }
7  config {
8    name: "recommender"
9    base_path: "/models/recommender"
10    model_platform: "tensorflow"
11  }
12}

That tells the server to load both models and expose them under separate names.

Running with Docker

A common deployment pattern is to mount both the model directory and the config file into the container:

1docker run -p 8501:8501 \
2  -v "$PWD/models:/models" \
3  -v "$PWD/models.config:/models/models.config" \
4  tensorflow/serving \
5  --rest_api_port=8501 \
6  --model_config_file=/models/models.config

Once the container starts, TensorFlow Serving loads all configured models it can find on disk.

Calling each model over REST

Each model gets its own URL path.

Example request for the sentiment model:

1curl -X POST http://localhost:8501/v1/models/sentiment:predict \
2  -H "Content-Type: application/json" \
3  -d '{
4    "instances": [[0.1, 0.2, 0.3]]
5  }'

Example request for the recommender model:

1curl -X POST http://localhost:8501/v1/models/recommender:predict \
2  -H "Content-Type: application/json" \
3  -d '{
4    "instances": [[1, 42, 7]]
5  }'

The important detail is that the endpoint path contains the model name from the config file.

Serving multiple versions of one model

You can also keep more than one version of the same model under a single model name. TensorFlow Serving can choose the latest version automatically or be configured to serve specific versions depending on your needs.

Directory example:

/models/sentiment/
  1/
  2/

That is different from serving multiple named models. One model name can have many versions, while a multi-model server can expose many model names.

When one server is a good idea

Serving multiple models in one process is convenient when:

• the traffic level is moderate
• the models are related and deployed together
• operational simplicity matters more than strict isolation

If the models have very different memory usage, scaling requirements, or release schedules, separate serving instances are often cleaner. Multi-model serving is convenient, but it also couples resource usage and failure domains.

Common Pitfalls

• Using --model_name and --model_base_path when you actually need a multi-model config file.
• Forgetting the required directory structure with version-number subfolders.
• Mounting the model data into Docker but not mounting the config file correctly.
• Calling the wrong REST path and assuming the model failed to load.
• Packing too many large models into one server and running into memory pressure.

Summary

• Use a model_config_file to serve multiple TensorFlow models from one TensorFlow Serving instance.
• Store each model under its own directory with version-number subdirectories.
• Mount the models and config file into the serving container.
• Each configured model is exposed under its own REST or gRPC endpoint.
• Multi-model serving is convenient, but separate instances may be better when models need isolation.