Introduction

When a Google Cloud ML Engine or legacy AI Platform batch prediction job appears to use the wrong Python version, the root cause is usually deployment metadata rather than the batch input itself. The fix is to align the model version runtime, custom prediction package, and local build environment so they all target the same interpreter.

Where the Python Version Actually Comes From

A common mistake is assuming the service will infer Python from the machine that created the package. It does not work that way. In legacy AI Platform, the runtime used for online or batch prediction is chosen by the model version settings you deploy.

That means three different environments may exist at the same time:

• your local development interpreter
• the environment used to package custom code
• the cloud runtime attached to the deployed model version

If any one of those differs, batch prediction can fail with import errors, syntax errors, or odd library incompatibilities.

You can inspect the deployed version directly:

gcloud ai-platform versions describe v1 \
  --model=my_model

Look for the runtime version and Python version fields. Those are the settings that matter when the job executes.

Typical Symptoms of a Mismatch

The error message depends on how the mismatch shows up. A Python 3 package deployed into a Python 2.7 runtime often fails immediately on syntax such as f-strings or keyword-only arguments. A more subtle mismatch happens when the code imports cleanly but a dependency wheel was built for another interpreter.

For example, this file is valid only on Python 3.6+:

def preprocess(record):
    name = record["name"]
    return f"user:{name}"

If the runtime is older, the deployment may succeed but prediction will fail as soon as the module is imported. The same pattern appears with packages compiled against the wrong Python ABI.

Deploy the Version with an Explicit Runtime

Do not rely on defaults. Set the runtime and Python version deliberately when you create the model version.

1gcloud ai-platform versions create v1 \
2  --model=my_model \
3  --origin=gs://my-bucket/model-export \
4  --runtime-version=2.11 \
5  --python-version=3.7

The exact valid combinations depend on the platform generation and available runtimes, but the operational rule stays the same: choose a supported runtime that matches the code you are shipping.

If you update the package without updating the runtime settings, the cloud service will continue using whatever version is attached to the deployed model version.

Build and Test with the Same Interpreter

Before redeploying, recreate the target runtime locally. This catches obvious incompatibilities in minutes instead of after another failed batch job.

1python3.7 -m venv .venv37
2source .venv37/bin/activate
3python --version
4pip install --upgrade pip
5pip install -r requirements.txt
6python -c "import predictor; print('import ok')"

That final import test matters. If your custom prediction module cannot even import in the matching interpreter, the batch prediction job will not work in the cloud either.

If you package code with setup.py, make the requirement explicit:

1from setuptools import setup, find_packages
2
3setup(
4    name="my_predictor",
5    version="0.1.0",
6    packages=find_packages(),
7    python_requires=">=3.7,<3.8",
8)

This does not force the cloud runtime by itself, but it documents the requirement and helps catch local packaging mistakes earlier.

Check Runtime Compatibility as a Set

In practice, Python version is only one part of the compatibility set. Batch prediction depends on all of the following lining up:

• Python version
• TensorFlow or framework version
• custom prediction code
• packaged dependencies
• saved model artifact format

If your exported model was trained with one framework version and the deployed runtime expects another, the failure may look like a Python problem when it is actually a larger compatibility issue. That is why a disciplined debugging flow is better than changing random settings.

Use a minimal checklist:

1. describe the deployed model version
2. note the runtime and Python settings
3. recreate that interpreter locally
4. install the same dependencies
5. run an import or sample prediction locally
6. rebuild and redeploy only after the local environment matches

Rebuild the Package Instead of Patching Around It

If the artifact was built under the wrong interpreter, start from a clean environment and rebuild it. Mixed environments are a common source of confusing failures because cached wheels or copied site-packages content can hide the real problem.

A clean rebuild flow looks like this:

1rm -rf dist build .venv37
2python3.7 -m venv .venv37
3source .venv37/bin/activate
4pip install --upgrade pip setuptools wheel
5python setup.py sdist

Upload the rebuilt artifact and redeploy the version with the matching runtime. That approach is slower than guessing once, but faster than failing repeatedly for unclear reasons.

Common Pitfalls

The most common mistake is checking only the local machine and assuming the cloud runtime will match it. Another is updating dependencies or model code while leaving the deployed model version on an older Python setting. Teams also sometimes verify training compatibility but forget that custom prediction code has its own packaging and import requirements. A final trap is trying to patch a broken build artifact in place instead of recreating the package from a clean virtual environment that matches the target runtime.

Summary

• Batch prediction uses the runtime configured on the deployed model version, not your local Python interpreter.
• Inspect the model version first with gcloud ai-platform versions describe.
• Set runtime and Python values explicitly when deploying a version.
• Recreate the same interpreter locally and test imports before redeploying.
• Treat Python, framework version, dependencies, and model format as one compatibility set.