Building TensorFlow from source can be a rewarding endeavor, allowing you to customize the build for specific requirements and better optimize it for the hardware you possess. However, it's not uncommon to encounter errors during this process, especially when dealing with CUDA-related libraries such as libcudart.so and libcudnn. This article aims to guide you through understanding these libraries, potential errors, and how to resolve them.

Understanding libcudart.so and libcudnn

libcudart.so

libcudart.so is the CUDA Runtime library, a critical component provided by NVIDIA for utilizing CUDA capabilities. It is essential for executing CUDA programs and must be correctly installed and found on the library path for TensorFlow builds that leverage GPU acceleration.

libcudnn

libcudnn is the NVIDIA CUDA Deep Neural Network (cuDNN) library, which provides highly tuned implementations for standard routines such as forward and backward convolutions, pooling, normalization, and activation layers. It's crucial for optimizing deep learning operations on NVIDIA GPUs.

Common Errors

When building TensorFlow from source, common errors related to these libraries include:

• Library Not Found: The build process cannot locate libcudart.so or libcudnn.
• Version Mismatch: The libraries found do not match the versions expected by the TensorFlow source code.
• Path Configuration Issues: Incorrect or incomplete environment path configurations leading to build errors.

Example Error Messages

1. Library Not Found:

   ERROR: /tensorflow/stream_executor/BUILD:112:1: error while parsing .d file: /usr/local/cuda/lib64/libcudart.so: No such file or directory

1. Version Mismatch:

   ERROR: /tensorflow/core/BUILD:3700:1: error: incompatible library version: cudnn version is '7.0', expecting '8.0' or later.

Troubleshooting Steps

To resolve these issues, consider the following steps:

1. Verify Installation

Ensure that CUDA and cuDNN are correctly installed. You can check the existence of libraries using:

ls /usr/local/cuda/lib64/libcudart.so
ls /usr/local/cuda/lib64/libcudnn.so

2. Verify Environment Variables

Ensure environment variables are properly set. This includes:

• CUDA_HOME: Should point to the CUDA installation directory.
• LD_LIBRARY_PATH: Should include paths to both CUDA and cuDNN libraries, e.g., /usr/local/cuda/lib64.

Example:

export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

3. Install Correct Versions

Make sure the versions of CUDA and cuDNN match the requirements of the TensorFlow version you are building. TensorFlow's build documentation or configuration files should specify compatible versions. Upgrade or downgrade them if necessary.

4. Symbolic Links

If versions are correct but builds fail due to missing specific library names, create symbolic links. For instance:

sudo ln -s /usr/local/cuda/lib64/libcudart.so.<version> /usr/local/cuda/lib64/libcudart.so

5. Ensure Proper Permissions

Ensure that the current user has read permissions for the CUDA directory and libraries. You may need to adjust permissions or run the build command with sudo if permission issues are suspected.

Examples and Use Cases

To illustrate, consider a scenario where upgrading TensorFlow requires a specific libcudart.so version that is not present:

1. Check Current Version:

   cuda-install-samples-<version> linux -version

1. Download Required CUDA Version from NVIDIA's official site.
2. Update Environment Variables and verify the paths.
3. Rebuild TensorFlow after ensuring that the library paths and versions are correct.

Summary Table

Additional Considerations

• Consider Docker: Using pre-built Docker images for TensorFlow with GPU support can circumvent these source build issues by packaging compatible CUDA and cuDNN versions.
• Use Virtual Environments: If using Python virtual environments, ensure that Python and builders are using the same environment context.

In conclusion, while building TensorFlow from source can lead to errors related to libcudart.so and libcudnn, understanding these libraries and following the outlined steps can effectively troubleshoot and resolve the issues, paving the way for a successful build and optimized TensorFlow performance on your hardware.