Exception#
2023-04-03 16:05:09.410087: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at xla_ops.cc:362 : INTERNAL: RET_CHECK failure (tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:618) dnn != nullptr
2023-04-03 16:05:09.428729: E tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:417] Loaded runtime CuDNN library: 8.0.5 but source was compiled with: 8.6.0. CuDNN library needs to have matching major version and equal or higher minor version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
How to Fix#
A number of unclear bits of documentation exist for this, that seem to conflict. Install cudnn with pip and then update paths:
$ conda install -c conda-forge cudatoolkit=11.8.0
$ pip install nvidia-cudnn-cu11==8.6.0.163 # this should be compiled version
$ CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib/:$CUDNN_PATH/lib
Versions of cuDNN before 8.6 aren’t on pip. Install from nvidia’s CUDNN archive: https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#download
See also installing any version of CUDA (for CUDA toolkit)
Without Conda#
Install Prereqs#
nvidia drivers (maybe), cuda toolkit, cudnn tensorflow.org/install/source#gpu - match CUDA and CUDNN versions set CUDNN_PATH (whereis libcudnn.so.MAJOR.MINOR.REVISION) set LD_LIBRARY_PATH (usually /usr/local/cuda-VERSION or /usr/local/cuda/)
Note: This will work if running from a terminal SSH connection. For some reason running via Pycharm’s Run dialog doesn’t work correctly.