今天爱分享给大家带来ubuntu tensorflow 1.14运行程序报错cuDNN failed to initialize【解决办法】，希望能够帮助到大家。
今天再运行程序的时候出现了下面的错误：
2020-02-12 13:06:06.483007: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. 2020-02-12 13:06:06.567509: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0 2020-02-12 13:06:06.587495: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-02-12 13:06:06.589641: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-02-12 13:06:06.592919: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-02-12 13:06:06.594236: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-02-12 13:06:06.596050: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-02-12 13:06:06.597353: E tensorflow/stream_executor/cuda/cuda_blas.cc:238] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2020-02-12 13:06:06.608829: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2020-02-12 13:06:06.611414: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2020-02-12 13:06:06.612753: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR Traceback (most recent call last): File "train_mspeech.py", line 47, inms.TrainModel(datapath, epoch = 50, batch_size = 8, save_step = 500) File "/home/eric/Documents/speech_recognition/ASRT_SpeechRecognition/SpeechModel251.py", line 179, in TrainModel self._model.fit_generator(yielddatas, save_step) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, **kwargs) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1418, in fit_generator initial_epoch=initial_epoch) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/engine/training_generator.py", line 217, in fit_generator class_weight=class_weight) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/engine/training.py", line 1217, in train_on_batch outputs = self.train_function(ins) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/home/eric/anaconda3/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/home/eric/anaconda3/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1458, in __call__ run_metadata_ptr) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_1/convolution}}]] [[ctc/ToInt32_2/_299]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node conv2d_1/convolution}}]] 0 successful operations. 0 derived errors ignored.
解决方法
注释掉下面的代码或者删除就行了，我的tensorflow-gpu是1.14，cuda10.0，cudnn7.0
config = tf.ConfigProto() config.gpu_options.per_process_gpu_memory_fraction = 0.95 #config.gpu_options.allow_growth=True #不全部占满显存, 按需分配 set_session(tf.Session(config=config))