Debugging CUDA in Matlab without having to restart

Every time I try to launch a CUDA kernel, if the kernel crashes, it seems unrecoverable and I have to restart Matlab. First, the kernel will crash and I'll get this generic error message:

Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_UNKNOWN

Not helpful certainly but fine, I'll debug my code. But the error is fatal for Matlab: On my next attempt to merely construct the kernel I get

Error using parallel.gpu.CUDAKernel
An error occurred during PTX compilation of <image>.
The information log was:
<No information>
The error log was:
<No information>
The CUDA error code was: CUDA_ERROR_UNKNOWN.

It also fails to execute a kernel if I supply a pre-built one. My googling has indicated that

>> reset(gpuDevice(1))

will reset things (I have a single GPU, nVidia 650m) but when I do that Matlab segfaults and does not let me continue.

Is there any way to recover from a CUDA crash? Surely it's not necessary to restart Matlab every time a CUDA kernel crashes?

Edit: This is on Windows 8. On a Linux system with a Tesla K20X the same code runs with no problem. The kernel is compiled with

>> !nvcc my_kernel.cu -ptx

in either case.

[Nick Chng]

It's really frustrating, I agree. Hope they fix this in an upcoming release. I know it's my fault but I don't think Matlab is meant to be this punishing!

[Vincent Roberge]

I have the exact same problem using Matlab 2017a. If I call a CUDA kernel with an error, the GPU device becomes busy or unavailable and I need to restart Matlab to use it again. This is really annoying.

[covariant_cat]

Same issue here (2017a). This issue makes debugging CUDA in matlab really annoying. Really hope this to be fixed. I'm surprised that this doesn't get fixed after so many years. Without good support of CUDA, more and more people will switch to python for scientific computing (and deep learning). Mathworks should pay more attention to this.

Answer 1

[Joss Knight]

If you crash your card, you often need to reload the driver, in much the same way that if a program segfaults, it has to be restarted. Any number of things could go wrong but for the sake of argument let's say the GPU memory containing your kernel instructions has been trashed and is unrecoverable except by being reloaded.

The way to reload the driver is to restart MATLAB. There isn't really any other way to do it because that's the way OSs work.

So really you just need to stop thinking of your CUDA kernel as isolated from the MATLAB executable. When you write your own binaries, whether they be MEX functions or CUDAKernels, you put the entire MATLAB application at risk of your bugs.

[covariant_cat]

Thanks for the answer. Why couldn't we run the CUDA kernel as a separate process (or has an option to do so)? This would be particularly useful in the debug phase.

[Walter Roberson]

The overhead for setting up a separate process and communicating with it is somewhat high. When using a shared gpu instead of a tesla class gpu set to dedicated mode, the overhead would typically be an unacceptable burden.

[Joss Knight]

If you want to do that, start a parallel pool and use parfeval to run your code in your parallel process. If it crashes your card, your client process will be fine.