How do I clear GPU memory on a multi-GPU system?

Currently, I use reset(gpuDevice(#)) to clear my GPUs, which I am using in spmd-end. reset(gpuDevice(1)) works fine. However, reset(gpuDevice(2)) and any #>1 just freezes and cannot be stopped once run.

So can I assume that reset cannot work on gpuDevices that are above the main one? How do I clear memory in the other GPUs?

[Jill Reese]

I haven't been able to reproduce this issue; I am able to reset both of my GPUs using version R2012a (both inside and outside the spmd block).

What version of MATLAB are you using? Are you executing reset(gpuDevice(#)) inside or outside the spmd block? Is your matlabpool size the same as the number of GPUs you are using? Could you provide a bit of MATLAB code showing exactly what you have tried?

[Stephen Won]

I did a bit more digging. It turns out that it's just my 2nd GPU that hangs; it wasn't transferring/accepting data with the main memory. As a result, even though I was calling reset(gpuDevice(2)), it wasn't able to to send the command to it or something like that. So, although reset(gpuDevice(1)) and reset(gpuDevice(3)) may have worked, I couldn't tell the difference. I only knew reset(gpuDevice(1)) worked since I wanted to make sure that I could still use GPUmat as an alternative.

I'm assuming this means it's a hardware issue. Regardless, I thought it was strange that no error message or indicator popped up. I had to use nvidia-smi to notice that the memory in the 2nd GPU was staying low and constant while all other GPUs were increasing when data was passed to them. Still, I'd like to know what the hardware error is and what to do to detect it.

Some relevant info: I am using MATLAB R2012a. I am executing reset inside the spmd block. My matlabpool size is equal to the number of GPUs. All 3 GPUs can be detected using nvidia-smi on Linux or GPU-Z on windows. CUDA 4.0 is being used. The GPUs are 1 GTX 580 and 2 GTX 470s.