GPU computations slowing down

Within a loop, my code carries out some iterative solving procedure on the GPU. With every loop iteration, the time taken by the solving procedure increases. There doesn't seem to be a memory leak: gpuDevice consistently reports approximately the following:

TotalMemory: 1.2885e+10
AvailableMemory: 5.6773e+09

However, when I pause the loop for a while by means of a debug stop, after relaunching again, computations are up to speed again, after which they start slowing down again...

I'm running matlab2016b on Windows 10, using a Nvidia Titan X (Pascal) GPU. Any suggestions for this problem?

[Mohammad Abouali]

.

[Jan]

Without seeing the code or knowing any details, it is hard to guess, what's going on. Did you control the temperature of the graphics board?

[Peter]

thanks Jan, I already started to be suspicious about temperature and monitored it. After launching the matlab script, temperature starts rising, GPU performance starts being capped by Temperature. (See performance monitor screen shot below) After reaching 95(!) degrees, the crash occurred again. So we do have strong indications for a cooling issue... What does this mean?

• a) I got myself a wrong system set-up, need better cooling ?
• b) this GPU card doesn't suit my needs, perhaps go for one of the Keplers (more expensive for similar memory)?

Answer 1

[Peter]

Monitoring the GPU performance revealed that most probably the temperature is causing the issue: Slowing down of performance goes with rising of temperature and performance is capped by temperature.

Crash of the GPU occurred when GPU reached 95 degrees...

[Joss Knight]

Most people who report issues like this are simply seeing an issue of asynchronous execution. They see fast times until the kernel execution queue is full, and then they start seeing the actual complete computation times.

To avoid this try using wait(gpuDevice) to add synchronisation points to your code.