MEX memory leak - is this a bug?

I tried reporting this as a bug, with no response so far. Before I try a little harder, I want to make sure this is indeed a bug. I asked a few people, here & other places, and no one can explain this:

mxGPUArray * tmp=mxGPUCopyFromMxArray(prhs[0]);
plhs[0]=mxGPUCreateMxArrayOnGPU(tmp);
mxGPUDestroyGPUArray(out);

This code causes memory leaks (at least in Matlab 2013a, compiled with Visual Studio 2010). I run the following on loop in matlab:

foo=gpuArray.randn(1000);
res=MyMex(foo);
gpuDevice

And I see that with every iteration, there is less free memory in the GPU device. Correct me if i'm wrong, but this SHOULD NOT be happening. Matlab should free the values previously stored by "res", but It doesn't seem to.

NOTE: This is not bug 954239.

I've been working on this problem for two weeks, with no luck. There's always some memory leak. Does anyone have any idea what's going on? Did I miss something?

Answer 1

[Edric Ellis]

It appears as though there's a problem with mxGPUCopyFromMxArray. Please could you try the following workaround:

/**
 * Replacement for mxGPUCopyFromMxArray
 */
mxGPUArray * mxGPUCopyFromMxArray2(mxArray const * mx) {
    if (mxIsGPUArray(mx)) {
        mxGPUArray const * tmp = mxGPUCreateFromMxArray(mx);
        mxGPUArray * returnValue = mxGPUCopyGPUArray(tmp);
        mxGPUDestroyGPUArray(tmp);
        return returnValue;
    } else {
        return const_cast<mxGPUArray *>(mxGPUCreateFromMxArray(mx));
    }
}

[Ayal]

I will not be able to test this until Sunday. I'm glad to see I didn't do anything wrong - because If I had, It would mean that 2 weeks weren't enough to figure out the problem.

Would you like me to inform you of the results when I test this? Should I go ahead and report this as a bug, or have you taken care of it? (I assume that if this is indeed a bug, you told someone who might make sure it is fixed, since you work at Mathworks :) )

Thank you very much!

[Edric Ellis]

You don't need to report this again, thanks; but it would be great to hear if this does indeed fix the problem.

[Ayal]

This mostly works! I still see some odd behavior as far as the memory goes, but I guess it's just something I don't understand related to the memory managment.

using "clear dumdum" actually releases memory, and after the 2nd time I run the code, memory doesn't disappear anymore. However, using "clear foo2" doesn't release any memory. But all the memory returns after clearing all gpuArrays, but it seems like the major problems are resolved.

I also tried

foo=foo2;
testMyMex;
clear foo;

This won't release memory either. I guess there is more to be fixed. But your solution should be a big help to me. Thank you!

[Edric Ellis]

It's possible that the remaining memory usage behaviour might simply be artefacts of the allocation pooling.

Answer 2

[Edric Ellis]

I think this is nothing to do with the MEX interface, and instead to do with the way MATLAB tries to recycle memory allocations on the device, as demonstrated by the following example:

% First, check the free memory on the device
>> dev = gpuDevice(); dev.FreeMemory

ans =

     5.5265e+09

% Next, make 100 allocations of 8MB
>> for idx = 1:100, c{idx} = gpuArray.ones(1000); end
>> dev.FreeMemory

ans =

     4.7138e+09

% Now, clear all but one of those and check the free memory
>> c(2:end) = [];
>> dev.FreeMemory

ans =

     4.7138e+09

% Clear all arrays, and check free memory again
>> clear c
>> dev.FreeMemory

ans =

     5.5265e+09

You can see here that freeing some arrays doesn't change the amount of free memory on the device, but when all arrays have been freed then all the memory is returned.

[Ayal]

first of all, WOW. 5.5 GB free memory? I'm jealous. not that I really need one, but It sure would help with the memory leaks.

On another note, It doesn't work quite the same for me (what version of MATLAB are you using?)

>> dev = gpuDevice(); dev.FreeMemory

ans =

     1.8480e+09

>> for idx = 1:100, c{idx} = gpuArray.ones(1000); end
>> dev.FreeMemory

ans =

     1.0354e+09

>> c(2:end) = []

c = 

      [1000x1000 gpuArray]

>> dev.FreeMemory

ans =

     1.2954e+09

>> clear c
>> dev.FreeMemory

ans =

     1.8480e+09

Also:

>> dev=gpuDevice;
>> dev.FreeMemory

ans =

     1.8477e+09

>> testMyMex

ans =

     1.6529e+09

>> testMyMex

ans =

     1.4582e+09

Not sure why I lose more memory the 2nd time too, but if I run the script again and again, it doesn't leak more memory. but let's continue:

>> foo4=foo;
>> testMyMex

ans =

     1.4581e+09

>> numel(foo4)

ans =

      14520000

no memory lost (well, a little bit. seems like a tenth of a megabyte - my guess is that it's actually less, you know - because of the exponent, we get approximate values, rounded). How come? was supposed to lose sizeof(double)*numel(foo4). Continuing:

>> clear foo4
>> dev.FreeMemory

ans =

     1.4581e+09

Where is my memory? I didn't get it back. >> clear foo4 >> dev.FreeMemory

ans =

     1.4581e+09

>> clear foo3
>> dev.FreeMemory

ans =

     1.4581e+09

>> clear foo
>> dev.FreeMemory

ans =

     1.4581e+09

>> clear foo2
>> dev.FreeMemory

ans =

     1.8476e+09

finally, I got my memory back after clearing all gpuArrays. But here's the thing. My code runs a lot of loops, and uses gpuArrays a lot. I use "clear" when i'm done with certain variables, but what use is it if it doesn't clear these certain variables? why do I need to clear ALL gpuArrays, even though most of them have different values from one another, and thus aren't used for lazy copies?

Correct me if i'm wrong (really, if I misunderstood something, please correct me), but that's not how it's supposed to be. (my script, tests the MEX on big chunks of data, so the memory loss is appropriate, but the fact that I don't get the memory back until I clear ALL gpuArrays isn't).

foo, foo2 and foo3 are variables used to test the MEX. I'm currently using a different code from the one posted here, but it's the same thing - create, destroy everything, still leaking memory.

EDIT: I've tried it with the code I posted here, and it's even worse. This is because the code I used for the example in this post doesn't return LHS, but changes the value of an input in-place (using const_cast<mxGPUArray *> to remove the const and overwrite values). The code I posted here is even worse. I clear ALL variables, and I still have missing memory. This seems to be a MEX issue. But additionally, from your post I've learned that indeed there is also a problem with the way MATLAB recycles memory. At least to me it seems like a problem - note that when I ran it, there were 550+ megabytes for a 1000*1000 array. meaning 550 million bytes for 1 million elements of type double, meaning 8,000,000 bytes. WAY too much. This makes no sense to me.

[Edric Ellis]

I was using R2013a with (as you observe) a pretty large GPU. The memory pooling is parameterized on the total amount of memory on the GPU, so you can expect some differences in detail.

The intention of the memory pooling is that you should never run out of memory because of the allocations that MATLAB no longer needs, but haven't actually been freed from the device.

If your code eventually fails because it runs out of memory, then there's a real problem - if so, please could you post a minimal complete self-contained example. Obviously, by using const_cast, you are living dangerously. The CUDAKernel has safe support for in-place operations (you can have in-out parameters to the FEVAL method) whereas the MEX interface does not.

[Ayal]

I don't want to use feval because I need streams, and (as I understand it), feval won't run CUDA code on the host, only on the device (only code begining with _global_). Using streams allows for a much faster code than using a Matlab loop with feval (I need the loop because some part of the computation has to be sequential)

Ignore the in-place editing. Here is the MEX for the code I posted in the question:

#include "mex.h"
#include "gpu/mxGPUArray.h"

void mexFunction(int nlhs, mxArray *plhs[],
                 int nrhs, mxArray const *prhs[])
{
    mxGPUArray * out=mxGPUCopyFromMxArray(prhs[0]);
    plhs[0] = mxGPUCreateMxArrayOnGPU(out);
mxGPUDestroyGPUArray(out);
}

and the script I use to run it:

foo2=gpuArray.randn([7 7 1000 1000]);
tic;
dumdum=MexMemoryTrack(foo2);
toc;

I run this a few times (for a more dramatic effect, increase the size of foo2). and the amount of memory decreases. when there isn't enough memory for the next time (according to gpuDevice), I run it again, and suddenly the amount of memory increases (a little). Then I run it one more time - and I get an error - not enough memory.

[Edric Ellis]

Thanks for that - I'll take a look and get back to you.

[Ayal]

And thank you for your help. (Just to be sure: do you get the same problem, or am I just unlucky and need to update every software related to this in hopes that the problem is some corrupted file?)

[Mateusz Pieniak]

I have the same issue. Did anybody resolved it? It was 3rd September 2013 and a few years passed...