Out of memory during parfor

I'm trying to execute code like this

 for i = 1:100
   parfor 1:10
     funResult = function(data);
   end
 end

but I get some error like

Out of Memory during deserialization

or

Out of Memory during serialization

or some workers aborted during execution. I tried this code with different data (smaller of those that cause this error)

How i can solve this problem? I get the error after 8/10 iteration of first for loop.

[Matt J]

Use less memory?

[Andrea Stevanato]

Sorry?

[Matt J]

The error message seems to say you are using too much memory. So, use less. Or move to a computer with more memory.

[Andrea Stevanato]

I have 32GB ram, i need to fix error in some way if is possible.

[Matt J]

But how many workers, and how much memory is each one using? What did you do to investigate whether 32GB is enough?

[Andrea Stevanato]

It crash always, previous were 12 workers and after crash i tried with 10, 8 and 6 workers but it crash always.

How i could see how much memory each one is using?

Edit: I do nothing to see if 32GB is enough, but i have this and it must be enough.

[Matt J]

Are you on Windows? The Task Manager will show you how much RAM you are consuming, as the code runs, as well as how occupied all of your cores are.

[Andrea Stevanato]

No, i'm running under Linux and yes I could look how much RAM I'm using.

Answer 1

[Walter Roberson]

I do not know if it is still the case, but historically there was a limit of 2 gigabytes per array transferred, including on 64 bit versions of MATLAB.

Check your code in case you are using broadcast variables. Sometimes broadcast variables can be rewritten in a way that uses distributed variables instead. For example if two different columns need to be accessed, create two variables, one columns 1:end-1 and the other columns 2:end, and then index each by the parfor index: this will trigger distributed use instead of broadcast use in that situation.

[Andrea Stevanato]

I think that i already use distributed variable because the parameter of my function is a single element of array at position index, something like that:

myArray = repmat(MyClass, 1, 10)
for i = 1:1000
  parfor index = 1:10
    myResult = fun(myArray(index))
  end
end

I think that in this case myArray is a dristibuted variable.

I'm looking that in Preferences > Workspace There are MATLAB array size limit checked, I unckeded and now I'm tring again to run my code!

Edit: the function output is a Class that contain a "matrix" (tensor) with 99*99*2 elements.

[Weidong Lin]

Hi, have u fixed it? I am facing the same problem.

[Walter Roberson]

Is MyClass a class variable instead of being numeric? If it is, then I wonder if MATLAB is sending the entire class variable to each worker? With classes, it would not necessarily be correct to subsref on the client and send to the worker. repmat() isn't just a "make this many copies" operator, it is also an implicit cat() operator so the result of repmat is not necessarily 10 different objects: it could be one object with internal "size" 10.

I would have to think more about the circumstances under which it is safe to subref on the client and send to the worker; I can definitely think of some cases where that would get you the wrong answer, or at least behavior that isn't all that well defined (e.g., what if the object is keeping usage stats internally?)