Parfor is consuming all my RAM and does not liberate it

39 visualizaciones (últimos 30 días)
Hi everyone,
I am running a parfor with small matrices (around 6MB) and I am having problems of memory. At the beginning, everything works perfectly, but then each processor starts accumulating RAM with each iteration at it does not clean it. I have monitored the memory using for and parfor:
Is it normal that the parfor is not "cleaning" the data which it loads after using it? With the for the processor does not accumulate the RAM... The code goes like that. Some functions are commercial so I do not have access to them, but in the for they don't do anything strange. Any help would be appreciated!!
%%PREPARE DATA
parpool(3)
CVerr=nan(size(PARAMETERS,1),LV);
SVM_CVerr=nan(size(PARAMETERS,1),1);
parfor_progress(500);
%%START LOOP
parfor i=1:500
%%SELECT PREPROCESSING AND REGION
Reg=[];
DATAi=DATA;
for j=1:N
Reg=[Reg;PARAMETERS{i,j}];
end
DATAi.include{2,1}=gTranslator(Reg,DATAi.axisscale{2,1});
prep=PARAMETERS(i,N+1);
%%PLSDA
options_plsda.include{2,1}=gTranslator(Reg,DATA.axisscale{2,1});
options_plsda.preprocessing={prep{1} MC};
plsda_model=plsda(DATAi,LV,options_plsda);
%%CV
options_cvplsda.preprocessing={prep{1} MC};
plsdamodelcv = crossval(DATAi,[],plsda_model,{'vet' 20},LV,options_cvplsda);
CVerr(i,:)=plsdamodelcv.detail.classerrcv(1,:);
%%SHOW PERCENTAGE AND MONITOR MEMORY
percent = parfor_progress
display( sprintf('Progress %.2f %%',percent))
[A,B]=memory;
ramused(i)=A.MemUsedMATLAB;
ramav(i)=B.PhysicalMemory.Available;
t(i)=now;
end

Respuesta aceptada

Walter Roberson
Walter Roberson el 2 de Mayo de 2017
"Is it normal that the parfor is not "cleaning" the data which it loads after using it?"
Yes. The workers are only reset when the pool is initialized. The temporary variables that are named specifically in the body of the parfor loop get cleared when appropriate. Variables created in functions called by the body of the parfor loop get deallocated as usual when the called function returns. But if there is any kind of memory leak in the work being done, then that memory is lost until the worker terminates.
This is the only model that is consistent with the ability to use parfeval() and parfevalOnAll()> to initialize data structures once per worker without having to worry about initializing the data once per loop.
  4 comentarios
Jonas
Jonas el 12 de Jun. de 2017
Editada: Jonas el 12 de Jun. de 2017
Walter, thank you!
My question is similar to that.
But I found that memory consumption differs for Windows and Linux OS, can you comment why it can be so?
Based on your comment, I can assume that the ismember uses any mex or cpp function and allocated memory can't be cleaned.
Walter Roberson
Walter Roberson el 12 de Jun. de 2017
"Based on your comment, I can assume that the ismember uses any mex or cpp function and allocated memory can't be cleaned."
No, ismember calls upon an internal builtin, so it has a direct implementation however MATLAB implements its internal libraries (which is not necessarily the same as the mex interface.) I would very much doubt it uses malloc.
I will reply in the other thread.

Iniciar sesión para comentar.

Más respuestas (0)

Categorías

Más información sobre Parallel for-Loops (parfor) en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by