Parfeval memory leak? Clear memory on parallel workers?

13 visualizaciones (últimos 30 días)
Matthias
Matthias el 9 de Dic. de 2014
Comentada: Francis Plamondon el 18 de Nov. de 2020
Hi,
I have some sort of memory leak when using parfeval.
I have a function that starts a number of parfeval-computations and then uses fetchNext in a loop to retrieve all results.
The memory taken up by the Matlab instances in my parallel pool grows with each execution of the function that uses parfeval.
My question: Am I doing something wrong? Do I have to explicitly clear the memory of the workers after using parfeval? How do I do this? I tried "pctRunOnAll clear all" but it did not free up any memory on the workers (just the client instance).
Right now, the only way I can clear this memory is to shut down and re-start my parallel pool, but that's not a feasible solution for my application.
Thanks, Matthias
  3 comentarios
Atakan Atay
Atakan Atay el 25 de Mzo. de 2020
Editada: Atakan Atay el 25 de Mzo. de 2020
Hi Matthias,
Currently I experience the very same problem. How did you manage to solve it? I would really appreciate your help.
Thanks for your time.
Francis Plamondon
Francis Plamondon el 18 de Nov. de 2020
Hello,
I also recently struggled with parfeval. I would get 'out of memory' errors after several hours of running a huge batch of jobs.
Here is what I found out: each parallel.FevalFuture object holds a Diary property that contains everything that was displayed in the console window for each job. Since I had a function use disp() for monitoring purpose, it was being recorded in the Diary variable, eventually filling all the memory.
After commenting out the disp function, the memory stopped slowly increasing.

Iniciar sesión para comentar.

Respuesta aceptada

Edric Ellis
Edric Ellis el 15 de Dic. de 2014
Please could you try applying the workaround from this bug report. (The bug report mentions mapreduce, but it includes a fix that affects only parfeval).
  1 comentario
Seth
Seth el 13 de Feb. de 2015
I have a similar problem and tried using this patch and now get errors indicating "no method 'createInstance' with matching signatures found for class 'com.mathworks.toolboox.distcomp.objects.EntityFileProxy'
Do you have any other suggestions? I was running two workers with PARFEVAL and seeing 3gb increase in RAM used each call to them. (The PARFEVAL commands are in a loop)
Thanks.

Iniciar sesión para comentar.

Más respuestas (0)

Categorías

Más información sobre Asynchronous Parallel Programming en Help Center y File Exchange.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by