parfor inside for consuming different run times

Hello,

As it has been already advised, parfor within for isn't really recommended, but I am somewhat bounded by the intention of my code. I am running an optimization algorithm, and within each iteration of the algorithm I need to update around ~1k variables.

So, my code (please bear with me for the rough pseudo-code) -

for i=1:MaxIters
  % I calculate some prerequisites here
  parfor (i=1:1000, 10)
    % I perform some independent calculations here, as the 1k 
    % variables are independent of each other.
    % However, I need to load and save them (which I do using 
    % a function call)
 end
 % I update my optimization variable here, and then go back to 
 % the for loop for another round of optimization
end

If I replace the parfor loop with a for loop, I see that one inner iteration takes roughly around ~30mins. I managed to get it under control using a parfor loop, and I happily see a reduction in simulation time to ~6-7mins. However, I am observing that this simulation time keeps on increasing as it goes for another round of optimization (next outer iteration), and it keeps on happening.

On some further checks, I checked that for the first outer loop, my cores/workers are nearly running at ~100%, but for the subsequent ones, they run at around ~20-60%, which I think is contributing to the increment. However, I am not able to figure out what is the problem, as the inner process is nearly the same for every outer iteration.

I have even tried parfevalOnAll('clear', 0, 'all') to clear the workers' data and delete(gcp('nocreate')) methods to even shut down the parpool after the inner loop completes, but nothing seems to be stopping the increment. Can somebody please help me out??

Thanks so much!

[Nicolas Schmit]

Could you share your code and also tell which version of MATLAB you are using?

[AA]

Hello,

Thanks for you response. I am running my code on MatlabR2016a. I have attached my code for your kind reference(sorry for not posting it here, as it is ~700 lines long). If there is anything else I can provide, I would be more than happy to.

Thanks again!