MATLAB Answers

Parfor loop: how to keep the temporary variables ?

36 views (last 30 days)
V.D-C
V.D-C on 28 Apr 2020
Commented: V.D-C on 29 Apr 2020
Hello,
I am working with a time-consuming code and I want to put in parallel computing the independant functions I use.
I have the following script (only a fraction of it):
parfor i = 1:2
if i == 1
[T20cm_land, albedo_land, smb_land, swe_land, T1m_land, T0_land, abl_land,...
acc_land, runoff_land] = display_output_land(point_output_path_land, yr, mm, dd, date);
else
[T20cm_glacier, albedo_glacier, smb_glacier, swe_glacier, T1m_glacier, T0_glacier,...
abl_glacier, acc_glacier, runoff_glacier] = display_output_glacier(point_output_path_glacier, yr, mm, dd, date);
end
end
These lines are themselves part of a function. In it, I use the resulting variables (eg: albedo_land, albedo_glacier etc..) to make some calculations and plots. Originally I just ran the functions by writing :
[T20cm_land, albedo_land, smb_land, swe_land, T1m_land, T0_land, abl_land,...
acc_land, runoff_land] = display_output_land(point_output_path_land, yr, mm, dd, date);
[T20cm_glacier, albedo_glacier, smb_glacier, swe_glacier, T1m_glacier, T0_glacier,...
abl_glacier, acc_glacier, runoff_glacier] = display_output_glacier(point_output_path_glacier, yr, mm, dd, date);
But these two functions are independant and take around 30 minutes to run. Is there a way to save all the output variables from these 2 functions outside of the parfor loop, in order to use them later in the script ?
Have a nice day !

  0 Comments

Sign in to comment.

Accepted Answer

Edric Ellis
Edric Ellis on 29 Apr 2020
Output variables from parfor must be either sliced or reduction variables. So, you could adapt your code like so:
parfor i = 1:2
if i == 1
[T20cm{i}, albedo{i}, . . .] = display_output_land(. . .);
else
[T20cm{i}, albedo{i}, . . .] = display_output_glacier(. . .);
end
end
This converts all the outputs into sliced form.
An alternative given the fact that you really essentially have two completely separate function calls is to use parfeval instead, like this:
% Invoke both function evaluations in parallel.
fut_land = parfeval(@display_output_land, 9, point_output_path_land, yr, mm, dd, date);
fut_glacier = parfeval(@display_output_glacier, 9, point_output_path_glacier, yr, mm, dd, date);
% Collect results - this blocks until the evaluation is complete.
[T20cm_land, albedo_land, . . .] = fetchOutputs(fut_land);
[T20cm_glacier, albedo_glacier, . . .] = fetchOutputs(fut_glacier);

  3 Comments

V.D-C
V.D-C on 29 Apr 2020
Thank you very much for your answer ! I actually found another way to deal with it. I saved my variables inside the functions themselves into temporary folders. Then, later in the script I load these variables and once everything is calculated, I delete the temporary file. It worked so far but I guess might not be as efficient as your solution.
Edric Ellis
Edric Ellis on 29 Apr 2020
Glad you got things working!
Saving files within parfor is generally not a great idea - and not just from an efficiency standpoint - it might work in a simple case like this providing you can ensure the workers are always writing to different files, but it's easy to trip up if you end up trying to write to the same files from different workers. Also, if you scaled up to a remote cluster, then there's a chance that the client MATLAB might not be able to see the same filesystem as the workers.
V.D-C
V.D-C on 29 Apr 2020
Then I should change my code for your solution, I'm working on a cluster (even though the files are written in a different folder). Thank you for your advices !

Sign in to comment.

More Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by