Parallel for loop problem

2 views (last 30 days)
Simone Fiumi
Simone Fiumi on 12 Aug 2022
Commented: Simone Fiumi on 12 Aug 2022
I'm running the following code in parallel. The variables of interes are w_b and alpha_b. The problem is that if I pause at a random time and then quit debugging the variables computed up to now are not saved. If i do that without the parallel cicle but with a simple for loop the variables are saved. For example, let's say I stop after 1 hour and I have 5 variables saved using a standard for loop, while with a parfor loop there are no saved variables (they are still equal to 0 as preallocated).
Nsim=50;
alpha_b=zeros(Nsim,1);
w_b=zeros(Nsim,1);
parfor k=1:Nsim
seq=zeros(size(M,1),size(M,2));
for j=1:size(M,1)
idx=find(M(j,:)==0,1)-1;
seq(j,:)=myhmmgenerate(idx,size(M,2),P_est,E,M(j,1));
end
sequence_s=seq_to_sequence(seq);
jumps_s=make_jumps(sequence_s);
[Y_btstrp,N_btstrp]=Y_N(jumps_s);
P_start_b=Pstart_nd(seq,2,2,Y_btstrp,N_btstrp);
[LL, ~, P, obsmat, nrIterations]=dhmm_em(seq, prior, P_start_b, E,N_btstrp,Y_btstrp);
[w_b(k),alpha_b(k)]=parameters(P,ne,nn,N_btstrp,Y_btstrp,1); % VARIABLES OF INTEREST
end
  2 Comments
Simone Fiumi
Simone Fiumi on 12 Aug 2022
I don't know if that happened or not. Right now it is running and since I don't want to risk I'll wait for it to end completely so that I won't encounter this problem. Thanks.

Sign in to comment.

Answers (2)

Raymond Norris
Raymond Norris on 12 Aug 2022
@Simone Fiumi think of the code running in the parfor as running on some entirely different machine. There is no "debugging" per se. So pausing/stopping the parfor terminates the work entiring including the ability to checkpoint variables.
What you could consider doing is creating a data queue that sends data back to the MATLAB client. The client (outside of the parfor) could view the data queue. For instance:
function sf
D = parallel.pool.DataQueue;
afterEach(D,@(data)logdata(data))
parfor idx = 1:10
...
[wb, ab] = parameters(P,ne,nn,N_btstrp,Y_btstrp,1);
% Send data back to client
send(D,[wb ab])
w_b(k) = wb;
alpha_b(k) = ab;
end
end
function logdata(D)
% Evaluate a snapshot of wb, ab
end
Not a perfect solution, but might help.
  1 Comment
Simone Fiumi
Simone Fiumi on 12 Aug 2022
Yes it could be helpful but unfortunatly I don't how to use and write such functions. Thanks.

Sign in to comment.


Benjamin Thompson
Benjamin Thompson on 12 Aug 2022
You may want to review this thread and use parfeval if the partial results option is something you want.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by