Parallel for loop problem

1 visualización (últimos 30 días)
Simone Fiumi
Simone Fiumi el 12 de Ag. de 2022
Comentada: Simone Fiumi el 12 de Ag. de 2022
I'm running the following code in parallel. The variables of interes are w_b and alpha_b. The problem is that if I pause at a random time and then quit debugging the variables computed up to now are not saved. If i do that without the parallel cicle but with a simple for loop the variables are saved. For example, let's say I stop after 1 hour and I have 5 variables saved using a standard for loop, while with a parfor loop there are no saved variables (they are still equal to 0 as preallocated).
Nsim=50;
alpha_b=zeros(Nsim,1);
w_b=zeros(Nsim,1);
parfor k=1:Nsim
seq=zeros(size(M,1),size(M,2));
for j=1:size(M,1)
idx=find(M(j,:)==0,1)-1;
seq(j,:)=myhmmgenerate(idx,size(M,2),P_est,E,M(j,1));
end
sequence_s=seq_to_sequence(seq);
jumps_s=make_jumps(sequence_s);
[Y_btstrp,N_btstrp]=Y_N(jumps_s);
P_start_b=Pstart_nd(seq,2,2,Y_btstrp,N_btstrp);
[LL, ~, P, obsmat, nrIterations]=dhmm_em(seq, prior, P_start_b, E,N_btstrp,Y_btstrp);
[w_b(k),alpha_b(k)]=parameters(P,ne,nn,N_btstrp,Y_btstrp,1); % VARIABLES OF INTEREST
end
  2 comentarios
Jeffrey Clark
Jeffrey Clark el 12 de Ag. de 2022
@Simone Fiumi, there is no guarenteed order of processing of the parfor indexs. Are you sure you didn't get 5 or more values somewhere (but the same somewhere) in each of w_b and alpha_b vectors? If Nsim is large the run might have filled in end-(0:4:20) and you might not have noticed.
Simone Fiumi
Simone Fiumi el 12 de Ag. de 2022
I don't know if that happened or not. Right now it is running and since I don't want to risk I'll wait for it to end completely so that I won't encounter this problem. Thanks.

Iniciar sesión para comentar.

Respuestas (2)

Raymond Norris
Raymond Norris el 12 de Ag. de 2022
@Simone Fiumi think of the code running in the parfor as running on some entirely different machine. There is no "debugging" per se. So pausing/stopping the parfor terminates the work entiring including the ability to checkpoint variables.
What you could consider doing is creating a data queue that sends data back to the MATLAB client. The client (outside of the parfor) could view the data queue. For instance:
function sf
D = parallel.pool.DataQueue;
afterEach(D,@(data)logdata(data))
parfor idx = 1:10
...
[wb, ab] = parameters(P,ne,nn,N_btstrp,Y_btstrp,1);
% Send data back to client
send(D,[wb ab])
w_b(k) = wb;
alpha_b(k) = ab;
end
end
function logdata(D)
% Evaluate a snapshot of wb, ab
end
Not a perfect solution, but might help.
  1 comentario
Simone Fiumi
Simone Fiumi el 12 de Ag. de 2022
Yes it could be helpful but unfortunatly I don't how to use and write such functions. Thanks.

Iniciar sesión para comentar.


Benjamin Thompson
Benjamin Thompson el 12 de Ag. de 2022
You may want to review this thread and use parfeval if the partial results option is something you want.

Categorías

Más información sobre Parallel for-Loops (parfor) en Help Center y File Exchange.

Etiquetas

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by