Writing a matfile in a parloop

2 visualizaciones (últimos 30 días)
Angelo Cuzzola
Angelo Cuzzola el 4 de Mayo de 2020
Comentada: Angelo Cuzzola el 5 de Mayo de 2020
I have a very simple code that works pretty well when the size of the involved objects is manageable by the memory.
%%% Variables
% Z -> matrix of dimensions (r,T)
% V -> matrix of dimensions (r,r,T)
% nanY -> matrix of dimensions (n,T)
% y -> matrix of dimensions (n,T)
nanY = isnan(y);
y(nanY) = 0;
denom = sparse(zeros(n*r,n*r));
nom = sparse(zeros(n,r));
parfor t=1:T
nanYt = sparse(diag(~nanY(:,t)))p;
denom = denom + kron(Z(:,t+1)*Z(:,t+1)'+Vsmooth(:,:,t+1),nanYt);
nom = nom + y(:,t)*Z(:,t+1)';
end
vec = denom\nom(:);
In practical application, that bunch of code has to run with the parameter n taking values around 80k leading quickly to an unfeasible memory load. The solution I figured out relies on matfiles, at the expenses of paralellizionation and general performance. This is the version with which I am able to handle it for large n.
nanY = isnan(y);
y(nanY) = 0;
denom = zeros(n*r,n*r,T);
nom = zeros(n,r,T);
save var_cyc.mat denom nom y nanY -v7.3;
v = matfile('var_cyc.mat', 'Writable', true);
for t=1:T
v.nanYt = sparse(diag(~v.nanY(:,t)))p;
v.denom(:,:,T) = kron(Z(:,t+1)*Z(:,t+1)' + V(:,:,t+1),v.nanYt);
v.nom(:,:,T) = v.y(:,t)*Z(:,t+1)';
end
vec = sum(v.denom,3)\sum(v.nom(:),3);
Unfortunately this script cannot be paralellizable because conflicts generate when workers try to write on the matfile. I'm wondering if there is a way to recover the initial parallelization structure using matlab file to handle huge file without incurring in 'out of memory errors'.
Thanks.

Respuestas (1)

Jason Ross
Jason Ross el 4 de Mayo de 2020
Have you looked into using tall arrays?
  1 comentario
Angelo Cuzzola
Angelo Cuzzola el 5 de Mayo de 2020
Yes, but nom and denom are not exactly tall. They are 60kx60k matrices

Iniciar sesión para comentar.

Categorías

Más información sobre Call Python from MATLAB en Help Center y File Exchange.

Etiquetas

Productos


Versión

R2020a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by