MATLAB Answers

0

Use Parfor to split a 4D matrix into several 3D matrices

Asked by Philipp Heinirch on 4 Sep 2018
Latest activity Edited by Stephen Cobeldick on 4 Sep 2018
Hey,
so I to change my current code:
for i=1:d(4)
save_file_name=sprintf('layer_%i.mat',i); %name of the save file
variable_name=sprintf('layer_%i',i); %name of the variable
reduced_dimension=squeeze(reducedData(:,:,:,i)); %reduces the size of the matrix down to 3 since
eval(['layer_' num2str(i) '=reduced_dimension;']);%renames the variable
save(save_file_name,variable_name);%saves the files
eval(['clear ', 'layer_' num2str(i) ])
end
This code is supposed to split a giant 4D Matrix into several smaller 3D matrices while automatically renaming and saving them separatly. I am aware that using eval is considered to be bad, but so far I have not found any other solution.
I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop. Has anyone an idea how to solve this kind of problem ?

  2 Comments

What you're trying to do seems to be going in the wrong direction in terms of code efficiency... Seems like you already have the code to split a 4D to 3D, and you're already aware that using eval to rename variables is a bad practice. Note that using parfor won't help much because you are limited by the hard drive write speed. It'll be better if you tell us:
  • What is the real task you'd like to accomplish AFTER you split the 4D matrix?
  • Why do you need to split this 4D matrix?
"Use Parfor to split a 4D matrix into several 3D matrices"
You can already access the data in the 4D array very efficiently using indexing (and indexing is very efficient). In contrast, what you are trying to do with eval will only force you into writing slow, complex, buggy code. For example, by using eval you remove any ability of the MATLAB JIT engine to speed up your code, so you automatically force your code into being pointlessly slow, all of the time:
If you used simple indexing then MATLAB would speed up your code automatically.
"I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop"
If you are planning on writing fast and efficient code (which is usually what beginners intend when they start talking about parfor) then your approach is in entirely the wrong direction. If you really want to write efficient code:
  • do NOT make unnecessary duplicates of data in memory.
  • do NOT use eval or anything else to magically access variable names.
  • follow the guidelines in the MATLAB documentation:

Sign in to comment.

Products


Release

R2017b

2 Answers

Answer by Titus Edelhofer on 4 Sep 2018

Hi,
I agree with OCDER: parfor will only do huge memory movements from the Client to the workers but I doubt you will get much speedup.
If the "save" is indeed too slow, you might experiment with saving without compression: use "-v6" as flag when saving. The files are then probably much bigger but often saving is faster.
And by the way: if you have the file name with the layer number, why bothering what the variable name is? Why don't simply use "layer" for all of them?
Titus

  0 Comments

Sign in to comment.


Answer by Stephen Cobeldick on 4 Sep 2018
Edited by Stephen Cobeldick on 4 Sep 2018

Firstly it is important to note that processing a sequence of .mat files is much simpler when the variable names inside each file are exactly the same. This applies to both reading and writing them. Then you could trivially do this:
for slice = 1:d(4)
file = sprintf('layer_%d.mat',slice);
data = squeeze(reducedData(:,:,:,slice));
save(file,'data','slice')
end
Note how I wrote simpler, much more efficient code than what you have in your question, and that I saved the slice value too, so you know exactly which slice it is when you load that data:
S = load(...)
S.slice
S.data
Simpler to write, simpler to read, much more efficient: why waste your time fighting eval ?
If you really want to have lots of numbered variables (which are a sign of bad data design, and bad code), then you could split the data into the fields of a structure, and save that structure using the '-struct' option. But of course that would be slow, complex, and not recommended.

  0 Comments

Sign in to comment.