I need help with parpool and optimizing recources for image reconstruction, running matlab with batch script

Im running an image reconstruction (1798 head volumes) script on a computation cluster

These are my recources

Nodes: 1

Mem per core: 43 GB

Cores per node: 24

Mem tot: 1024 GB

The reconstruction script reconstructs each full head volume (total 1798 head volumes) in the for loop for k=1:length(tframes), tframes=1798:

Now, my idea is to use parpool to parallelize the for loop. When using parpool one needs to specify the number of workers. This number should match the number of CPUs requested.

IN the computation cluster I use recources are requested as follows:

#SBATCH -p hugemem % name of the partition

#SBATCH -t 072:00:00 &time

#SBATCH --nodes=1 %nodes

#SBATCH --mem-per-cpu= 43G

#SBATCH --ntasks= 24

#SBATCH --cpus-per-task= 1

My question, how should I use parpool to parallelize this task of reconstructing the 1798 head volumes? Or do you think in this case parfor would be better?

Answer 1

[Raymond Norris]

Hi,

I haven't taken the time to see if parfor will work in your case, but have some general comments/thoughts.

parfor is a MATLAB parallel construct for parallelizing a for loop. In order for a parfor to run correctly, it need a pool of workers (i.e. parpool). By default, if a parallel pool is not running, calling parfor will instantiate the pool with a default number of workers (12 in your case).

I would suggest the following. Change your Slurm jobscript as such

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24

You're only running 1 task (MATLAB), but you'd like to dedicate 24 cores to it(your pool). In your MATLAB code, add the following

% Query for available cores
sz = str2num(getenv('SLURM_CPUS_PER_TASK'));
if isempty(sz), sz = maxNumCompThreads; end
p = parpool('local',sz);
parfor k = 1:length(tframes)
...
end

You might decriment sz by 1 to account for the MATLAB client running.

Raymond