Parallel Computing on cluster
1 visualización (últimos 30 días)
Mostrar comentarios más antiguos
I can use Matlab on Pronghorn(a cluster on university of nevada Reno). For my work I need to utilize the parallel computing option on this cluster.
This is my test code that starts parallel computing for matlab(I name this code, PCTest.m, for calling it on sbatch file).
clear all
tic
parfor i = 1:5
fprintf('Progress %d of %d...\n',i,5);
pause(5);
end
toc
The above code should take around 5 seconds to run, if parallel computing is working.
This is my job submission code for cluster(xyz.sh),
#!/bin/bash
#SBATCH --job-name disable_multithreading
#SBATCH --time=01:00:00
#SBATCH --nodes=1 --ntasks-per-node=6
#SBATCH --cpus-per-task=1
module load matlab
matlab -nosplash -nodesktop -r "PCTest; exit"
I type sbatch xyz.sh to run the matlab file. But this does not start workers on cluster, like it does on my desktop or laptop. Any help will be hightly appreciated. Thanks
0 comentarios
Respuestas (1)
Raymond Norris
el 12 de Abr. de 2021
As it is written, the parallel pool will start when the parfor is called, which is then included in the timing. And since you're not explicitly starting it, it could be starting as many as 12 workers. Try the following:
clear all
% Time how long it takes to start the pool
tic
parpool(5);
toc
% Time how long it takes to run the parfor
tic
parfor i = 1:5
fprintf('Progress %d of %d...\n',i,5);
pause(5);
end
toc
To make this a bit more robust, query Slurm for the number of assigned tasks to start your pool (doing this off the top of my head, I think it's SLURM_NTASKS).
sz = getenv('SLURM_NTASKS');
if isempty(sz)
% For some reason, we're not running in a Slurm job
sz = maxNumCompThreads;
end
% Time how long it takes to start the pool
tic
parpool(sz-1);
toc
% Time how long it takes to run the parfor
tic
parfor i = 1:5
fprintf('Progress %d of %d...\n',i,5);
pause(5);
end
toc
Lastly, since R2019a, you can shorten
matlab -nosplash -nodesktop -r "PCTest; exit"
to
matlab -batch PCTest
4 comentarios
Raymond Norris
el 13 de Abr. de 2021
I've seen this before. Are you submitting several Slurm scripts at once? If so, contact Technical Support (support@mathworks.com) and they'll show you the best way to create a temporary job storage location (i.e. local_cluster_jobs) for each job.
Ver también
Categorías
Más información sobre Cluster Configuration en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!