why does my job on cluster stop to produce output

Hey, I am using parallel toolbox on a linux cluster (istan nodes and SLURM scheduler). The main routine (the parfor loop section) looks as follows. An 2-d array (MASKE) is used to extract time series which have values, and the function core_eQM is applied on these time series: ...

cd $WORKDIR

pc = parcluster('local')
pc.JobStorageLocation = strcat('$WORKDIR/',getenv('SLURM_JOB_ID'))

% start the matlabpool with maximum available workers
% control how many workers by setting ntasks in your sbatch script

 matlabpool(pc, getenv('SLURM_CPUS_ON_NODE'))

...

pardim=size(MASKE,2);
XX1=NaN(7305,pardim);

parfor ii=1:pardim
     if ~isnan(MASKE(1,ii))
         fprintf('%i \t \n', ii);    %shows me progress of job (creates files, which are empty)
            if meth == 1;
              xx1 = core_eQM(squeeze(VARIABLE_BSE(ii,jj,1:3653)),squeeze(WRF_VARIABLE(ii,jj,1:3653)),squeeze(WRF_VARIABLE(ii,jj,3654:7305)))
              elseif meth == 2
            ...
            end
   end
end

Now, I use the following script to submit it on the cluster:

#!/bin/bash
#SBATCH --job-name=imat_par_test
#SBATCH --output=matlab_parfor.out
#SBATCH --error=matlab_parfor.err
#SBATCH --partition=ivy
#SBATCH --time=72:00:00
#SBATCH --nodes=1
#SBATCH --ntasks=20

source /etc/profile.d/00-modules.sh

module load app/matlab2014b

cd $WORKDIR

# Create a local work directory
mkdir -p $WORKDIR/$SLURM_JOB_ID

#cd $WORKSDIR/$SLURM_JOB_ID

# Kick off matlab
matlab -nodesktop < script_apply_BC.m &

#wait
# Cleanup local work directory
rm -rf $WORKSDIR/$SLURM_JOB_ID

At the beginning (first few hours) the job runs fine. The size of pardim is 420. After pardim reaching approx. 250, the procedure slows down and finally does not "continue", i.e. the job is still running without producing output files. Thus, no problems are reported in the matlab_parfor.err file. I do not know exactly how I can analyse the problems in this case.

Any ideas?

[Igor Yanovich]

Hi Patrick,

Unfortunately, I don't have an answer, but just want to add that I have a similar problem. I have a long loop that runs well for the first 150-600 iterations, but then stops producing output, but the process continues to run on the (Linux) cluster.

In my case, there is a difference in which exact iteration the process stops to produce output, depending on the starting conditions (which determine the size of the permanent and temporary matrices involved) - e.g., in one condition, the process usually goes incommunicando around iteration 278-280.

So I have a suspicion that this might have something to do with memory. But it's utterly weird, as I don't allocate new stuff within the loop - thus it's not really clear where the memory problems are coming from, if it's memory problems.

I do hope somebody else here knows what's going on and can let us two know!

[Igor Yanovich]

P.S. To add some info - in my case, the global loop itself is not a parfor, but parfor is actively used within each iteration.

Another thing which I suspected is that I'm using VPA - but this doesn't seem to be your case, so probably that's irrelevant.

[Simone Stünzi]

Hi you all, did you ever find a solution to this or pinpoint the exat issue?

I'm currently experiencing the exact same thing and wondering what I could do differently to avoid this.

Any help is greatly appreciated.

Best, Simone

[Patrick Laux]

unfortunately not, Simone. I just gave up.

If you find out more, I would be happy if you let me know.

Patrick

[Simone Stünzi]

I've increased idleTimeout to Inf and will let you know if that solves my issue.

Best, Simone