Cluster validation used to pass, now fails after upgrade to 2011b

We have an SGE cluster on which we were running MATLAB. Worked fine till R2011a but fails after we have upgraded to R2011b. Validation just hangs and does not even time out from my Desktop (MAC OSX). Used to work before the upgrade. Cannot submit jobs from the desktop either.

No other change made on the cluster. (eg filesystem, paths, etc..)

[Thomas]

Make sure the version of the parallel computing toolbox on the desktop and the cluster are in sync. If matlab has not been upgraded on the cluster validation will fail.

Ask your cluster admin, if MATLAB R2011b has been installed and the DCS license updated on the cluster.

[Jisha]

Both are running R2011b, PCT ver 5.2 and the DCS has also been upgraded.

Answer 1

[Thomas]

Ok, Does the local validation pass? Run the validation from the command window so that it gives out more details:

setSchedulerMessageHandler(@disp) % additional debug information

jm =findResource('scheduler','configuration','NAME_OF_CONFIGURATION')

If the command window throws out a number of old jobs here, that means there was some issue when you moved up and it is trying to find metadata files of old jobs on the cluster and in your working directory and hence cannot complete the validation (this causes it to not time out either since it is still working). IF so do the following procedure

1. clear the local_scheduler _data (you can also rename the folder) /Users/USERNAME/.matlab/local_scheduler_data/R2011a

2. Empty all the metadata files and job directories in the DataLocation (on your desktop) - Parallel>Manage Configurations>Slect your Configuration and find the Data Location > Folder where Job directory is stored.

3. Remove the files that Matlab writes on the cluster for each job i.e Job#.lockstate, Job#.in.mat, Job#.out.mat, Job#.common.mat, Job#.jobout.mat, and Job#.state.mat

(In some cases these files can be found where you put the ParallelSubmitFcn, and SubmitFcn on the cluster)

Rerun Validation.

[Jisha]

Thanks, that worked..

My findResource was spitting out a lot of old jobs and trying to delete those files. I have deleted everything (local sched, DataLocation and our scratch space metadata files) and it works now..

thanks again..

[Thomas]

You are welcome, I'm glad this worked.. we had a similar issue not too long ago, but that was after a license upgrade.