surrogateopt: Dot indexing is not supported for variables of this type.

I am trying to use the function surrogateopt in a script that I submit to a computer cluster. I don't want to plot the solver progress. To illustrate the problem that I have, I will use the toy code "Minimization_test.m":

minobj=@(xj)(xj+2).^2;
lbe= [-20];
ube= [20];
opts = optimoptions(@surrogateopt,'PlotFcn',[],'MaxFunctionEvaluations',600);
xj = surrogateopt(minobj,lbe,ube,[],opts) 

This script works correctly on my computer (matlab 2020a) and on the cluster (matlab 2020a) if create an interactive matlab session.

However, I get an error if I request that the job scheduling program SLURM execute the script. My SLURM script reads:

#!/bin/bash

#SBATCH -t 1:00:00

#SBATCH --mem=1g

#SBATCH -J matlab

#SBATCH -o slurm.out

module load Matlab/R2020a

matlab -nojvm -nodisplay -singleCompThread -r Minimization_test

If I submit this job, I get the following error in the output file:

Dot indexing is not supported for variables of this type.

Error in globaloptim.bmo.solver

Error in globaloptim.bmo.createSolver

Error in globaloptim.bmo.BlackboxModelOptimizer (line 92)

self =

globaloptim.bmo.createSolver(self,expensive,lb,ub,intcon,options);

Error in surrogateopt>createController (line 329)

controller =

globaloptim.bmo.BlackboxModelOptimizer(expensive,lb,ub,intcon,options);

Error in surrogateopt (line 292)

controller = createController(objconstr,lb,ub,intcon,options);

Error in Minimization_test (line 5)

xj = surrogateopt(minobj,lbe,ube,[],opts)

[Farhad Omidvar]

Hi

Did you manage to solve this issue?

I'm facing the same problem. Please let me know

[Walter Roberson]

What happens if you try to execute it interactively in a MATLAB session that was started with -nojvm ?

toolbox/globaloptim/+globaloptim/+bmo/createSolver.p is a .p file so we cannot examine it to see what the problem might be.

[Farhad Omidvar]

Hi Walter, thanks for the reply.

what do you mean by "interactively"? sorry if my question is basic.

the error I'm getting is a bit different from the OP.

Here is my matlab code:

ub = [0.015 0.2];
lb = -ub;
options = optimoptions('surrogateopt','PlotFcn',{@surrogateoptplot,@optimplotx},...
    'MinSampleDistance',1e-2,'UseParallel',true,...
    'MaxFunctionEvaluations',75,'Display','iter');
objective = @(x)SCA_optimize(x);
[x,fval,exitflag,output,trials] = surrogateopt(objective,lb,ub,options);

Here is my slurm script:

#!/bin/bash -l
#SBATCH --time=12:00:00       
#SBATCH --nodes=1      
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=32      
#SBATCH --mem=42000           
module load matlab/2020a
matlab -nodisplay -r "optmexe"

The optimization runs for 2 hours (about 60 iterations) and all of a sudden it stops with the following error:

F-count       Time        Best          Current         Trial
  61         7744.89     -3.4054e-27     -2.3363e-28     adaptive
  62         7767.71     -3.4054e-27     -3.1658e-28     adaptive
  63         7774.31     -3.4054e-27     -1.8851e-28     adaptive
  64         7789.41     -3.4054e-27     -2.2733e-28     adaptive
  65         8687.23     -3.4054e-27     -2.2102e-28     adaptive
{Error using globaloptim.bmo.BlackboxModelOptimizer/optimize (line 100)
Dot indexing is not supported for variables of this type.

Error in surrogateopt (line 139)
controller = controller.optimize();

Error in optmexe (line 11)
[x,fval,exitflag,output,trials] = surrogateopt(objective,lb,ub,options);
} 

However, when I run this on my PC with 4 cores it seems to be working ok. Could it be related to the high number of cluster cores that I'm using?

[Walter Roberson]

If you go to your desktop command line, and invoke MATLAB with the -nojvm option but without using -nodisplay or -r and at the command line you run optmexe then do you still encounter the same problem?

I recommand putting in some code that records how many iterations are successfully done. I also recommend putting in a try/catch block in optmexe to catch that error and do a smooth shutdown (save variables as appropriate, save debugging information)

[LBR]

Thank you for your suggestion. Indeed, it looks like it is -nojvm to create problems. If I execute the script interactively in a MATLAB session that was started with -nojvm but without -nodisplay or -r I get the same error as before.

1) If I run:

try

xj = surrogateopt(minobj,lbe,ube,[],opts)

catch ME

end

I get the following variable ME:

MException with properties:

identifier: 'MATLAB:structRefFromNonStruct'

message: 'Dot indexing is not supported for variables of this type.'

cause: {}

stack: [6x1 struct]

Correction: []

2) It looks like the error message is displayed before any iteration is completed: if I set the 'Display' option to 'iter', so that surrogateopt returns an iterative display, nothing gets printed on the display before the error message.

[LBR]

To conclude, I am now able to submit the job to the SLURM system using:

matlab -nodisplay -singleCompThread -r Minimization_test

Thank you, Walter.

However, I still don't understand what was going on, so any clarification is greatly appreciated!

[Walter Roberson]

I have not happened to look into how surrogateopt() works.

I am, though, put in mind that MATLAB implements some of its parallelism in the Parallel Computing Toolbox in terms of Java. You are not using that toolbox, but it seems to me that if surrogateopt() had some reason to create parallelism, that it would probably use java methods to do so.

[LBR]

I see, thank you.

[Farhad Omidvar]

Thanks, Walter, for the suggestions.

Before seeing them, I submitted the job for running on MATLAB/2020a (insteaad of the 2018b version) and the problem magically got fixed.

Answer 1

[Walter Roberson]

Removing the -nojvm solves the problem (for reasons currently unknown.)

[Daniel Rischawy]

Unfortunately I have the same problem. But I get the error if I run surrogates with parallel computing normally in Matlab2019a. The error appears random after around 200-300 iterations. Sample tasks for surrogates, however, did not cause any problems. Therefore I think it should have something to do with my objective function or parallel computing, but I do not understand the cause and why the error occurs so randomly. Could the reason be the Java version I use or the Matlab version? Thanks a lot in advance.

Regards