bayesopt: how do I know when the hyperparameter optimization has converged?

This is a conceptual question. We are using Bayesian hyperparameter search to find optimal hyper parameters for our neural networks. Over time (i.e. with increasing iterations of bayesopt), the estimates of the optimal hyperparameter values change, and the objective improves (obviously). But how do we know when it has converged on a final set of values? What criterion can we use to determine whether more iterations of the optimization are needed?

Answer 1

[Alan Weiss]

Sorry to say, this is almost impossible to determine. It is a very difficult problem to determine stopping conditions for any optimization, and when your problem includes large data, neural networks and the like, it is even harder.

For some examples why I say this, see Isolated Global Minimum, Global Vs. Local Minimum Using ga, When the Solver Succeeds, and When the Solver Might Have Succeeded.

Alan Weiss

MATLAB mathematical toolbox documentation

[Alexander]

Sorry for resurrecting the thread, but my question would have exactly the same topic.

Is there a way to provide a custom stopping function? Like

result = bayesopt(..., 'StoppingFunction', @customCriteriaFunction); ?

If not, I'm forced to do something like

results = bayesopt(..., 'MaxObjectiveEvaluations', N);

while % customCriteria not fulfilled

results = resume(results);

end

with N set to a number suitable to wait for, after setting a debug breakpoint to stop optimization deliberately.

[Alan Weiss]

To stop an optimization early, use an output function. Set the OutputFcn name-value pair. For an example, see Bayesian Optimization Output Functions.

Alan Weiss

MATLAB mathematical toolbox documentation

[Chandrakanth Reddy Kancharla]

Hey Alan,

I saw the documentation that you have mentioned. But is it the same when using bayesopt with fit models like fitcknn, fitcsvm, etc.?

Context: I have few different datasets, I have written script that loops through these datasets and tries to fit a classifier model for each dataset using fitcknn that uses bayesopt as optimizer. Sometimes it takes more iterations and sometimes less to get to a MinObjective of '0'. What I am trying to do is stop the iterations once the MinObjective goes to '0'.

I tried using it as a part of the HyperparameterOptimizationOptions input argument. But that definitely seems to be wrong as it says there is no 'OutputFunc' as input. Here is the part of the function that uses the fitcknn model. Could you please point me in the right direction for the MinObjective based stop function implementation.

Thank you very much.

CrossValPart = cvpartition(FaultIndTrain, 'KFold',10);
    
opts = struct('Optimizer','bayesopt','CVPartition',CrossValPart, 'AcquisitionFunctionName','expected-improvement-plus',...
        'MaxObjectiveEvaluations',60, 'Verbose', 1, 'ShowPlots', false);
Classifier = fitcknn(ExtFeatTrain, FaultIndTrain,...
        'OptimizeHyperparameters','all','HyperparameterOptimizationOptions',opts);

[Alan Weiss]

To use an output function you will have to use bayesopt directly, not indirectly in a fitcknn as you have done. Write your objective function to call fitcknn, and then use the classification error as the objective function (or whatever objective suits you). For a similar example, see Optimize a Cross-Validated SVM Classifier Using bayesopt.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation

Answer 2

[Greg Heath]

For good estimators I tend to be satisfied when the squared error is at least 100 times smaller than that of the reference naive guess

            outref = mean(target)

with

            MSEref = mean(var(target',0))

i.e.,

             MSEgoal = 0.01*MSEref

I have hundreds of examples in both comp.soft-sys.matlab and ANSWERS.

Hope this helps.

Thank you for formally accepting my answer

Greg