Stopping Criteria in bayesopt for Early Termination

I'm working with bayesopt for Bayesian optimization in MATLAB. I want to implement a stopping criterion that terminates the optimization process when a specific condition is met like minimum tolerence level 1e-3, allowing for early termination if improvement becomes insignificant. Here's the relevant portion of my bayesopt command:

results = bayesopt(@optimizeParameters, parRange, ...
    'IsObjectiveDeterministic', false, ...
    'AcquisitionFunctionName', 'expected-improvement-plus', ...
    'Verbose', 0, ...
    'ExplorationRatio', 0.6, ...
    'MaxObjectiveEvaluations', 100);

Could you provide guidance on how to best achieve this using bayesopt.

Answer 1

[Tejas]

Hello Muhammad,

It seems you want a method to halt Bayesian optimization prematurely when a certain parameter value falls beneath a specified tolerance level.

This objective can be achieved by crafting a custom function designed to halt the optimization process once a specific condition is satisfied. This function is then invoked using the ‘OutputFcn’ property of ‘bayesopt’. The ‘OutputFcn’ triggers this custom function at the end of each iteration.

I have created a custom function named ‘terminationFunc’ to demonstrate this approach. Here is how it operates:

• The function receives the results and state of the current iteration as input.
• It monitors the ‘ObjectiveMinimumTrace’ parameter across each iteration.
• Optimization is halted early if this parameter's value drops below the tolerance threshold of 1e-3.

function stop = terminationFunc(results, state)
    persistent bestObjective
    stop = false;
    
    if strcmp(state, 'iteration')
        if isempty(bestObjective) % True for first iteration
            bestObjective = results.ObjectiveMinimumTrace(end);
            fprintf('Initializing bestObjective: %e\n', bestObjective); 
        else
            currentObjective = results.ObjectiveMinimumTrace(end);
            improvement = abs(currentObjective - bestObjective);
            fprintf('Current Improvement: %e\n', improvement);

            % Check if improvement is less than the tolerance
            if improvement < 1e-3
                fprintf('Stopping Optimization: Improvement below minimum tolerance level of 1e-3.\n');
                stop = true;  % Signal to stop optimization
            else
                bestObjective = currentObjective;
            end
        end
    elseif strcmp(state, 'initial')
        fprintf('Optimization initializing...\n'); 
    elseif strcmp(state, 'done')
        fprintf('Optimization completed.\n'); 
    end
end

The function can be executed using the following syntax:

results = bayesopt(@optimizeParameters, parRange, ...
    'IsObjectiveDeterministic', false, ...
    'AcquisitionFunctionName', 'expected-improvement-plus', ...
    'Verbose', 0, ...
    'ExplorationRatio', 0.6, ...
    'MaxObjectiveEvaluations', 100,'OutputFcn',@terminationFunc);

The effectiveness of this approach was evaluated using a sample Bayesian optimization:

load ionosphere
rng default
num = optimizableVariable('n',[1,30],'Type','integer');
dst = optimizableVariable('dst',{'chebychev','euclidean','minkowski'},'Type','categorical');
c = cvpartition(351,'Kfold',5);
fun = @(x)kfoldLoss(fitcknn(X,Y,'CVPartition',c,'NumNeighbors',x.n,'Distance',char(x.dst),'NSMethod','exhaustive'));
results = bayesopt(fun,[num,dst],'Verbose',0,'AcquisitionFunctionName','expected-improvement-plus','OutputFcn',@terminationFunc);

File is saved as ‘bayesian.m’. Below is the screenshot of the output:

To know more about the ‘OutputFcn’ property of ‘bayesopt’ function, refer to this documentation:

https://www.mathworks.com/help/stats/bayesopt.html#:~:text=OutputFcn%20%E2%80%94%20Function,cell%20%7C%20function_handle .

Hope it helps!

[Muhammad]

Thank you for your help