BayesianOptimization

Bayesian optimization results

Description

A BayesianOptimization object contains the results of a Bayesian optimization. It is the output of bayesopt or a fit function that accepts the OptimizeHyperparameters name-value pair such as fitcdiscr. In addition, a BayesianOptimization object contains data for each iteration of bayesopt that can be accessed by a plot function or an output function.

Creation

Create a BayesianOptimization object by using the bayesopt function or one of the following fit functions with the OptimizeHyperparameters name-value argument.

• Classification fit functions: fitcdiscr, fitcecoc, fitcensemble, fitcgam, fitckernel, fitcknn, fitclinear, fitcnb, fitcnet, fitcsvm, fitctree

• Regression fit functions: fitrensemble, fitrgam, fitrgp, fitrkernel, fitrlinear, fitrnet, fitrsvm, fitrtree

You can also create a BayesianOptimization object by using the fitcauto and fitrauto functions, or by using the Classification Learner and Regression Learner apps. For more information, see Ways to Perform Bayesian Optimization.

Properties

Problem Definition Properties

ObjectiveFcn — ObjectiveFcn argument used by bayesopt

function handle

This property is read-only.

ObjectiveFcn argument used by bayesopt, specified as a function handle.

• If you call bayesopt directly, ObjectiveFcn is the bayesopt objective function argument.

• If you call a fit function containing the 'OptimizeHyperparameters' name-value pair argument, ObjectiveFcn is a function handle that returns the misclassification rate for classification or returns the logarithm of one plus the cross-validation loss for regression, measured by five-fold cross-validation.

Data Types: function_handle

VariableDescriptions — VariableDescriptions argument that bayesopt used

vector of optimizableVariable objects

This property is read-only.

VariableDescriptions argument that bayesopt used, specified as a vector of optimizableVariable objects.

• If you called bayesopt directly, VariableDescriptions is the bayesopt variable description argument.

• If you called a fit function with the OptimizeHyperparameters name-value pair, VariableDescriptions is the vector of hyperparameters.

Options — Options that bayesopt used

structure

This property is read-only.

Options that bayesopt used, specified as a structure.

• If you called bayesopt directly, Options is the options used in bayesopt, which are the name-value pairs See bayesopt Input Arguments.

• If you called a fit function with the OptimizeHyperparameters name-value pair, Options are the default bayesopt options, modified by the HyperparameterOptimizationOptions name-value pair.

Options is a read-only structure containing the following fields.

Option Name	Meaning
AcquisitionFunctionName	Acquisition function name (with dashes removed). See Acquisition Function Types.
IsObjectiveDeterministic	true means the objective function is deterministic, false otherwise.
ExplorationRatio	Used only when AcquisitionFunctionName is 'expectedimprovementplus' or 'expectedimprovement-persecondplus'. See Plus.
MaxObjectiveEvaluations	Objective function evaluation limit.
MaxTime	Time limit.
XConstraintFcn	Deterministic constraints on variables. See Deterministic Constraints — XConstraintFcn.
ConditionalVariableFcn	Conditional variable constraints. See Conditional Constraints — ConditionalVariableFcn.
NumCoupledConstraints	Number of coupled constraints. See Coupled Constraints.
CoupledConstraintTolerances	Coupled constraint tolerances. See Coupled Constraints.
AreCoupledConstraintsDeterministic	Logical vector specifying whether each coupled constraint is deterministic.
Verbose	Command-line display level.
OutputFcn	Function called after each iteration. See Bayesian Optimization Output Functions.
SaveVariableName	Variable name for the @assignInBase output function.
SaveFileName	File name for the @saveToFile output function.
PlotFcn	Plot function called after each iteration. See Bayesian Optimization Plot Functions
InitialX	Points where bayesopt evaluated the objective function.
InitialObjective	Objective function values at InitialX.
InitialConstraintViolations	Coupled constraint function values at InitialX.
InitialErrorValues	Error values at InitialX.
InitialObjectiveEvaluationTimes	Objective function evaluation times at InitialX.
InitialIterationTimes	Time for each iteration, including objective function evaluation and other computations.

Data Types: struct

Solution Properties

MinObjective — Minimum observed value of objective function

real scalar

This property is read-only.

Minimum observed value of objective function, specified as a real scalar. When there are coupled constraints or evaluation errors, this value is the minimum over all observed points that are feasible according to the final constraint and Error models.

Data Types: double

XAtMinObjective — Observed point with minimum objective function value

1-by-D table

This property is read-only.

Observed point with minimum objective function value, specified as a 1-by-D table, where D is the number of variables.

Data Types: table

MinEstimatedObjective — Estimated objective function value

real scalar

This property is read-only.

Estimated objective function value at XAtMinEstimatedObjective, specified as a real scalar.

MinEstimatedObjective is the mean value of the posterior distribution of the final objective model. The software estimates the MinEstimatedObjective value by passing XAtMinEstimatedObjective to the object function predictObjective.

Data Types: double

XAtMinEstimatedObjective — Point with minimum upper confidence bound of objective function value

1-by-D table

This property is read-only.

Point with the minimum upper confidence bound of the objective function value among the visited points, specified as a 1-by-D table, where D is the number of variables. The software uses the final objective model to find the upper confidence bounds of the visited points.

XAtMinEstimatedObjective is the same as the best point returned by the bestPoint function with the default criterion ('min-visited-upper-confidence-interval').

Data Types: table

NumObjectiveEvaluations — Number of objective function evaluations

positive integer

This property is read-only.

Number of objective function evaluations, specified as a positive integer. This includes the initial evaluations to form a posterior model as well as evaluation during the optimization iterations.

Data Types: double

TotalElapsedTime — Total elapsed time of optimization in seconds

positive scalar

This property is read-only.

Total elapsed time of optimization in seconds, specified as a positive scalar.

Data Types: double

NextPoint — Next point to evaluate if optimization continues

1-by-D table

This property is read-only.

Next point to evaluate if optimization continues, specified as a 1-by-D table, where D is the number of variables.

Data Types: table

Trace Properties

XTrace — Points where the objective function was evaluated

T-by-D table

This property is read-only.

Points where the objective function was evaluated, specified as a T-by-D table, where T is the number of evaluation points and D is the number of variables.

Data Types: table

ObjectiveTrace — Objective function values

column vector of length T

This property is read-only.

Objective function values, specified as a column vector of length T, where T is the number of evaluation points. ObjectiveTrace contains the history of objective function evaluations.

Data Types: double

ObjectiveEvaluationTimeTrace — Objective function evaluation times

column vector of length T

This property is read-only.

Objective function evaluation times, specified as a column vector of length T, where T is the number of evaluation points. ObjectiveEvaluationTimeTrace includes the time in evaluating coupled constraints, because the objective function computes these constraints.

Data Types: double

IterationTimeTrace — Iteration times

column vector of length T

This property is read-only.

Iteration times, specified as a column vector of length T, where T is the number of evaluation points. IterationTimeTrace includes both objective function evaluation time and other overhead.

Data Types: double

ConstraintsTrace — Coupled constraint values

T-by-K array

This property is read-only.

Coupled constraint values, specified as a T-by-K array, where T is the number of evaluation points and K is the number of coupled constraints.

Data Types: double

ErrorTrace — Error indications

column vector of length T of -1 or 1 entries

This property is read-only.

Error indications, specified as a column vector of length T of -1 or 1 entries, where T is the number of evaluation points. Each 1 entry indicates that the objective function errored or returned NaN on the corresponding point in XTrace. Each -1 entry indicates that the objective function value was computed.

Data Types: double

FeasibilityTrace — Feasibility indications

logical column vector of length T

This property is read-only.

Feasibility indications, specified as a logical column vector of length T, where T is the number of evaluation points. Each 1 entry indicates that the final constraint model predicts feasibility at the corresponding point in XTrace.

Data Types: logical

FeasibilityProbabilityTrace — Probability that evaluation point is feasible

column vector of length T

This property is read-only.

Probability that evaluation point is feasible, specified as a column vector of length T, where T is the number of evaluation points. The probabilities come from the final constraint model, including the error constraint model, on the corresponding points in XTrace.

Data Types: double

IndexOfMinimumTrace — Which evaluation gave minimum feasible objective

column vector of integer indices of length T

This property is read-only.

Which evaluation gave minimum feasible objective, specified as a column vector of integer indices of length T, where T is the number of evaluation points. Feasibility is determined with respect to the constraint models that existed at each iteration, including the error constraint model.

Data Types: double

ObjectiveMinimumTrace — Minimum observed objective

column vector of length T

This property is read-only.

Minimum observed objective, specified as a column vector of length T, where T is the number of evaluation points.

Data Types: double

EstimatedObjectiveMinimumTrace — Estimated objective

column vector of length T

This property is read-only.

Estimated objective, specified as a column vector of length T, where T is the number of evaluation points. The estimated objective at each iteration is determined with respect to the objective model at that iteration. At each iteration, the software uses the object function predictObjective to estimate the objective function value at the point with the minimum upper confidence bound of the objective function among the visited points.

Data Types: double

UserDataTrace — Auxiliary data from the objective function

cell array of length T

This property is read-only.

Auxiliary data from the objective function, specified as a cell array of length T, where T is the number of evaluation points. Each entry in the cell array is the UserData returned in the third output of the objective function.

Data Types: cell

Object Functions

bestPoint	Best point in a Bayesian optimization according to a criterion
plot	Plot Bayesian optimization results
predictConstraints	Predict coupled constraint violations at a set of points
predictError	Predict error value at a set of points
predictObjective	Predict objective function at a set of points
predictObjectiveEvaluationTime	Predict objective function run times at a set of points
resume	Resume a Bayesian optimization

Examples

Create a BayesianOptimization Object Using bayesopt

This example shows how to create a BayesianOptimization object by using bayesopt to minimize cross-validation loss.

Optimize hyperparameters of a KNN classifier for the ionosphere data, that is, find KNN hyperparameters that minimize the cross-validation loss. Have bayesopt minimize over the following hyperparameters:

• Nearest-neighborhood sizes from 1 to 30

• Distance functions 'chebychev', 'euclidean', and 'minkowski'.

For reproducibility, set the random seed, set the partition, and set the AcquisitionFunctionName option to 'expected-improvement-plus'. To suppress iterative display, set 'Verbose' to 0. Pass the partition c and fitting data X and Y to the objective function fun by creating fun as an anonymous function that incorporates this data. See Parameterizing Functions.

load ionosphere
rng default
num = optimizableVariable('n',[1,30],'Type','integer');
dst = optimizableVariable('dst',{'chebychev','euclidean','minkowski'},'Type','categorical');
c = cvpartition(351,'Kfold',5);
fun = @(x)kfoldLoss(fitcknn(X,Y,'CVPartition',c,'NumNeighbors',x.n,...
    'Distance',char(x.dst),'NSMethod','exhaustive'));
results = bayesopt(fun,[num,dst],'Verbose',0,...
    'AcquisitionFunctionName','expected-improvement-plus')

results = 
  BayesianOptimization with properties:

                      ObjectiveFcn: @(x)kfoldLoss(fitcknn(X,Y,'CVPartition',c,'NumNeighbors',x.n,'Distance',char(x.dst),'NSMethod','exhaustive'))
              VariableDescriptions: [1x2 optimizableVariable]
                           Options: [1x1 struct]
                      MinObjective: 0.1197
                   XAtMinObjective: [1x2 table]
             MinEstimatedObjective: 0.1213
          XAtMinEstimatedObjective: [1x2 table]
           NumObjectiveEvaluations: 30
                  TotalElapsedTime: 23.0930
                         NextPoint: [1x2 table]
                            XTrace: [30x2 table]
                    ObjectiveTrace: [30x1 double]
                  ConstraintsTrace: []
                     UserDataTrace: {30x1 cell}
      ObjectiveEvaluationTimeTrace: [30x1 double]
                IterationTimeTrace: [30x1 double]
                        ErrorTrace: [30x1 double]
                  FeasibilityTrace: [30x1 logical]
       FeasibilityProbabilityTrace: [30x1 double]
               IndexOfMinimumTrace: [30x1 double]
             ObjectiveMinimumTrace: [30x1 double]
    EstimatedObjectiveMinimumTrace: [30x1 double]

Create a BayesianOptimization Object Using a Fit Function

This example shows how to minimize the cross-validation loss in the ionosphere data using Bayesian optimization of an SVM classifier.

Load the data.

load ionosphere

Optimize the classification using the 'auto' parameters.

rng default % For reproducibility
Mdl = fitcsvm(X,Y,'OptimizeHyperparameters','auto')

|====================================================================================================================|
| Iter | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   | BoxConstraint|  KernelScale |  Standardize |
|      | result |             | runtime     | (observed)  | (estim.)    |              |              |              |
|====================================================================================================================|
|    1 | Best   |     0.35897 |     0.54397 |     0.35897 |     0.35897 |       3.8653 |       961.53 |         true |
|    2 | Best   |     0.12821 |      9.5694 |     0.12821 |     0.15646 |       429.99 |       0.2378 |        false |
|    3 | Accept |     0.35897 |     0.10545 |     0.12821 |      0.1315 |      0.11801 |       8.9479 |        false |
|    4 | Accept |      0.1339 |      4.7223 |     0.12821 |     0.12965 |    0.0010694 |    0.0032063 |         true |
|    5 | Accept |     0.15954 |       10.64 |     0.12821 |     0.12824 |       973.65 |      0.15179 |        false |
|    6 | Accept |     0.35897 |    0.054807 |     0.12821 |     0.12825 |    0.0025729 |       956.93 |        false |
|    7 | Accept |      0.1339 |      4.7428 |     0.12821 |     0.12824 |       997.19 |       2.1789 |         true |
|    8 | Accept |     0.35897 |     0.12047 |     0.12821 |     0.12826 |    0.0053099 |       928.24 |         true |
|    9 | Accept |     0.23077 |     0.10806 |     0.12821 |     0.12826 |    0.0020013 |       0.5832 |        false |
|   10 | Accept |     0.35897 |     0.15974 |     0.12821 |     0.14631 |    0.0044264 |       92.226 |        false |
|   11 | Accept |      0.1339 |     0.26601 |     0.12821 |     0.14682 |    0.0030346 |     0.015604 |        false |
|   12 | Accept |      0.1396 |      9.1847 |     0.12821 |     0.13146 |    0.0017799 |    0.0010468 |        false |
|   13 | Best   |     0.12251 |    0.083765 |     0.12251 |     0.12789 |    0.0032101 |     0.033317 |        false |
|   14 | Accept |     0.12536 |     0.14405 |     0.12251 |     0.12642 |    0.0037535 |     0.042671 |        false |
|   15 | Accept |     0.12821 |     0.21287 |     0.12251 |      0.1264 |    0.0010124 |      0.28112 |         true |
|   16 | Accept |     0.12821 |    0.087625 |     0.12251 |      0.1229 |    0.0010187 |      0.19559 |         true |
|   17 | Accept |     0.12251 |     0.11186 |     0.12251 |      0.1259 |     0.018732 |      0.15447 |         true |
|   18 | Best   |     0.11681 |      0.1019 |     0.11681 |     0.12251 |    0.0058673 |     0.067109 |         true |
|   19 | Accept |     0.11966 |     0.25491 |     0.11681 |     0.11836 |    0.0062672 |     0.056993 |         true |
|   20 | Accept |      0.1339 |      4.2124 |     0.11681 |     0.11811 |      0.72635 |     0.061362 |         true |
|====================================================================================================================|
| Iter | Eval   | Objective   | Objective   | BestSoFar   | BestSoFar   | BoxConstraint|  KernelScale |  Standardize |
|      | result |             | runtime     | (observed)  | (estim.)    |              |              |              |
|====================================================================================================================|
|   21 | Accept |     0.13675 |    0.056627 |     0.11681 |     0.11821 |     0.014105 |       1.2796 |         true |
|   22 | Accept |     0.35897 |     0.07056 |     0.11681 |     0.11711 |     0.011627 |       8.4005 |         true |
|   23 | Accept |     0.12251 |     0.16781 |     0.11681 |     0.11705 |      0.10003 |      0.60258 |         true |
|   24 | Accept |     0.13105 |     0.65504 |     0.11681 |     0.11714 |    0.0023102 |     0.015314 |         true |
|   25 | Accept |     0.15385 |    0.054876 |     0.11681 |     0.11705 |    0.0010278 |      0.68833 |         true |
|   26 | Accept |      0.1339 |     0.11083 |     0.11681 |     0.11705 |    0.0028538 |      0.14353 |        false |
|   27 | Accept |      0.1339 |      9.9544 |     0.11681 |     0.11794 |        958.3 |      0.79153 |         true |
|   28 | Accept |      0.1339 |     0.16088 |     0.11681 |     0.11752 |      0.21861 |      0.16713 |        false |
|   29 | Accept |     0.12536 |     0.11811 |     0.11681 |      0.1183 |      0.89788 |       1.2063 |         true |
|   30 | Accept |     0.12251 |    0.071874 |     0.11681 |      0.1188 |    0.0010229 |     0.083578 |         true |

__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 69.5866 seconds
Total objective function evaluation time: 56.8479

Best observed feasible point:
    BoxConstraint    KernelScale    Standardize
    _____________    ___________    ___________

      0.0058673       0.067109         true    

Observed objective function value = 0.11681
Estimated objective function value = 0.1188
Function evaluation time = 0.1019

Best estimated feasible point (according to models):
    BoxConstraint    KernelScale    Standardize
    _____________    ___________    ___________

      0.0058673       0.067109         true    

Estimated objective function value = 0.1188
Estimated function evaluation time = 0.16821

Mdl = 
  ClassificationSVM
                         ResponseName: 'Y'
                CategoricalPredictors: []
                           ClassNames: {'b'  'g'}
                       ScoreTransform: 'none'
                      NumObservations: 351
    HyperparameterOptimizationResults: [1x1 BayesianOptimization]
                                Alpha: [87x1 double]
                                 Bias: -0.2196
                     KernelParameters: [1x1 struct]
                                   Mu: [0.8917 0 0.6413 0.0444 0.6011 0.1159 0.5501 0.1194 0.5118 0.1813 0.4762 0.1550 0.4008 0.0934 0.3442 0.0711 0.3819 -0.0036 0.3594 -0.0240 0.3367 0.0083 0.3625 -0.0574 0.3961 -0.0712 0.5416 -0.0695 ... ] (1x34 double)
                                Sigma: [0.3112 0 0.4977 0.4414 0.5199 0.4608 0.4927 0.5207 0.5071 0.4839 0.5635 0.4948 0.6222 0.4949 0.6528 0.4584 0.6180 0.4968 0.6263 0.5191 0.6098 0.5182 0.6038 0.5275 0.5785 0.5085 0.5162 0.5500 ... ] (1x34 double)
                       BoxConstraints: [351x1 double]
                      ConvergenceInfo: [1x1 struct]
                      IsSupportVector: [351x1 logical]
                               Solver: 'SMO'

The fit achieved about 12% loss for the default 5-fold cross validation.

Examine the BayesianOptimization object that is returned in the HyperparameterOptimizationResults property of the returned model.

disp(Mdl.HyperparameterOptimizationResults)

  BayesianOptimization with properties:

                      ObjectiveFcn: @createObjFcn/inMemoryObjFcn
              VariableDescriptions: [5x1 optimizableVariable]
                           Options: [1x1 struct]
                      MinObjective: 0.1168
                   XAtMinObjective: [1x3 table]
             MinEstimatedObjective: 0.1188
          XAtMinEstimatedObjective: [1x3 table]
           NumObjectiveEvaluations: 30
                  TotalElapsedTime: 69.5866
                         NextPoint: [1x3 table]
                            XTrace: [30x3 table]
                    ObjectiveTrace: [30x1 double]
                  ConstraintsTrace: []
                     UserDataTrace: {30x1 cell}
      ObjectiveEvaluationTimeTrace: [30x1 double]
                IterationTimeTrace: [30x1 double]
                        ErrorTrace: [30x1 double]
                  FeasibilityTrace: [30x1 logical]
       FeasibilityProbabilityTrace: [30x1 double]
               IndexOfMinimumTrace: [30x1 double]
             ObjectiveMinimumTrace: [30x1 double]
    EstimatedObjectiveMinimumTrace: [30x1 double]

Version History

Introduced in R2016b