Main Content

Automated Classifier Selection with Bayesian and ASHA Optimization

This example shows how to use fitcauto to automatically try a selection of classification model types with different hyperparameter values, given training predictor and response data. By default, the function uses Bayesian optimization to select and assess models. If your training data set contains many observations, you can use an asynchronous successive halving algorithm (ASHA) instead. After the optimization is complete, fitcauto returns the model, trained on the entire data set, that is expected to best classify new data. Check the model performance on test data.

Load Sample Data

This example uses the 1994 census data stored in census1994.mat. The data set consists of demographic information from the US Census Bureau that can be used to predict whether an individual makes over $50,000 per year.

Load the sample data census1994, which contains the training data adultdata and the test data adulttest. Preview the first few rows of the training data set.

load census1994
head(adultdata)
    age       workClass          fnlwgt      education    education_num       marital_status           occupation        relationship     race      sex      capital_gain    capital_loss    hours_per_week    native_country    salary
    ___    ________________    __________    _________    _____________    _____________________    _________________    _____________    _____    ______    ____________    ____________    ______________    ______________    ______

    39     State-gov                77516    Bachelors         13          Never-married            Adm-clerical         Not-in-family    White    Male          2174             0                40          United-States     <=50K 
    50     Self-emp-not-inc         83311    Bachelors         13          Married-civ-spouse       Exec-managerial      Husband          White    Male             0             0                13          United-States     <=50K 
    38     Private             2.1565e+05    HS-grad            9          Divorced                 Handlers-cleaners    Not-in-family    White    Male             0             0                40          United-States     <=50K 
    53     Private             2.3472e+05    11th               7          Married-civ-spouse       Handlers-cleaners    Husband          Black    Male             0             0                40          United-States     <=50K 
    28     Private             3.3841e+05    Bachelors         13          Married-civ-spouse       Prof-specialty       Wife             Black    Female           0             0                40          Cuba              <=50K 
    37     Private             2.8458e+05    Masters           14          Married-civ-spouse       Exec-managerial      Wife             White    Female           0             0                40          United-States     <=50K 
    49     Private             1.6019e+05    9th                5          Married-spouse-absent    Other-service        Not-in-family    Black    Female           0             0                16          Jamaica           <=50K 
    52     Self-emp-not-inc    2.0964e+05    HS-grad            9          Married-civ-spouse       Exec-managerial      Husband          White    Male             0             0                45          United-States     >50K  

Each row contains the demographic information for one adult. The last column salary shows whether a person has a salary less than or equal to $50,000 per year or greater than $50,000 per year.

Remove observations from adultdata and adulttest that contain missing values.

adultdata = rmmissing(adultdata);
adulttest = rmmissing(adulttest);

Use Automated Model Selection with Bayesian Optimization

Find an appropriate classifier for the data in adultdata by using fitcauto. By default, fitcauto uses Bayesian optimization to select models and their hyperparameter values, and computes the cross-validation classification error (Validation loss) for each model. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.

Set the observation weights, and specify to run the Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. Because of the complexity of the optimization, this process can take some time, especially for larger data sets.

bayesianOptions = struct("UseParallel",true);
[bayesianMdl,bayesianResults] = fitcauto(adultdata,"salary","Weights","fnlwgt", ...
    "HyperparameterOptimizationOptions",bayesianOptions);
Warning: Data set has more than 10000 observations. Because ASHA optimization often finds good solutions faster than Bayesian optimization for data sets with many observations, try specifying the 'Optimizer' field value as 'asha' in the 'HyperparameterOptimizationOptions' value structure.
Starting parallel pool (parpool) using the 'Processes' profile ...
Connected to parallel pool with 6 workers.
Copying objective function to workers...
Done copying objective function to workers.
Learner types to explore: ensemble, nb, svm, tree
Total iterations (MaxObjectiveEvaluations): 120
Total time (MaxTime): Inf

|=======================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                                         |
|=======================================================================================================================================================|
|    1 |       6 | Best   |    0.24677 |            43.754 |         0.24677 |         0.24677 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  207 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       8946 |
|    2 |       6 | Accept |    0.24677 |           0.88853 |         0.24677 |         0.24677 |         tree | MinLeafSize:                       6845 |
|    3 |       6 | Best   |    0.14543 |            2.2401 |         0.14543 |         0.19002 |         tree | MinLeafSize:                         45 |
|    4 |       6 | Accept |    0.15419 |            80.045 |         0.14543 |         0.19002 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  209 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                          2 |
|    5 |       6 | Accept |    0.14928 |            93.473 |         0.14543 |         0.18783 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  201 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        480 |
|    6 |       6 | Accept |    0.15158 |            44.195 |         0.14543 |         0.15158 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                        0.0088611 |
|      |         |        |            |                   |                 |                 |              | Standardize:                      false |
|    7 |       6 | Accept |    0.14889 |            139.61 |         0.14543 |         0.15158 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  297 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        510 |
|    8 |       6 | Accept |    0.18439 |            1.4145 |         0.14543 |         0.15158 |         tree | MinLeafSize:                       3104 |
|    9 |       6 | Accept |    0.14554 |             161.4 |         0.14543 |         0.14554 |          svm | BoxConstraint:                  0.65272 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     6.0716 |
|   10 |       6 | Accept |    0.15127 |            166.83 |         0.14543 |         0.14881 |          svm | BoxConstraint:                  0.29711 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     12.637 |
|   11 |       6 | Accept |    0.16715 |            76.475 |         0.14543 |         0.14881 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                          0.22835 |
|      |         |        |            |                   |                 |                 |              | Standardize:                       true |
|   12 |       6 | Accept |    0.20124 |            101.74 |         0.14543 |         0.14881 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                            18.24 |
|      |         |        |            |                   |                 |                 |              | Standardize:                       true |
|   13 |       6 | Accept |    0.24677 |           0.46731 |         0.14543 |         0.14881 |         tree | MinLeafSize:                      12859 |
|   14 |       6 | Accept |      0.168 |           0.69093 |         0.14543 |         0.14881 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |                 |              | Width:                              NaN |
|      |         |        |            |                   |                 |                 |              | Standardize:                          - |
|   15 |       6 | Accept |    0.24677 |            3.7861 |         0.14543 |         0.17365 |          svm | BoxConstraint:                  0.24765 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                  0.0016358 |
|   16 |       6 | Accept |    0.14757 |            1.8122 |         0.14543 |         0.17365 |         tree | MinLeafSize:                        135 |
|   17 |       6 | Accept |      0.168 |           0.36422 |         0.14543 |         0.17099 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |                 |              | Width:                              NaN |
|      |         |        |            |                   |                 |                 |              | Standardize:                          - |
|   18 |       6 | Accept |    0.23488 |            55.736 |         0.14543 |         0.17099 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  243 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       5284 |
|   19 |       6 | Accept |    0.24677 |            48.557 |         0.14543 |         0.17099 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  217 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       6309 |
|   20 |       6 | Accept |    0.16312 |            44.074 |         0.14543 |         0.16836 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                         0.043878 |
|      |         |        |            |                   |                 |                 |              | Standardize:                       true |
|=======================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                                         |
|=======================================================================================================================================================|
|   21 |       6 | Accept |    0.19857 |            99.234 |         0.14543 |         0.17104 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                           3.1517 |
|      |         |        |            |                   |                 |                 |              | Standardize:                       true |
|   22 |       6 | Accept |    0.15361 |            1.5262 |         0.14543 |         0.16123 |         tree | MinLeafSize:                        338 |
|   23 |       6 | Accept |    0.15417 |            92.662 |         0.14543 |         0.16123 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  277 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        101 |
|   24 |       6 | Accept |     0.1628 |            2.9924 |         0.14543 |         0.15776 |         tree | MinLeafSize:                          6 |
|   25 |       6 | Accept |    0.15479 |            85.937 |         0.14543 |         0.15776 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                           4.0678 |
|      |         |        |            |                   |                 |                 |              | Standardize:                      false |
|   26 |       6 | Best   |     0.1445 |            1.7904 |          0.1445 |         0.15547 |         tree | MinLeafSize:                        105 |
|   27 |       6 | Accept |    0.15763 |            2.8146 |          0.1445 |         0.15223 |         tree | MinLeafSize:                          9 |
|   28 |       6 | Accept |    0.24677 |            1.9596 |          0.1445 |         0.15223 |          svm | BoxConstraint:                0.0047884 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                  0.0048876 |
|   29 |       6 | Accept |    0.15158 |            42.575 |          0.1445 |         0.15223 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                        0.0075341 |
|      |         |        |            |                   |                 |                 |              | Standardize:                      false |
|   30 |       6 | Accept |    0.16813 |            1.1328 |          0.1445 |         0.15262 |         tree | MinLeafSize:                        598 |
|   31 |       6 | Accept |    0.14994 |            2.2018 |          0.1445 |         0.15084 |         tree | MinLeafSize:                         21 |
|   32 |       6 | Accept |      0.168 |            0.4177 |          0.1445 |         0.15084 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |                 |              | Width:                              NaN |
|      |         |        |            |                   |                 |                 |              | Standardize:                          - |
|   33 |       6 | Accept |    0.14535 |            165.79 |          0.1445 |         0.15084 |          svm | BoxConstraint:                  0.14315 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.2371 |
|   34 |       6 | Accept |    0.17901 |            5.6933 |          0.1445 |          0.1502 |         tree | MinLeafSize:                          2 |
|   35 |       6 | Accept |      0.168 |           0.48234 |          0.1445 |          0.1502 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |                 |              | Width:                              NaN |
|      |         |        |            |                   |                 |                 |              | Standardize:                          - |
|   36 |       6 | Accept |    0.16795 |            1.0938 |          0.1445 |         0.15088 |         tree | MinLeafSize:                        570 |
|   37 |       6 | Accept |    0.18403 |           0.65343 |          0.1445 |         0.15009 |         tree | MinLeafSize:                       2119 |
|   38 |       6 | Accept |    0.24677 |            2.0224 |          0.1445 |         0.15009 |          svm | BoxConstraint:                   216.63 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                   0.006762 |
|   39 |       6 | Accept |    0.14945 |            83.448 |          0.1445 |         0.15009 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  223 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        345 |
|   40 |       6 | Accept |    0.16129 |            90.614 |          0.1445 |         0.15009 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  278 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        388 |
|=======================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                                         |
|=======================================================================================================================================================|
|   41 |       6 | Accept |     0.1647 |            32.478 |          0.1445 |         0.15009 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  256 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       7032 |
|   42 |       6 | Accept |    0.16698 |            71.592 |          0.1445 |         0.15009 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  226 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        861 |
|   43 |       6 | Accept |    0.15872 |            189.88 |          0.1445 |         0.15009 |          svm | BoxConstraint:                   11.079 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     155.11 |
|   44 |       6 | Best   |    0.14353 |             1.687 |         0.14353 |         0.14816 |         tree | MinLeafSize:                         79 |
|   45 |       6 | Accept |    0.15374 |            39.891 |         0.14353 |         0.14816 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                        0.0035642 |
|      |         |        |            |                   |                 |                 |              | Standardize:                       true |
|   46 |       6 | Accept |    0.14979 |             83.79 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  203 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                        911 |
|   47 |       6 | Accept |    0.24677 |            2.0222 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   2.0475 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                  0.0049311 |
|   48 |       6 | Accept |    0.20139 |            94.777 |         0.14353 |         0.14816 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                           93.844 |
|      |         |        |            |                   |                 |                 |              | Standardize:                       true |
|   49 |       6 | Accept |      0.168 |           0.38418 |         0.14353 |         0.14816 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |                 |              | Width:                              NaN |
|      |         |        |            |                   |                 |                 |              | Standardize:                          - |
|   50 |       6 | Accept |    0.15146 |            47.829 |         0.14353 |         0.14816 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |                 |              | Width:                          0.36089 |
|      |         |        |            |                   |                 |                 |              | Standardize:                      false |
|   51 |       6 | Accept |     0.1686 |            27.403 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  261 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       8358 |
|   52 |       6 | Accept |    0.23867 |             195.4 |         0.14353 |         0.14816 |          svm | BoxConstraint:                 0.011694 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     18.422 |
|   53 |       6 | Accept |    0.15319 |            162.54 |         0.14353 |         0.14816 |          svm | BoxConstraint:                 0.010059 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     4.0044 |
|   54 |       6 | Accept |    0.15501 |            79.775 |         0.14353 |         0.14816 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  232 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                         24 |
|   55 |       6 | Accept |    0.16874 |            2864.2 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   2.3437 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     1.5289 |
|   56 |       6 | Accept |    0.15742 |            2870.4 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   586.48 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     6.2332 |
|   57 |       6 | Accept |    0.15119 |             81.33 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  209 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       1622 |
|   58 |       6 | Accept |    0.16761 |            3059.5 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   380.05 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.8304 |
|   59 |       6 | Accept |     0.1522 |            77.374 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  204 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       1676 |
|   60 |       6 | Accept |    0.17435 |            3048.3 |         0.14353 |         0.14816 |          svm | BoxConstraint:                    1.424 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     1.1302 |
|=======================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                                         |
|=======================================================================================================================================================|
|   61 |       6 | Accept |    0.15944 |            62.648 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  228 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       2945 |
|   62 |       6 | Accept |     0.1622 |            2716.5 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   8.1184 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     2.3874 |
|   63 |       6 | Accept |    0.15986 |            53.967 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  203 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       3145 |
|   64 |       6 | Accept |    0.14553 |            150.62 |         0.14353 |         0.14816 |          svm | BoxConstraint:                  0.26261 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     5.0699 |
|   65 |       6 | Accept |    0.14799 |            157.27 |         0.14353 |         0.14816 |          svm | BoxConstraint:                 0.081785 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     4.9516 |
|   66 |       6 | Accept |    0.14833 |            158.24 |         0.14353 |         0.14816 |          svm | BoxConstraint:                  0.74137 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     12.172 |
|   67 |       6 | Accept |    0.20535 |            205.52 |         0.14353 |         0.14816 |          svm | BoxConstraint:                  0.26823 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     60.591 |
|   68 |       6 | Accept |    0.15533 |            173.46 |         0.14353 |         0.14816 |          svm | BoxConstraint:                 0.031445 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     7.4327 |
|   69 |       6 | Accept |    0.16052 |            179.49 |         0.14353 |         0.14816 |          svm | BoxConstraint:                0.0062986 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                      4.855 |
|   70 |       6 | Accept |    0.16085 |            46.246 |         0.14353 |         0.14816 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  211 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       3991 |
|   71 |       6 | Accept |    0.15032 |            159.22 |         0.14353 |         0.14816 |          svm | BoxConstraint:                  0.34417 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     12.342 |
|   72 |       6 | Accept |    0.18532 |            46.838 |         0.14353 |         0.14816 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  202 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       4515 |
|   73 |       6 | Accept |    0.18875 |            189.42 |         0.14353 |         0.14816 |          svm | BoxConstraint:                0.0019135 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     5.2624 |
|   74 |       6 | Accept |    0.15372 |            161.15 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   0.2716 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     15.245 |
|   75 |       6 | Accept |    0.15399 |            159.07 |         0.14353 |         0.14816 |          svm | BoxConstraint:                  0.01694 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     4.9823 |
|   76 |       6 | Accept |    0.15324 |            160.44 |         0.14353 |          0.1477 |          svm | BoxConstraint:                  0.35465 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     16.266 |
|   77 |       6 | Accept |    0.14811 |            149.74 |         0.14353 |          0.1474 |          svm | BoxConstraint:                  0.13086 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     5.7377 |
|   78 |       6 | Accept |    0.15523 |            165.32 |         0.14353 |         0.14769 |          svm | BoxConstraint:                  0.18542 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     15.367 |
|   79 |       6 | Accept |    0.15441 |            160.44 |         0.14353 |         0.14811 |          svm | BoxConstraint:                 0.025242 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     5.8741 |
|   80 |       6 | Accept |    0.15792 |            169.29 |         0.14353 |         0.14816 |          svm | BoxConstraint:                 0.013558 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     5.9968 |
|=======================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                                         |
|=======================================================================================================================================================|
|   81 |       6 | Accept |    0.14914 |            151.25 |         0.14353 |         0.14816 |          svm | BoxConstraint:                   1.0545 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     15.459 |
|   82 |       6 | Accept |    0.18184 |             48.24 |         0.14353 |         0.14816 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |                 |              | NumLearningCycles:                  205 |
|      |         |        |            |                   |                 |                 |              | MinLeafSize:                       4097 |
|   83 |       6 | Accept |    0.14507 |            164.62 |         0.14353 |         0.14715 |          svm | BoxConstraint:                  0.29922 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     4.0302 |
|   84 |       6 | Accept |    0.15297 |            159.83 |         0.14353 |         0.14669 |          svm | BoxConstraint:                  0.42207 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     17.016 |
|   85 |       6 | Accept |    0.14538 |            164.56 |         0.14353 |         0.14584 |          svm | BoxConstraint:                  0.30157 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.9176 |
|   86 |       6 | Accept |    0.15171 |            146.64 |         0.14353 |         0.14681 |          svm | BoxConstraint:                 0.011941 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.7945 |
|   87 |       6 | Accept |    0.15038 |            159.26 |         0.14353 |         0.14624 |          svm | BoxConstraint:                   0.8291 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     17.344 |
|   88 |       6 | Accept |    0.14811 |            145.43 |         0.14353 |         0.14638 |          svm | BoxConstraint:                   2.3513 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     17.792 |
|   89 |       6 | Accept |    0.14957 |            145.61 |         0.14353 |         0.14614 |          svm | BoxConstraint:                   1.2721 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     17.942 |
|   90 |       6 | Accept |    0.43333 |              3817 |         0.14353 |         0.14739 |          svm | BoxConstraint:                  0.46095 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.11266 |
|   91 |       6 | Accept |    0.14777 |             148.7 |         0.14353 |         0.14651 |          svm | BoxConstraint:                   3.0283 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     18.144 |
|   92 |       6 | Accept |    0.37513 |            2462.6 |         0.14353 |         0.14633 |          svm | BoxConstraint:                  0.15218 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.15693 |
|   93 |       6 | Accept |    0.37822 |              2568 |         0.14353 |         0.14674 |          svm | BoxConstraint:                 0.085986 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.15131 |
|   94 |       6 | Accept |    0.31891 |              2559 |         0.14353 |         0.14582 |          svm | BoxConstraint:                    0.142 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.14667 |
|   95 |       6 | Accept |    0.35037 |            2484.8 |         0.14353 |         0.14719 |          svm | BoxConstraint:                   732.35 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.30475 |
|   96 |       6 | Accept |    0.20831 |            3290.1 |         0.14353 |         0.14655 |          svm | BoxConstraint:                  0.36082 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.55221 |
|   97 |       6 | Accept |    0.24678 |            4132.6 |         0.14353 |         0.14626 |          svm | BoxConstraint:                  0.24182 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                   0.032229 |
|   98 |       6 | Accept |    0.21716 |            3318.5 |         0.14353 |         0.14621 |          svm | BoxConstraint:                  0.66883 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.59271 |
|   99 |       6 | Accept |    0.14578 |            248.04 |         0.14353 |         0.14582 |          svm | BoxConstraint:                   0.4278 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.0737 |
|  100 |       6 | Accept |    0.15224 |            162.91 |         0.14353 |         0.14686 |          svm | BoxConstraint:                   1.1624 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     24.402 |
|=======================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Estimated min   | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | validation loss |              |                                         |
|=======================================================================================================================================================|
|  101 |       6 | Accept |    0.22215 |              3344 |         0.14353 |         0.14576 |          svm | BoxConstraint:                  0.17981 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.41659 |
|  102 |       6 | Accept |    0.14999 |            160.35 |         0.14353 |          0.1458 |          svm | BoxConstraint:                   1.1293 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     18.197 |
|  103 |       6 | Accept |    0.14671 |            265.17 |         0.14353 |         0.14599 |          svm | BoxConstraint:                  0.29559 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     2.7758 |
|  104 |       6 | Accept |    0.17255 |            191.76 |         0.14353 |         0.14571 |          svm | BoxConstraint:                 0.055759 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     18.892 |
|  105 |       6 | Accept |    0.15195 |            158.78 |         0.14353 |         0.14564 |          svm | BoxConstraint:                 0.012192 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.9397 |
|  106 |       6 | Accept |    0.16752 |            186.75 |         0.14353 |         0.14569 |          svm | BoxConstraint:                 0.074389 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     18.359 |
|  107 |       6 | Accept |    0.14911 |            153.52 |         0.14353 |         0.14596 |          svm | BoxConstraint:                 0.011498 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.1239 |
|  108 |       6 | Accept |    0.22432 |            3370.4 |         0.14353 |         0.14617 |          svm | BoxConstraint:                  0.26224 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.48333 |
|  109 |       6 | Accept |    0.14851 |            150.42 |         0.14353 |         0.14601 |          svm | BoxConstraint:                 0.011062 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     2.9993 |
|  110 |       6 | Accept |    0.14605 |            302.86 |         0.14353 |         0.14605 |          svm | BoxConstraint:                   1.2847 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.6109 |
|  111 |       6 | Accept |    0.14905 |            479.09 |         0.14353 |         0.14572 |          svm | BoxConstraint:                  0.37863 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     2.2975 |
|  112 |       6 | Accept |    0.17238 |            188.74 |         0.14353 |         0.14591 |          svm | BoxConstraint:                 0.011768 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     9.0656 |
|  113 |       6 | Accept |    0.14986 |            154.37 |         0.14353 |          0.1458 |          svm | BoxConstraint:                0.0091472 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.0854 |
|  114 |       6 | Accept |    0.14945 |            150.17 |         0.14353 |          0.1456 |          svm | BoxConstraint:                 0.010472 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     3.1579 |
|  115 |       6 | Accept |    0.14585 |            336.82 |         0.14353 |         0.14577 |          svm | BoxConstraint:                    2.477 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     4.0884 |
|  116 |       6 | Accept |    0.14579 |            226.31 |         0.14353 |         0.14583 |          svm | BoxConstraint:                  0.19394 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     2.6603 |
|  117 |       6 | Accept |    0.15424 |            157.96 |         0.14353 |         0.14587 |          svm | BoxConstraint:                  0.33622 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     18.198 |
|  118 |       6 | Accept |    0.14706 |            311.23 |         0.14353 |         0.14599 |          svm | BoxConstraint:                  0.27404 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     2.4769 |
|  119 |       6 | Accept |    0.15064 |            576.64 |         0.14353 |          0.1456 |          svm | BoxConstraint:                  0.20057 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                     1.9187 |
|  120 |       6 | Accept |    0.22663 |            3502.9 |         0.14353 |         0.14568 |          svm | BoxConstraint:                   26.696 |
|      |         |        |            |                   |                 |                 |              | KernelScale:                    0.83468 |

__________________________________________________________
Optimization completed.
Total iterations: 120
Total elapsed time: 11529.2732 seconds
Total time for training and validation: 61054.1309 seconds

Best observed learner is a tree model with:
	Learner:                  tree
	MinLeafSize:                79
Observed validation loss: 0.14353
Time for training and validation: 1.687 seconds

Best estimated learner (returned model) is an svm model with:
	Learner:                   svm
	BoxConstraint:         0.29922
	KernelScale:            4.0302
Estimated validation loss: 0.14568
Estimated time for training and validation: 168.4716 seconds

Documentation for fitcauto display

The Total elapsed time value shows that the Bayesian optimization took a while to run (about 3.2 hours).

The final model returned by fitcauto corresponds to the best estimated learner. Before returning the model, the function retrains it using the entire training data set (adultdata), the listed Learner (or model) type, and the displayed hyperparameter values.

Use Automated Model Selection with ASHA Optimization

When fitcauto with Bayesian optimization takes a long time to run because of the number of observations in your training set, consider using fitcauto with ASHA optimization instead. Given that adultdata contains over 10,000 observations, try using fitcauto with ASHA optimization to automatically find an appropriate classifier. When you use fitcauto with ASHA optimization, the function randomly chooses several models with different hyperparameter values and trains them on a small subset of the training data. If the cross-validation classification error (Validation Loss) of a particular model is promising, the model is promoted and trained on a larger amount of the training data. This process repeats, and successful models are trained on progressively larger amounts of data. By default, fitcauto provides a plot of the optimization and an iterative display of the optimization results. For more information on how to interpret these results, see Verbose Display.

Set the observation weights, and specify to run the ASHA optimization in parallel. Note that ASHA optimization often has more iterations than Bayesian optimization by default. If you have a time constraint, you can specify the MaxTime field of the HyperparameterOptimizationOptions structure to limit the number of seconds fitcauto runs.

ashaOptions = struct("Optimizer","asha","UseParallel",true);
[ashaMdl,ashaResults] = fitcauto(adultdata,"salary","Weights","fnlwgt", ...
   "HyperparameterOptimizationOptions",ashaOptions);
Copying objective function to workers...
Warning: Files that have already been attached are being ignored. To see which files are attached see the 'AttachedFiles' property of the parallel pool.
Done copying objective function to workers.
Learner types to explore: ensemble, nb, svm, tree
Total iterations (MaxObjectiveEvaluations): 425
Total time (MaxTime): Inf

|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|    1 |       6 | Best   |    0.24677 |           0.41223 |         0.24677 |          378 |         tree | MinLeafSize:                        293 |
|    2 |       5 | Best   |    0.21242 |           0.69352 |         0.21242 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|    3 |       5 | Accept |    0.24677 |           0.49992 |         0.21242 |          378 |         tree | MinLeafSize:                       1192 |
|    4 |       5 | Accept |    0.24677 |           0.89242 |         0.21242 |          378 |         tree | MinLeafSize:                      10141 |
|    5 |       5 | Best   |    0.18925 |           0.26458 |         0.18925 |          378 |         tree | MinLeafSize:                         16 |
|    6 |       6 | Best   |    0.18343 |           0.41943 |         0.18343 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|    7 |       6 | Accept |    0.24677 |           0.98959 |         0.18343 |          378 |          svm | BoxConstraint:                   1.3947 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     65.678 |
|    8 |       6 | Accept |    0.18372 |           0.53536 |         0.18343 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|    9 |       6 | Accept |     0.1954 |            9.5036 |         0.18343 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                        0.0034879 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   10 |       6 | Best   |     0.1792 |            0.5341 |          0.1792 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   11 |       6 | Best   |    0.17449 |             0.425 |         0.17449 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   12 |       6 | Accept |    0.24677 |            14.424 |         0.17449 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  295 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       4119 |
|   13 |       6 | Accept |    0.20024 |           0.42798 |         0.17449 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   14 |       6 | Accept |    0.20304 |             13.33 |         0.17449 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           2.8059 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   15 |       6 | Accept |    0.19474 |            14.061 |         0.17449 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           25.229 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   16 |       6 | Best   |    0.16052 |            0.3778 |         0.16052 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   17 |       6 | Accept |    0.20451 |           0.20402 |         0.16052 |          378 |         tree | MinLeafSize:                         19 |
|   18 |       6 | Accept |    0.24677 |            0.1917 |         0.16052 |          378 |         tree | MinLeafSize:                       1648 |
|   19 |       6 | Accept |      0.182 |           0.48008 |         0.16052 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   20 |       6 | Accept |    0.21976 |           0.84268 |         0.16052 |          378 |          svm | BoxConstraint:                    1.374 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     1.1138 |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|   21 |       6 | Accept |    0.16445 |           0.40264 |         0.16052 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   22 |       6 | Accept |    0.17111 |           0.39243 |         0.16052 |         6033 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   23 |       6 | Accept |    0.17395 |           0.87746 |         0.16052 |          378 |          svm | BoxConstraint:                   1.3291 |
|      |         |        |            |                   |                 |              |              | KernelScale:                       11.5 |
|   24 |       6 | Accept |    0.24677 |           0.88712 |         0.16052 |          378 |          svm | BoxConstraint:                     2.32 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     249.93 |
|   25 |       6 | Accept |    0.18863 |            8.5476 |         0.16052 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                         0.050938 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   26 |       6 | Accept |    0.39894 |            19.121 |         0.16052 |          378 |          svm | BoxConstraint:                 0.082795 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.11799 |
|   27 |       6 | Accept |    0.21388 |           0.84483 |         0.16052 |          378 |          svm | BoxConstraint:                   536.31 |
|      |         |        |            |                   |                 |              |              | KernelScale:                       2.96 |
|   28 |       6 | Accept |    0.17673 |           0.40821 |         0.16052 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   29 |       6 | Accept |    0.24677 |           0.14927 |         0.16052 |          378 |         tree | MinLeafSize:                        228 |
|   30 |       6 | Best   |    0.15579 |             2.158 |         0.15579 |         1509 |          svm | BoxConstraint:                   1.3291 |
|      |         |        |            |                   |                 |              |              | KernelScale:                       11.5 |
|   31 |       6 | Accept |    0.16985 |           0.38832 |         0.15579 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   32 |       6 | Accept |    0.24677 |           0.30699 |         0.15579 |          378 |          svm | BoxConstraint:                 0.045199 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0063948 |
|   33 |       6 | Accept |    0.20589 |           0.20066 |         0.15579 |          378 |         tree | MinLeafSize:                         22 |
|   34 |       6 | Accept |    0.24677 |           0.90412 |         0.15579 |          378 |          svm | BoxConstraint:                 0.059749 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     179.59 |
|   35 |       6 | Accept |    0.17167 |            11.632 |         0.15579 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           0.3336 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   36 |       6 | Accept |    0.22055 |             12.98 |         0.15579 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           12.182 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   37 |       6 | Accept |    0.16867 |            12.841 |         0.15579 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                            108.9 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   38 |       6 | Accept |    0.16266 |           0.86177 |         0.15579 |          378 |          svm | BoxConstraint:                   385.35 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     65.814 |
|   39 |       6 | Accept |    0.22491 |            0.8454 |         0.15579 |          378 |          svm | BoxConstraint:                   11.832 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     1.7245 |
|   40 |       6 | Accept |     0.1561 |            2.1629 |         0.15579 |         1509 |          svm | BoxConstraint:                   385.35 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     65.814 |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|   41 |       6 | Accept |    0.22074 |           0.28317 |         0.15579 |          378 |         tree | MinLeafSize:                         22 |
|   42 |       6 | Accept |    0.24677 |            12.744 |         0.15579 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  297 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       7579 |
|   43 |       6 | Accept |    0.24677 |            14.118 |         0.15579 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  282 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       1543 |
|   44 |       6 | Accept |    0.16998 |           0.33582 |         0.15579 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   45 |       6 | Accept |    0.31094 |            37.128 |         0.15579 |          378 |          svm | BoxConstraint:                   70.951 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.033775 |
|   46 |       6 | Accept |    0.24677 |           0.34815 |         0.15579 |          378 |          svm | BoxConstraint:                   623.48 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0010192 |
|   47 |       6 | Accept |    0.17045 |            10.999 |         0.15579 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           1.8198 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   48 |       6 | Accept |    0.15807 |            13.413 |         0.15579 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  220 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         30 |
|   49 |       6 | Accept |    0.17396 |            29.339 |         0.15579 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           0.3336 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|   50 |       6 | Accept |      0.676 |            23.204 |         0.15579 |          378 |          svm | BoxConstraint:                   39.423 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.10596 |
|   51 |       6 | Accept |    0.20495 |           0.22109 |         0.15579 |          378 |         tree | MinLeafSize:                         21 |
|   52 |       6 | Accept |    0.18563 |           0.20139 |         0.15579 |          378 |         tree | MinLeafSize:                         42 |
|   53 |       6 | Accept |    0.24677 |           0.92178 |         0.15579 |          378 |          svm | BoxConstraint:                  0.05188 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     44.452 |
|   54 |       6 | Accept |    0.18321 |           0.38725 |         0.15579 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   55 |       6 | Accept |    0.17723 |            0.4152 |         0.15579 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   56 |       6 | Accept |    0.48003 |             26.08 |         0.15579 |          378 |          svm | BoxConstraint:                   213.45 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.092072 |
|   57 |       6 | Best   |    0.15393 |            17.242 |         0.15393 |         1509 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  220 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         30 |
|   58 |       6 | Accept |    0.17298 |           0.89838 |         0.15393 |          378 |          svm | BoxConstraint:                   962.26 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     207.85 |
|   59 |       6 | Best   |      0.153 |            12.348 |           0.153 |         6033 |          svm | BoxConstraint:                   1.3291 |
|      |         |        |            |                   |                 |              |              | KernelScale:                       11.5 |
|   60 |       6 | Accept |    0.18192 |           0.89064 |           0.153 |          378 |          svm | BoxConstraint:                0.0055891 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.90898 |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|   61 |       6 | Accept |    0.18271 |            32.278 |           0.153 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                            108.9 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   62 |       6 | Accept |    0.24677 |           0.14112 |           0.153 |          378 |         tree | MinLeafSize:                       3783 |
|   63 |       6 | Accept |    0.26476 |            34.581 |           0.153 |          378 |          svm | BoxConstraint:                   74.583 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.059475 |
|   64 |       6 | Accept |    0.20762 |            8.8087 |           0.153 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.10297 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   65 |       6 | Accept |    0.24677 |            1.0071 |           0.153 |          378 |          svm | BoxConstraint:                  0.09297 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     536.28 |
|   66 |       6 | Accept |    0.24194 |             1.071 |           0.153 |          378 |          svm | BoxConstraint:                  0.67333 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     35.903 |
|   67 |       6 | Accept |    0.20385 |            18.897 |           0.153 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  252 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         77 |
|   68 |       6 | Accept |    0.15503 |            2.2706 |           0.153 |         1509 |          svm | BoxConstraint:                   962.26 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     207.85 |
|   69 |       6 | Accept |    0.24677 |            13.432 |           0.153 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  245 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       3552 |
|   70 |       6 | Accept |    0.19077 |            8.1226 |           0.153 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                         0.027553 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   71 |       6 | Accept |    0.16757 |           0.36842 |           0.153 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   72 |       6 | Accept |    0.18779 |           0.35808 |           0.153 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   73 |       6 | Accept |    0.17793 |            12.899 |           0.153 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           21.131 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   74 |       6 | Accept |    0.16496 |            27.002 |           0.153 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           1.8198 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   75 |       6 | Accept |    0.24677 |           0.21047 |           0.153 |          378 |         tree | MinLeafSize:                      12519 |
|   76 |       6 | Accept |    0.16784 |           0.90871 |           0.153 |          378 |          svm | BoxConstraint:                   763.29 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     144.74 |
|   77 |       6 | Accept |    0.24677 |             8.515 |           0.153 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  210 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        604 |
|   78 |       6 | Accept |    0.15451 |            2.0399 |           0.153 |         1509 |          svm | BoxConstraint:                   763.29 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     144.74 |
|   79 |       6 | Accept |    0.61916 |            18.894 |           0.153 |          378 |          svm | BoxConstraint:                 0.043267 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.14745 |
|   80 |       6 | Accept |    0.17582 |            0.3839 |           0.153 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|   81 |       6 | Accept |    0.24677 |           0.93542 |           0.153 |          378 |          svm | BoxConstraint:                 0.024337 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     76.684 |
|   82 |       6 | Accept |    0.17903 |           0.39209 |           0.153 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   83 |       6 | Accept |    0.26744 |            38.332 |           0.153 |          378 |          svm | BoxConstraint:                   171.89 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0079675 |
|   84 |       6 | Accept |    0.20799 |           0.27606 |           0.153 |          378 |         tree | MinLeafSize:                         20 |
|   85 |       6 | Accept |    0.24677 |            12.493 |           0.153 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  225 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       8606 |
|   86 |       6 | Accept |    0.24677 |            1.0149 |           0.153 |          378 |          svm | BoxConstraint:                 0.038237 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     63.926 |
|   87 |       6 | Accept |    0.24677 |            11.714 |           0.153 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  203 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        332 |
|   88 |       6 | Best   |    0.15053 |            28.585 |         0.15053 |         6033 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  220 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         30 |
|   89 |       6 | Accept |    0.24677 |           0.19642 |         0.15053 |          378 |         tree | MinLeafSize:                        683 |
|   90 |       6 | Accept |    0.24677 |            14.949 |         0.15053 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  241 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        211 |
|   91 |       6 | Accept |    0.24677 |           0.97254 |         0.15053 |          378 |          svm | BoxConstraint:                0.0035269 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     117.31 |
|   92 |       6 | Best   |    0.15028 |            13.847 |         0.15028 |         6033 |          svm | BoxConstraint:                   763.29 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     144.74 |
|   93 |       6 | Accept |    0.18829 |            2.5565 |         0.15028 |         1509 |          svm | BoxConstraint:                0.0055891 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.90898 |
|   94 |       6 | Accept |    0.24677 |            12.589 |         0.15028 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  205 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       4488 |
|   95 |       6 | Accept |    0.19257 |             10.01 |         0.15028 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.92919 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|   96 |       6 | Accept |     0.1724 |            0.3932 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   97 |       6 | Accept |    0.24677 |            13.185 |         0.15028 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  251 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       5425 |
|   98 |       6 | Accept |    0.18466 |           0.41239 |         0.15028 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|   99 |       6 | Accept |    0.24677 |           0.15762 |         0.15028 |          378 |         tree | MinLeafSize:                        632 |
|  100 |       6 | Accept |    0.18538 |           0.40593 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  101 |       6 | Accept |    0.24735 |            0.1677 |         0.15028 |          378 |         tree | MinLeafSize:                        159 |
|  102 |       6 | Accept |    0.46239 |            19.011 |         0.15028 |          378 |          svm | BoxConstraint:                   622.29 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.13209 |
|  103 |       6 | Accept |    0.17613 |           0.43685 |         0.15028 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  104 |       6 | Accept |    0.45428 |            21.612 |         0.15028 |          378 |          svm | BoxConstraint:                   243.32 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.11649 |
|  105 |       6 | Accept |    0.17695 |            33.598 |         0.15028 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           21.131 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  106 |       6 | Accept |    0.20001 |           0.89752 |         0.15028 |          378 |          svm | BoxConstraint:                  0.13074 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     4.2955 |
|  107 |       6 | Accept |    0.26791 |            34.811 |         0.15028 |          378 |          svm | BoxConstraint:                  0.15408 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0081934 |
|  108 |       6 | Accept |    0.24677 |            12.075 |         0.15028 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  277 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       4646 |
|  109 |       6 | Accept |    0.17992 |           0.87394 |         0.15028 |          378 |          svm | BoxConstraint:                   97.764 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     20.653 |
|  110 |       6 | Accept |    0.17688 |           0.38115 |         0.15028 |         1509 |         tree | MinLeafSize:                         42 |
|  111 |       6 | Accept |     0.1517 |            13.832 |         0.15028 |         6033 |          svm | BoxConstraint:                   962.26 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     207.85 |
|  112 |       6 | Accept |    0.18863 |            0.4304 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  113 |       6 | Accept |    0.18648 |             0.453 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  114 |       6 | Accept |    0.15876 |            19.737 |         0.15028 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  271 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         14 |
|  115 |       6 | Accept |    0.24184 |           0.90896 |         0.15028 |          378 |          svm | BoxConstraint:                  0.26232 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     13.726 |
|  116 |       6 | Accept |    0.16007 |            16.574 |         0.15028 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  226 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          1 |
|  117 |       6 | Accept |    0.18577 |           0.43107 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  118 |       6 | Accept |    0.24677 |            0.2089 |         0.15028 |          378 |         tree | MinLeafSize:                        112 |
|  119 |       6 | Accept |    0.16662 |            23.782 |         0.15028 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  291 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         20 |
|  120 |       6 | Accept |    0.18481 |            0.3758 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  121 |       6 | Accept |    0.21737 |            13.332 |         0.15028 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           50.518 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  122 |       6 | Accept |    0.24677 |           0.92059 |         0.15028 |          378 |          svm | BoxConstraint:                0.0042093 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     11.241 |
|  123 |       6 | Accept |    0.15316 |            17.743 |         0.15028 |         1509 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  226 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          1 |
|  124 |       6 | Accept |    0.24677 |           0.16148 |         0.15028 |          378 |         tree | MinLeafSize:                        302 |
|  125 |       6 | Accept |    0.15316 |             21.59 |         0.15028 |         1509 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  271 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         14 |
|  126 |       6 | Accept |    0.23141 |            37.152 |         0.15028 |          378 |          svm | BoxConstraint:                   21.777 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.029276 |
|  127 |       6 | Accept |    0.23304 |            16.815 |         0.15028 |          378 |          svm | BoxConstraint:                   173.49 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.21811 |
|  128 |       6 | Accept |    0.24677 |             0.142 |         0.15028 |          378 |         tree | MinLeafSize:                       2502 |
|  129 |       6 | Accept |    0.15706 |            2.1789 |         0.15028 |         1509 |          svm | BoxConstraint:                   97.764 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     20.653 |
|  130 |       6 | Accept |     0.2435 |            12.745 |         0.15028 |          378 |          svm | BoxConstraint:                 0.053634 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.31732 |
|  131 |       6 | Accept |    0.21974 |            13.616 |         0.15028 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           3.3625 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  132 |       6 | Accept |    0.19986 |           0.42586 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  133 |       6 | Accept |    0.24677 |           0.33075 |         0.15028 |          378 |          svm | BoxConstraint:                   466.95 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0044945 |
|  134 |       6 | Accept |    0.16879 |           0.40206 |         0.15028 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  135 |       6 | Accept |    0.24677 |            0.1607 |         0.15028 |          378 |         tree | MinLeafSize:                        724 |
|  136 |       6 | Accept |    0.15942 |            27.353 |         0.15028 |         1509 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  291 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         20 |
|  137 |       6 | Accept |     0.1828 |           0.43307 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  138 |       6 | Accept |    0.24677 |            11.762 |         0.15028 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  212 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       2353 |
|  139 |       6 | Accept |    0.24677 |            14.322 |         0.15028 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  255 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       2166 |
|  140 |       6 | Accept |    0.19199 |           0.41275 |         0.15028 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  141 |       6 | Accept |    0.18811 |           0.37075 |         0.15028 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  142 |       6 | Accept |    0.24677 |            9.4582 |         0.15028 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  203 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       3125 |
|  143 |       6 | Accept |    0.24189 |           0.97907 |         0.15028 |          378 |          svm | BoxConstraint:                   2.2181 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     60.618 |
|  144 |       6 | Accept |    0.22596 |           0.81415 |         0.15028 |          378 |          svm | BoxConstraint:                   8.9199 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.93704 |
|  145 |       6 | Accept |    0.17702 |           0.43906 |         0.15028 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  146 |       6 | Accept |    0.24749 |            33.011 |         0.15028 |          378 |          svm | BoxConstraint:                0.0023424 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.02797 |
|  147 |       6 | Accept |    0.24677 |            0.3189 |         0.15028 |          378 |          svm | BoxConstraint:                  0.59521 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0066679 |
|  148 |       6 | Accept |    0.24591 |            13.225 |         0.15028 |          378 |          svm | BoxConstraint:                   521.24 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.28175 |
|  149 |       6 | Accept |    0.22288 |           0.91802 |         0.15028 |          378 |          svm | BoxConstraint:                 0.021474 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     4.5183 |
|  150 |       6 | Accept |    0.18334 |           0.40628 |         0.15028 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  151 |       6 | Accept |    0.18334 |            13.603 |         0.15028 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           693.96 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  152 |       6 | Best   |    0.14984 |              34.6 |         0.14984 |         6033 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  271 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         14 |
|  153 |       6 | Accept |    0.24677 |           0.93891 |         0.14984 |          378 |          svm | BoxConstraint:                 0.011464 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     887.95 |
|  154 |       6 | Accept |    0.24677 |            15.021 |         0.14984 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  271 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       3858 |
|  155 |       6 | Accept |    0.24677 |            0.2819 |         0.14984 |          378 |          svm | BoxConstraint:                    0.017 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0046035 |
|  156 |       6 | Accept |    0.16761 |            10.948 |         0.14984 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                            1.613 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  157 |       6 | Accept |    0.24677 |           0.30553 |         0.14984 |          378 |          svm | BoxConstraint:                   4.1988 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0011787 |
|  158 |       6 | Accept |    0.17316 |            22.007 |         0.14984 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  276 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         16 |
|  159 |       6 | Accept |    0.20481 |           0.86066 |         0.14984 |          378 |          svm | BoxConstraint:                  0.01545 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     1.0258 |
|  160 |       6 | Accept |    0.15008 |             29.64 |         0.14984 |         6033 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  226 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          1 |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  161 |       6 | Accept |    0.18437 |            0.9387 |         0.14984 |          378 |          svm | BoxConstraint:                0.0087948 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     1.1494 |
|  162 |       6 | Accept |    0.20904 |            13.569 |         0.14984 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           91.024 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  163 |       6 | Accept |    0.23451 |            6.5741 |         0.14984 |          378 |          svm | BoxConstraint:                  0.11495 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.34761 |
|  164 |       6 | Best   |     0.1493 |             147.7 |          0.1493 |        24130 |          svm | BoxConstraint:                   763.29 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     144.74 |
|  165 |       6 | Accept |    0.27656 |            40.771 |          0.1493 |          378 |          svm | BoxConstraint:                   1.0475 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.10172 |
|  166 |       6 | Accept |    0.16234 |            26.581 |          0.1493 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                            1.613 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  167 |       6 | Accept |    0.19497 |           0.19102 |          0.1493 |          378 |         tree | MinLeafSize:                         57 |
|  168 |       6 | Accept |    0.19417 |            34.123 |          0.1493 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           693.96 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  169 |       6 | Accept |    0.25929 |             19.29 |          0.1493 |          378 |          svm | BoxConstraint:                   19.843 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.14367 |
|  170 |       6 | Accept |      0.162 |           0.84268 |          0.1493 |          378 |          svm | BoxConstraint:                   13.978 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     30.295 |
|  171 |       6 | Accept |    0.16933 |           0.83709 |          0.1493 |          378 |          svm | BoxConstraint:                   3.9363 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     17.816 |
|  172 |       6 | Accept |    0.15652 |            2.1022 |          0.1493 |         1509 |          svm | BoxConstraint:                   13.978 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     30.295 |
|  173 |       6 | Accept |    0.22197 |            12.696 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           102.13 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  174 |       6 | Accept |    0.24677 |           0.18644 |          0.1493 |          378 |         tree | MinLeafSize:                        864 |
|  175 |       6 | Accept |    0.24677 |           0.15003 |          0.1493 |          378 |         tree | MinLeafSize:                       1322 |
|  176 |       6 | Accept |    0.18647 |            8.0636 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                         0.013221 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  177 |       6 | Accept |    0.15981 |            2.1353 |          0.1493 |         1509 |          svm | BoxConstraint:                   3.9363 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     17.816 |
|  178 |       6 | Accept |    0.18089 |           0.43329 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  179 |       6 | Accept |    0.17375 |           0.35426 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  180 |       6 | Accept |    0.16252 |           0.38204 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  181 |       6 | Accept |    0.38861 |            33.253 |          0.1493 |          378 |          svm | BoxConstraint:                   6.5307 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.072466 |
|  182 |       6 | Accept |    0.17638 |           0.43308 |          0.1493 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  183 |       6 | Accept |    0.15072 |            13.541 |          0.1493 |         6033 |          svm | BoxConstraint:                   385.35 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     65.814 |
|  184 |       6 | Accept |    0.15577 |            25.223 |          0.1493 |         1509 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  276 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         16 |
|  185 |       6 | Accept |    0.24677 |            14.695 |          0.1493 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  270 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       1304 |
|  186 |       6 | Accept |    0.19237 |           0.42507 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  187 |       6 | Accept |    0.19325 |           0.38775 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  188 |       6 | Accept |    0.19327 |            9.6642 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.58748 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  189 |       6 | Accept |    0.18255 |             13.18 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           54.029 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  190 |       6 | Accept |    0.16223 |           0.33709 |          0.1493 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  191 |       6 | Accept |    0.19958 |           0.17893 |          0.1493 |          378 |         tree | MinLeafSize:                         24 |
|  192 |       6 | Accept |    0.24677 |            0.9243 |          0.1493 |          378 |          svm | BoxConstraint:                 0.028158 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     21.174 |
|  193 |       6 | Accept |    0.18379 |            9.5712 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.69661 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  194 |       6 | Accept |    0.17841 |           0.41127 |          0.1493 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  195 |       6 | Accept |    0.49303 |            36.252 |          0.1493 |          378 |          svm | BoxConstraint:                 0.035455 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.016789 |
|  196 |       6 | Accept |    0.24677 |           0.12499 |          0.1493 |          378 |         tree | MinLeafSize:                        233 |
|  197 |       6 | Accept |    0.24677 |            11.467 |          0.1493 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  209 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        398 |
|  198 |       6 | Accept |    0.17691 |           0.86872 |          0.1493 |          378 |          svm | BoxConstraint:                   596.59 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     21.708 |
|  199 |       6 | Accept |    0.15712 |            2.9033 |          0.1493 |         1509 |          svm | BoxConstraint:                   596.59 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     21.708 |
|  200 |       6 | Accept |    0.17137 |            0.3622 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  201 |       6 | Accept |    0.24677 |            16.885 |          0.1493 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  299 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        216 |
|  202 |       6 | Accept |    0.24677 |            11.468 |          0.1493 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  213 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       1448 |
|  203 |       6 | Accept |    0.16841 |            1.0814 |          0.1493 |          378 |          svm | BoxConstraint:                   484.15 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     69.555 |
|  204 |       5 | Accept |    0.17402 |            13.714 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           3.8082 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  205 |       5 | Accept |    0.15642 |            2.2741 |          0.1493 |         1509 |          svm | BoxConstraint:                   484.15 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     69.555 |
|  206 |       6 | Accept |    0.24677 |           0.30604 |          0.1493 |          378 |          svm | BoxConstraint:                 0.022208 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0071215 |
|  207 |       6 | Accept |     0.1817 |           0.42424 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  208 |       6 | Accept |    0.18218 |           0.89991 |          0.1493 |          378 |          svm | BoxConstraint:                   162.42 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     10.207 |
|  209 |       6 | Accept |    0.24677 |           0.92031 |          0.1493 |          378 |          svm | BoxConstraint:                   28.319 |
|      |         |        |            |                   |                 |              |              | KernelScale:                      317.4 |
|  210 |       6 | Accept |    0.17479 |           0.38641 |          0.1493 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  211 |       6 | Accept |      0.225 |           0.86838 |          0.1493 |          378 |          svm | BoxConstraint:                   810.07 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     3.0301 |
|  212 |       6 | Accept |    0.17704 |           0.40973 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  213 |       6 | Accept |    0.19395 |            11.826 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.27286 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  214 |       6 | Accept |    0.17788 |              21.7 |          0.1493 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  283 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         47 |
|  215 |       6 | Accept |    0.14987 |            14.362 |          0.1493 |         6033 |          svm | BoxConstraint:                   484.15 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     69.555 |
|  216 |       6 | Accept |    0.21628 |            13.595 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                            9.774 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  217 |       5 | Accept |    0.17398 |           0.77326 |          0.1493 |          378 |          svm | BoxConstraint:                   112.02 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     13.915 |
|  218 |       5 | Accept |    0.24677 |            0.1336 |          0.1493 |          378 |         tree | MinLeafSize:                        653 |
|  219 |       6 | Accept |    0.15218 |            40.273 |          0.1493 |         6033 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  276 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         16 |
|  220 |       6 | Accept |    0.16554 |           0.37094 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  221 |       6 | Accept |    0.15334 |            2.3799 |          0.1493 |         1509 |          svm | BoxConstraint:                   112.02 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     13.915 |
|  222 |       6 | Accept |    0.19759 |            8.0999 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                        0.0023918 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  223 |       6 | Accept |    0.23132 |           0.77865 |          0.1493 |          378 |          svm | BoxConstraint:                   10.854 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.90505 |
|  224 |       6 | Accept |    0.18694 |            8.0651 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.16744 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  225 |       6 | Accept |    0.17874 |           0.37035 |          0.1493 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  226 |       6 | Accept |    0.24677 |           0.14836 |          0.1493 |          378 |         tree | MinLeafSize:                      12624 |
|  227 |       6 | Accept |    0.24677 |            11.097 |          0.1493 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  257 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        228 |
|  228 |       6 | Accept |    0.18473 |           0.45468 |          0.1493 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  229 |       6 | Accept |    0.16683 |            16.222 |          0.1493 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  274 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         52 |
|  230 |       6 | Accept |    0.24677 |           0.89615 |          0.1493 |          378 |          svm | BoxConstraint:                0.0023661 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     236.67 |
|  231 |       6 | Accept |    0.21503 |           0.27122 |          0.1493 |          378 |         tree | MinLeafSize:                          3 |
|  232 |       6 | Accept |    0.16733 |            18.627 |          0.1493 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  253 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         15 |
|  233 |       6 | Accept |    0.16675 |            29.958 |          0.1493 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           3.8082 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  234 |       6 | Accept |    0.16016 |            17.053 |          0.1493 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  261 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          7 |
|  235 |       6 | Accept |    0.16943 |            9.9624 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           1.4142 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  236 |       6 | Accept |    0.24677 |           0.18215 |          0.1493 |          378 |         tree | MinLeafSize:                       4920 |
|  237 |       6 | Accept |    0.24677 |            11.257 |          0.1493 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  268 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       2856 |
|  238 |       6 | Accept |    0.21822 |            0.1878 |          0.1493 |          378 |         tree | MinLeafSize:                         62 |
|  239 |       6 | Accept |    0.15342 |             21.65 |          0.1493 |         1509 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  274 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         52 |
|  240 |       6 | Accept |    0.19453 |            8.2977 |          0.1493 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                        0.0053849 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  241 |       6 | Best   |    0.14836 |             96.23 |         0.14836 |        24130 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  271 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         14 |
|  242 |       6 | Accept |    0.24677 |           0.33161 |         0.14836 |          378 |          svm | BoxConstraint:                   346.19 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0031468 |
|  243 |       6 | Accept |    0.15424 |            20.047 |         0.14836 |         1509 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  261 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          7 |
|  244 |       6 | Accept |    0.24677 |           0.18571 |         0.14836 |          378 |         tree | MinLeafSize:                       2558 |
|  245 |       6 | Accept |    0.24677 |           0.35537 |         0.14836 |          378 |          svm | BoxConstraint:                0.0016007 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0011917 |
|  246 |       6 | Accept |    0.17468 |            11.674 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.34418 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  247 |       6 | Accept |    0.25459 |            33.254 |         0.14836 |          378 |          svm | BoxConstraint:                   492.71 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.062372 |
|  248 |       6 | Accept |    0.24677 |            0.1379 |         0.14836 |          378 |         tree | MinLeafSize:                        374 |
|  249 |       6 | Accept |    0.15195 |            23.675 |         0.14836 |         1509 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  253 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         15 |
|  250 |       6 | Accept |    0.18005 |            10.469 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.18024 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  251 |       6 | Accept |    0.18113 |            10.289 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.15646 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  252 |       6 | Accept |    0.14951 |            23.305 |         0.14836 |         6033 |          svm | BoxConstraint:                   112.02 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     13.915 |
|  253 |       6 | Accept |    0.24661 |           0.17732 |         0.14836 |          378 |         tree | MinLeafSize:                        115 |
|  254 |       6 | Accept |    0.24677 |            14.536 |         0.14836 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  275 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       7160 |
|  255 |       6 | Accept |    0.24677 |           0.14128 |         0.14836 |          378 |         tree | MinLeafSize:                        342 |
|  256 |       6 | Accept |    0.17276 |           0.37834 |         0.14836 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  257 |       6 | Accept |    0.23016 |            12.927 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           35.855 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  258 |       6 | Accept |    0.24677 |           0.15532 |         0.14836 |          378 |         tree | MinLeafSize:                       7345 |
|  259 |       6 | Accept |    0.24677 |           0.95577 |         0.14836 |          378 |          svm | BoxConstraint:                 0.020628 |
|      |         |        |            |                   |                 |              |              | KernelScale:                      164.3 |
|  260 |       6 | Accept |     0.2025 |           0.18858 |         0.14836 |          378 |         tree | MinLeafSize:                         36 |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  261 |       6 | Accept |    0.16554 |            24.966 |         0.14836 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           1.4142 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  262 |       6 | Accept |    0.18069 |            10.339 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.11577 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  263 |       6 | Accept |      0.162 |            15.993 |         0.14836 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  245 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          2 |
|  264 |       6 | Accept |    0.24677 |            11.091 |         0.14836 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  266 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                        295 |
|  265 |       6 | Accept |    0.20847 |           0.27559 |         0.14836 |          378 |         tree | MinLeafSize:                          2 |
|  266 |       6 | Accept |     0.1769 |            30.334 |         0.14836 |         1509 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                          0.34418 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  267 |       6 | Accept |    0.24677 |           0.92783 |         0.14836 |          378 |          svm | BoxConstraint:                   1.8318 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     640.96 |
|  268 |       6 | Accept |    0.18559 |           0.38512 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  269 |       6 | Accept |    0.17281 |            17.472 |         0.14836 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  251 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         36 |
|  270 |       6 | Accept |    0.24677 |           0.13322 |         0.14836 |          378 |         tree | MinLeafSize:                       1065 |
|  271 |       6 | Accept |    0.23467 |            33.997 |         0.14836 |          378 |          svm | BoxConstraint:                   40.942 |
|      |         |        |            |                   |                 |              |              | KernelScale:                   0.060508 |
|  272 |       6 | Accept |    0.16424 |           0.40692 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  273 |       6 | Accept |    0.16911 |            25.586 |         0.14836 |         1509 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  283 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         47 |
|  274 |       6 | Accept |    0.17906 |           0.39597 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  275 |       6 | Accept |    0.18946 |            8.6561 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                        0.0097303 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|  276 |       6 | Accept |    0.17244 |           0.39582 |         0.14836 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  277 |       6 | Accept |    0.19106 |           0.17636 |         0.14836 |          378 |         tree | MinLeafSize:                         58 |
|  278 |       6 | Accept |    0.15421 |            18.602 |         0.14836 |         1509 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  245 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          2 |
|  279 |       6 | Accept |     0.1515 |            34.698 |         0.14836 |         6033 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  253 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         15 |
|  280 |       6 | Accept |    0.18897 |            13.381 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           697.61 |
|      |         |        |            |                   |                 |              |              | Standardize:                       true |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  281 |       6 | Accept |    0.16858 |           0.37927 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  282 |       6 | Accept |    0.22522 |            13.462 |         0.14836 |          378 |          svm | BoxConstraint:                  0.07043 |
|      |         |        |            |                   |                 |              |              | KernelScale:                    0.25792 |
|  283 |       6 | Accept |     0.2417 |            1.0039 |         0.14836 |          378 |          svm | BoxConstraint:                0.0013375 |
|      |         |        |            |                   |                 |              |              | KernelScale:                      3.417 |
|  284 |       6 | Accept |    0.18405 |            0.3932 |         0.14836 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  285 |       6 | Accept |    0.24677 |           0.33159 |         0.14836 |          378 |          svm | BoxConstraint:                 0.038245 |
|      |         |        |            |                   |                 |              |              | KernelScale:                  0.0021108 |
|  286 |       6 | Accept |    0.19548 |           0.43701 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  287 |       6 | Accept |    0.24677 |            1.0456 |         0.14836 |          378 |          svm | BoxConstraint:                0.0050787 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     664.76 |
|  288 |       6 | Accept |    0.15713 |           0.41854 |         0.14836 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  289 |       6 | Accept |    0.24677 |            12.752 |         0.14836 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  232 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       3957 |
|  290 |       6 | Accept |    0.17541 |           0.38528 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  291 |       6 | Accept |    0.16182 |            23.699 |         0.14836 |         1509 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  251 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                         36 |
|  292 |       6 | Accept |    0.20209 |            8.1049 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                         0.036333 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  293 |       6 | Accept |    0.24677 |            9.4156 |         0.14836 |          378 |     ensemble | Method:                      LogitBoost |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  208 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                       2039 |
|  294 |       6 | Accept |     0.1795 |           0.41664 |         0.14836 |         1509 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  295 |       5 | Accept |    0.20244 |            8.2445 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                         0.003178 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  296 |       5 | Accept |    0.22113 |           0.91625 |         0.14836 |          378 |          svm | BoxConstraint:                   83.258 |
|      |         |        |            |                   |                 |              |              | KernelScale:                     180.57 |
|  297 |       6 | Accept |    0.24677 |           0.15149 |         0.14836 |          378 |         tree | MinLeafSize:                        330 |
|  298 |       6 | Accept |    0.20546 |           0.23764 |         0.14836 |          378 |         tree | MinLeafSize:                          3 |
|  299 |       6 | Accept |    0.17061 |           0.40961 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|  300 |       6 | Accept |    0.19405 |           0.41665 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |              |              | Standardize:                          - |
|====================================================================================================================================================|
| Iter | Active  | Eval   | Validation | Time for training | Observed min    | Training set | Learner      | Hyperparameter:                 Value   |
|      | workers | result | loss       | & validation (sec)| validation loss | size         |              |                                         |
|====================================================================================================================================================|
|  301 |       6 | Accept |    0.16126 |            17.363 |         0.14836 |          378 |     ensemble | Method:                             Bag |
|      |         |        |            |                   |                 |              |              | NumLearningCycles:                  216 |
|      |         |        |            |                   |                 |              |              | MinLeafSize:                          2 |
|  302 |       6 | Accept |    0.24677 |            0.1915 |         0.14836 |          378 |         tree | MinLeafSize:                        572 |
|  303 |       6 | Accept |    0.20638 |            12.701 |         0.14836 |          378 |           nb | DistributionNames:               kernel |
|      |         |        |            |                   |                 |              |              | Width:                           4.8773 |
|      |         |        |            |                   |                 |              |              | Standardize:                      false |
|  304 |       6 | Accept |    0.15775 |           0.43674 |         0.14836 |          378 |           nb | DistributionNames:               normal |
|      |         |        |            |                   |                 |              |              | Width:                              NaN |
|      |         |        |            |                   |                 |          ...

__________________________________________________________
Optimization completed.
Total iterations: 425
Total elapsed time: 693.4747 seconds
Total time for training and validation: 3784.6551 seconds

Best observed learner is an ensemble model with:
	Learner:              ensemble
	Method:             LogitBoost
	NumLearningCycles:         271
	MinLeafSize:                14
Observed validation loss: 0.14836
Time for training and validation: 96.2296 seconds

Documentation for fitcauto display

The Total elapsed time value shows that the ASHA optimization took less time to run than the Bayesian optimization (about 0.2 hours).

The final model returned by fitcauto corresponds to the best observed learner. Before returning the model, the function retrains it using the entire training data set (adultdata), the listed Learner (or model) type, and the displayed hyperparameter values.

Evaluate Test Set Performance

Evaluate the performance of the returned bayesianMdl and ashaMdl models on the test set adulttest by using confusion matrices and receiver operating characteristic (ROC) curves.

For each model, find the predicted labels and score values for the test set.

[bayesianLabels,bayesianScores] = predict(bayesianMdl,adulttest);
[ashaLabels,ashaScores] = predict(ashaMdl,adulttest);

Create confusion matrices from the test set results. The diagonal elements indicate the number of correctly classified instances of a given class. The off-diagonal elements are instances of misclassified observations. Use a 1-by-2 tiled layout to compare the results.

tiledlayout(1,2)

nexttile
confusionchart(adulttest.salary,bayesianLabels)
title("Bayesian Optimization")

nexttile
confusionchart(adulttest.salary,ashaLabels)
title("ASHA Optimization")

Compute the test set classification accuracy for each model, where the accuracy is the percentage of correctly classified test set observations.

bayesianAccuracy = (1-loss(bayesianMdl,adulttest,"salary"))*100
bayesianAccuracy = 85.3243
ashaAccuracy = (1-loss(ashaMdl,adulttest,"salary"))*100
ashaAccuracy = 85.4724

Based on the confusion matrices and the accuracy values, both models perform well.

For each model, plot the ROC curve and compute the area under the ROC curve (AUC). The ROC curve shows the true positive rate versus the false positive rate for different thresholds of classification scores. For a perfect classifier, whose true positive rate is always 1 regardless of the threshold, AUC = 1. For a binary classifier that randomly assigns observations to classes, AUC = 0.5. A large AUC value (close to 1) indicates good classifier performance.

For each model, compute the metrics for the ROC curve and find the AUC value by creating a rocmetrics object.

bayesianROC = rocmetrics(adulttest.salary,bayesianScores,bayesianMdl.ClassNames);
ashaROC = rocmetrics(adulttest.salary,ashaScores,ashaMdl.ClassNames);

Plot the ROC curves for the label <=50K by using the plot function of rocmetrics.

figure
[r1,g1] = plot(bayesianROC,"ClassNames","<=50K");
hold on
[r2,g2] = plot(ashaROC,"ClassNames","<=50K");
r1.DisplayName = replace(r1.DisplayName,"<=50K","Bayesian Optimization");
r2.DisplayName = replace(r2.DisplayName,"<=50K","ASHA Optimization");
g1(1).DisplayName = "Bayesian Optimization Model Operating Point";
g2(1).DisplayName = "ASHA Optimization Model Operating Point";
title("ROC Curves for Class <=50K")
hold off

Based on the AUC values, both classifiers perform well on the test data.

See Also

| | |

Related Topics

Go to top of page