Classification problem parsed as regression problem when Split Criterion is supplied to fitcensemble

Hi

I ran a hyperparameter optimization to find the best parameters for a two-class classification problem using fitcensemble. But when I try to use these I get a strange warning:

Warning: You must pass 'SplitCriterion' as a character vector 'mse' for regression.

What is wrong with my code? The warning comes when I use a boosting ensemble as 'method'. When I remove the 'SplitCriterion' everything works fine, but I cannot understand why Matlab somewhere on the line thinks this is a regression problem when I use fit"c"ensemble. Here is a toy example with arbitrarily chosen settings that you can run to reproduce the Warning/Error.

load carsmall
X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,Model_Year,Weight,MPG);

X.Cylinders(X.Cylinders < 8) = 0;  % Create two classes in the Cylinders variable

t = templateTree(  'MaxNumSplits', 30,...
                    'MinLeafSize', 10,...
                    'SplitCriterion', 'gdi');

classificationEnsemble = fitcensemble(X,'Cylinders',...
    'Method', 'LogitBoost', ...
    'NumLearningCycles',12,...
    'Learners',t,...
    'KFold',7,...
    'LearnRate',0.1);

[Don Mathis]

Logitboost internally fits regression trees, and the gdi split criterion doesn't work for regression. So the problem is that 'LogitBoost' is incompatible with 'gdi'. The warning messages you get are pretty bad at explaining the problem, however.

When I run your code, I get a useless model as output. I don't understand how an optimization could have chosen these to be the best hyperparameters when the model is unusable. Did you use the 'bayesopt' function or the 'OptimizeHyperparameters' argument to do the optimization? If so, I would like to see the code for that.

[Tobias Pahlberg]

No, I just chose values at random here for this example :) so this is not from the optimization.

But when I run my real optimization I can get such a line with LogitBoost and gdi:

|=============================================================================================================================================================|
| Iter | Eval   | Objective  | Objective  | BestSoFar  | BestSoFar  |       Method |    LearnRate |  MinLeafSize | MaxNumSplits | SplitCriteri-| NumVariables-|
|      | result |            | runtime    | (observed) | (estim.)   |              |              |              |              | on           | ToSample     |
|=============================================================================================================================================================|
|  169 | Accept |    0.21084 |     25.592 |    0.14458 |    0.15189 |   LogitBoost |      0.70567 |            8 |            3 |          gdi |            - |

And the documentation does not explain this incompatibility as far as I can see. Here is what is seen in TemplateTree:

'SplitCriterion' — Split criterion
'gdi' | 'twoing' | 'deviance' | 'mse'
Split criterion, specified as the comma-separated pair consisting of 'SplitCriterion' and a split criterion valid for the tree type.

For classification trees:
'gdi' for Gini's diversity index (default)
'twoing' for the twoing rule
'deviance' for maximum deviance reduction (also known as cross entropy)
For regression trees:
'mse' for mean squared error (default)
Example: 'SplitCriterion','deviance'

[Don Mathis]

When I run an optimization I never see successes for LogitBoost+gdi. Nor GentleBoost+gdi. They fail and eventually are not tried any more. Could you post a reproducible example of an optimization that shows successes for those combinations? That would be very helpful.

In any case, there is a problem in that LogitBoost is never run with 'mse', which is the only SplitCriterion it can use. As a workaround, you might try running a separate optimization without optimizing SplitCriterion. Then you could take the best result from the 2 optimizations. Something like this:

load carsmall
X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,Model_Year,Weight,MPG);

X.Cylinders(X.Cylinders < 8) = 0;  % Create two classes in the Cylinders variable

classificationEnsemble = fitcensemble(X,'Cylinders',...
    'NumLearningCycles',12,...
    'Learners','Tree',...
    'OptimizeHyperparameters', {'Method', 'LearnRate', 'MinLeafSize', 'MaxNumSplits', 'NumVariablesToSample'})

[Tobias Pahlberg]

If I run the toy example like this, with 'OptimizeHyperparameters', 'all':

load carsmall
X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,Model_Year,Weight,MPG);
X.Cylinders(X.Cylinders < 8) = 0;  % Create two classes in the Cylinders variable

strct = struct( 'KFold', 7, 'Verbose',1, 'MaxObjectiveEvaluations',1000, 'SaveIntermediateResults', true, ...
                        'Repartition',false);        

classificationEnsemble = fitcensemble(X, 'Cylinders',...
            'Method', 'bag', ...
            'Type', 'Tree', ...
            'NumLearningCycles', 300,...
            'OptimizeHyperparameters', 'all',...
            'HyperparameterOptimizationOptions', strct);

I cannot see that they fail, but I don't know how that should look.

Answer 1

[Don Mathis]

Thanks, I now see cases succeeding with 'GentleBoost' and 'gdi' together.

This is a bug in the search space for hyperparameter optimization in fitcensemble, which will cause it to evaluate some unnecessary points. During the optimization, whenever 'Method' is LogitBoost or GentleBoost, 'SplitCriterion' is always internally set to 'mse'. But the search space is defined in a way that doesn't acknowledge that, so it unnecessarily passes SplitCrierion values of 'gdi' and 'deviance' during the optimization.

So all points that you see with LogitBoost/gdi and LogitBoost/deviance are really just LogitBoost/mse.

Oddly, you are not allowed to explicitly specify 'SplitCriterion','mse' with fitcensemble. Instead, when LogitBoost or GentleBoost are used, you need to omit the SplitCriterion argument entirely.

The optimization results are still valid. You just need to adjust what arguments you pass to fitcensemble at the end. The simple fix in your case is to delete line 8 of your original code sample, which specifies the SplitCriterion. Here's the result:

load carsmall
X = table(Acceleration,Cylinders,Displacement,Horsepower,Mfg,Model_Year,Weight,MPG);

X.Cylinders(X.Cylinders < 8) = 0;  % Create two classes in the Cylinders variable

t = templateTree(  'MaxNumSplits', 30,...
                    'MinLeafSize', 10);

classificationEnsemble = fitcensemble(X,'Cylinders',...
    'Method', 'LogitBoost', ...
    'NumLearningCycles',12,...
    'Learners',t,...
    'KFold',7,...
    'LearnRate',0.1);

This bug will be fixed. Thanks very much for reporting this. I hope the workaround isn't too much of a nuisance for you.

[Tobias Pahlberg]

Thanks, no its not a problem :)

[abhinav gaur]

@Don Mathis I am using Matlab 2024B and I am still facing this bug. Was this bug every fixed ? I am not trying to use Hyperparameter optimization. Instead, I am trying to specifiy SplitCriterion as shown below. Also sharing the error message. :(

template = templateTree(...
    'NumVariablesToSample', 'all',...
    'MaxNumSplits', 1000,...
    'PruneCriterion','impurity',...
    'SplitCriterion','gdi');
 

classificationEnsemble = fitcensemble(...
    predictors, ...
    response, ...
    'Learners', template, ...
    'ClassNames', classNames,...
    'Method', 'LogitBoost', ...
    'NumLearningCycles', 30, ...
    'LearnRate', 0.1);

Given below is the error stack.

Error using classreg.learning.modelparams.TreeParams.make (line 97)
'SplitCriterion' value must be 'mse' for regression.

Error in classreg.learning.FitTemplate/fillIfNeeded (line 728)
                    this.MakeModelParams(this.Type,this.MakeModelInputArgs{:});
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in classreg.learning.modelparams.EnsembleParams/fillDefaultParams (line 794)
                        learner = fillIfNeeded(learner,'regression');
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in classreg.learning.modelparams.ModelParams/fillIfNeeded (line 100)
                this = fillDefaultParams(this,X,Y,W,dataSummary,classSummary);
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in classreg.learning.classif.FullClassificationModel (line 35)
            this.ModelParams = fillIfNeeded(modelParams,X,Y,W,dataSummary,classSummary);
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in classreg.learning.classif.ClassificationEnsemble (line 66)
            this = [e-mail](X,Y,W,modelParams,...
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in classreg.learning.FitTemplate/fit (line 291)
            [varargout{1:nargout}] = this.MakeFitObject(X,Y,W,this.ModelParams,fitArgs{:});
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in fitensemble (line 364)
obj = fit(temp,X,Y);
      ^^^^^^^^^^^^^
Error in fitcensemble (line 215)
    this = fitensemble(X, Y, Method, NumLearningCycles, Learners, ...
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error in trainClassifier (line 65)
classificationEnsemble = fitcensemble(...
                         ^^^^^^^^^^^^^^^^
Error in after3 (line 10)
[trainedModel, validationAccuracy] = trainClassifier(trn_data_all);
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[Don Mathis]

The fact that you can't use GDI with Logitboost is not the bug. Logitboost by definition uses MSE. So the error is correct.

The bug that was fixed was that the hyperparameter optimization was trying points in which GDI was passed to Logitboost. That no longer happens.