Change the range of potential hyperparameters when optimizing hyperparameter choice using fitrensemble

Hi,

I am experimenting with randon search and grid search optimizers when optimizing hyperparameters for a boosted trees regression model, using the fitrensemble function.

To speed up the process, and also avoid overfitting, I want to specify non-default ranges to explore for the hyperparamters - however after reading documentation, I still don't understand how to do this.

For example, in the code below:

Mdl_ls = fitrensemble(X,Y,'Learners','tree', ...
    'OptimizeHyperparameters',{'NumLearningCycles','LearnRate','MaxNumSplits'},...
    'HyperparameterOptimizationOptions',struct(...
    'Optimizer','randomsearch', 'MaxObjectiveEvaluations', 500, 'ShowPlots', false, 'Verbose', 1 ));

How would I go about changing the search range for the LearnRate to between 0.001 and 0.2 - rather than the defaul range of [1e-3,1].

Answer 1

[Alan Weiss]

Looking in the documentation for fitrensemble I find this:

----

Set nondefault parameters by passing a vector of optimizableVariable objects that have nondefault values. For example,

load carsmall
params = hyperparameters('fitrensemble',[Horsepower,Weight],MPG,'Tree');
params(4).Range = [1,20];

Pass params as the value of OptimizeHyperparameters.

----

Is that clear to you? For more information, see https://www.mathworks.com/help/stats/hyperparameters.html#bvdtepr-1

Alan Weiss

MATLAB mathematical toolbox documentation

[Adam Richardson]

Thanks!

Therefore, for my question - the approach is (setting range for three hyperparameters and optimizing these):

params = hyperparameters('fitrensemble',X,Y,'Tree');
params(2).Range =  [500, 1000]    %NumLearningCycles
params(3).Range = [0.001, 0.2]     %LearnRate
params(5).Range =  [5, 15]    %MaxNumSplits

params(1).Optimize = false;
params(2).Optimize = true;
params(3).Optimize = true;
params(4).Optimize = false;
params(5).Optimize = true;
params(6).Optimize = false;


Mdl_ls = fitrensemble(X,Y,'Learners','tree', ...
    'OptimizeHyperparameters',params,...
    'HyperparameterOptimizationOptions',struct(...
    'Optimizer','randomsearch', 'MaxObjectiveEvaluations', 500, 'ShowPlots', false, 'Verbose', 1, 'Holdout', 2/3 ));
    

[Alan Weiss]

Good job! Thanks for the follow-up showing your solution.

Alan Weiss

MATLAB mathematical toolbox documentation

[Machine Learning Enthusiast]

@Adam Richardson @Alan Weiss How to restrict the other parameters? For example, I am tyring to use only 'Bag' method in the search space instead of 'LSBoost'

Name: 'Method'
        Range: {'Bag'  'LSBoost'}
         Type: 'categorical'
    Transform: 'none'
     Optimize: 1

[Alan Weiss]

Assuming that the Method hyperparameter is variable 1,

VariableDescriptions(1).Range = {'Bag'};
VariableDescriptions(1).Optimize = false;

Call hyperparameters with your data to find out which variable number it is.

Alan Weiss

MATLAB mathematical toolbox documentation

[Machine Learning Enthusiast]

I am getting following error:

params(1).Range = {'Bag'};

Error using optimizableVariable/checkRange (line 172)
'Range' value must be a string array, a cell array of character vectors, or a numeric vector of length 2.

[Alan Weiss]

Sorry, my mistake. It seems that if you want to have a single value there you have to repeat it.

params(1).Range = {'Bag' 'Bag'};
params(1).Optimize = false;

Maybe this is bad advice somehow, I don't know, but it seems to work.

Alan Weiss

MATLAB mathematical toolbox documentation