Classification Learner exported Function not generating the same results as the exported Model

I use the Classification Learner to interactively build a Model (from imported dataset OULearn), and then export the Model (as ModelFromExport) back to MatLab. I run a small Prediction set (OUPredict) on this model:

>> ExportResults = ModelFromExport.predictFcn(OUPredict)

And get the following results:

ExportResults =

1

1

1

1

1

1

1

1

-1

1

1

-1

I then go back to the Classification Learner, and generate the Function for this Model (and rename it as BuildFromFunction - Code shown below). This function is saved, creating the code file BuildFromFunction.m

A new Model (ModelFromFunction) is built using this function on the same Learning data set:

>> ModelFromFunction = BuildFromFunction(OULearn)

This new model is then used to predict against the same prediction data set:

>> FunctionResults = ModelFromFunction.predictFcn(OUPredict)

The results should be identical. Same Learning set. Same Prediction set. One set of results generated directly from the Classification Learner Model, and the other generated from the Function code for that model. But, the results are not the same:

FunctionResults =

1

1

1

-1

1

1

1

1

1

1

1

-1

My Question is: Why is this happening? Is it to be Expected?

My guess is that there is something the Classification Learner is doing when it builds a model interactively that is not accounted for in the generated Function code. The results from the Interactively built Model are consistently superior to the results from the model built from the Function. I need to automate the training process, but can't find any solution to this problem. I've attempted to boil this issue down to the simplest example. No matter what I've tried, the results are always different, and sometimes radically different.

NOTE: For brevity I have not included the contents of the Learning & Prediction data sets. They are small (140 records in the training set), very straight forward (all numeric fields, with no NaN) and contain 100% clean data. Also, if I just go back to Classification Learner and train the model interactively again and again, I always get the same results.

The type of model is: Kernel Logistic Regression, and the associated parameters can be found in the code.

Help!

Here is the Generated Function code from the Classification Learner (only edit being the Name of the function):

function [trainedClassifier, validationAccuracy] = BuildFromFunction(trainingData)

% [trainedClassifier, validationAccuracy] = trainClassifier(trainingData)

% Returns a trained classifier and its accuracy. This code recreates the

% classification model trained in Classification Learner app. Use the

% generated code to automate training the same model with new data, or to

% learn how to programmatically train models.

%

% Input:

% trainingData: A table containing the same predictor and response

% columns as those imported into the app.

%

%

% Output:

% trainedClassifier: A struct containing the trained classifier. The

% struct contains various fields with information about the trained

% classifier.

%

% trainedClassifier.predictFcn: A function to make predictions on new

% data.

%

% validationAccuracy: A double representing the validation accuracy as

% a percentage. In the app, the Models pane displays the validation

% accuracy for each model.

%

% Use the code to train the model with new data. To retrain your

% classifier, call the function from the command line with your original

% data or new data as the input argument trainingData.

%

% For example, to retrain a classifier trained with the original data set

% T, enter:

% [trainedClassifier, validationAccuracy] = trainClassifier(T)

%

% To make predictions with the returned 'trainedClassifier' on new data T2,

% use

% [yfit,scores] = trainedClassifier.predictFcn(T2)

%

% T2 must be a table containing at least the same predictor columns as used

% during training. For details, enter:

% trainedClassifier.HowToPredict

% Auto-generated by MATLAB on 16-Nov-2024 11:19:58

% Extract predictors and response

% This code processes the data into the right shape for training the

% model.

inputTable = trainingData;

predictorNames = {'InitOU', 'OpenOU', 'LineMove1', 'LineMove3', 'CloseOU', 'OpenHP', 'CloseHP', 'Number', 'Max', 'Min', 'StdDev', 'Var', 'Skew'};

predictors = inputTable(:, predictorNames);

response = inputTable.Result;

isCategoricalPredictor = [false, false, false, false, false, false, false, false, false, false, false, false, false];

classNames = [-1; 1];

% Train a classifier

% This code specifies all the classifier options and trains the classifier.

classificationKernel = fitckernel(...

predictors, ...

response, ...

'Learner', 'logistic', ...

'NumExpansionDimensions', 2048, ...

'Lambda', 'auto', ...

'KernelScale', 'auto', ...

'Standardize', true, ...

'IterationLimit', 50000, ...

'ClassNames', classNames);

% Create the result struct with predict function

predictorExtractionFcn = @(t) t(:, predictorNames);

kernelPredictFcn = @(x) predict(classificationKernel, x);

trainedClassifier.predictFcn = @(x) kernelPredictFcn(predictorExtractionFcn(x));

% Add additional fields to the result struct

trainedClassifier.RequiredVariables = {'InitOU', 'OpenOU', 'LineMove1', 'LineMove3', 'CloseOU', 'OpenHP', 'CloseHP', 'Number', 'Max', 'Min', 'StdDev', 'Var', 'Skew'};

trainedClassifier.ClassificationKernel = classificationKernel;

trainedClassifier.About = 'This struct is a trained model exported from Classification Learner R2024a.';

trainedClassifier.HowToPredict = sprintf('To make predictions on a new table, T, use: \n [yfit,scores] = c.predictFcn(T) \nreplacing ''c'' with the name of the variable that is this struct, e.g. ''trainedModel''. \n \nThe table, T, must contain the variables returned by: \n c.RequiredVariables \nVariable formats (e.g. matrix/vector, datatype) must match the original training data. \nAdditional variables are ignored. \n \nFor more information, see <a href="matlab:helpview(fullfile(docroot, ''stats'', ''stats.map''), ''appclassification_exportmodeltoworkspace'')">How to predict using an exported model</a>.');

% Extract predictors and response

% This code processes the data into the right shape for training the

% model.

inputTable = trainingData;

predictorNames = {'InitOU', 'OpenOU', 'LineMove1', 'LineMove3', 'CloseOU', 'OpenHP', 'CloseHP', 'Number', 'Max', 'Min', 'StdDev', 'Var', 'Skew'};

predictors = inputTable(:, predictorNames);

response = inputTable.Result;

isCategoricalPredictor = [false, false, false, false, false, false, false, false, false, false, false, false, false];

classNames = [-1; 1];

% Perform cross-validation

KFolds = 20;

cvp = cvpartition(response, 'KFold', KFolds);

% Initialize the predictions to the proper sizes

validationPredictions = response;

numObservations = size(predictors, 1);

numClasses = 2;

validationScores = NaN(numObservations, numClasses);

for fold = 1:KFolds

trainingPredictors = predictors(cvp.training(fold), :);

trainingResponse = response(cvp.training(fold), :);

foldIsCategoricalPredictor = isCategoricalPredictor;

% Train a classifier

% This code specifies all the classifier options and trains the classifier.

classificationKernel = fitckernel(...

trainingPredictors, ...

trainingResponse, ...

'Learner', 'logistic', ...

'NumExpansionDimensions', 2048, ...

'Lambda', 'auto', ...

'KernelScale', 'auto', ...

'Standardize', true, ...

'IterationLimit', 50000, ...

'ClassNames', classNames);

% Create the result struct with predict function

kernelPredictFcn = @(x) predict(classificationKernel, x);

validationPredictFcn = @(x) kernelPredictFcn(x);

% Add additional fields to the result struct

% Compute validation predictions

validationPredictors = predictors(cvp.test(fold), :);

[foldPredictions, foldScores] = validationPredictFcn(validationPredictors);

% Store predictions in the original order

validationPredictions(cvp.test(fold), :) = foldPredictions;

validationScores(cvp.test(fold), :) = foldScores;

end

% Compute validation accuracy

correctPredictions = (validationPredictions == response);

isMissing = isnan(response);

correctPredictions = correctPredictions(~isMissing);

validationAccuracy = sum(correctPredictions)/length(correctPredictions);

Answer 1

[Shantanu Dixit]

Hi Lawrence,

The unmatching results are likely due to the cross-validation performed in the generated function. When the model is exported from the Classification Learner app, it is fully trained on the entire dataset, resulting in a single deterministic model. However, the generated function employs K-fold cross-validation, training multiple models on subsets of the training data and aggregating their predictions.

This aggregation introduces variability because the validation predictions across different folds are generated by models trained on different subsets of the data. As a result, predictions for the same validation set can differ compared to those from a single model trained on the full dataset.

To check for consistency, you can modify the generated function to train the model on the complete training dataset (without cross-validation) and compare its predictions with those from the exported model for the prediction set.

I hope this helps!

[Image Analyst]

Yes, this is true. I learned this after a call to tech support a few years ago when I encountered the same curious thing.

Answer 2

[Drew]

The differences could be due to different state of the random number generation before training. For both the command line (CLI) and the app, try setting the random seed to the same value just before training the model. For example, use

rng("default")

For the app training, this command can be run at the command line before training the model in the app, and turn off "Use Parallel" in the app, to avoid different random seeds on different parallel threads or processes.