MATLAB Answers

Why is the accuracy reported in the Classification Learner app much higher than the accuracy of the exported model on the held out validation set?

2 views (last 30 days)
asaf benjamin
asaf benjamin on 3 Mar 2020
Edited: asaf benjamin on 10 Mar 2020
Problem description
I used the default 5-fold cross-validation (CV) scheme in the Classification Learner app and trained all the available models. The best model (quadratic SVM) has 74.2% accuracy. I used
export model => generate code
and then ran the generated code, again examining the 5-fold CV accuracy. Surprisingly, the validation accuracy of this generated model was 66.8%, which is much lower than that reported by the app.
This has happened repeatedly - everytime I follow this scheme (with different datasets and models) I observe reduced accuracy for the exported model.
Potential explanations
I understand that not resetting the random number generation seed may lead to some variability between runs, but this effect seems too large and consistent to be explained by chance. I also understand that choosing the empirically-best model on the validation set, without having an additional test set, may lead to overestimation of its performance. If the exported model has a different CV partition, that could explain some of the reduction in accuracy, but I wonder if there might be some other explanation that I am missing.

  0 Comments

Sign in to comment.

Answers (1)

Jalaj Gambhir
Jalaj Gambhir on 6 Mar 2020
Hi,
You can have a look at the answer here. Hope it helps!

  1 Comment

asaf benjamin
asaf benjamin on 10 Mar 2020
Hi,
Thanks for the response, but that answer deals with a very different issue, where one mistakingly compares cross-validation accuracy to accuracy on the training data. Under no circumstances would I try to test my model's performance on the data on which it trained. Rather, I am comparing
  1. The 5-fold cross-validation accuracy reported by the app, to
  2. The 5-fold cross-validation accuracy reported by the code generated by the app.
These should be equal (or at least very close), as I understand it, but they are in fact very different...

Sign in to comment.


Translated by