fitSVMPosterior

Fit posterior probabilities

Syntax

ScoreSVMModel = fitSVMPosterior(SVMModel)

ScoreSVMModel = fitSVMPosterior(SVMModel,TBL,ResponseVarName)

ScoreSVMModel = fitSVMPosterior(SVMModel,TBL,Y)

ScoreSVMModel = fitSVMPosterior(SVMModel,X,Y)

ScoreSVMModel = fitSVMPosterior(___,Name,Value)

[ScoreSVMModel,ScoreTransform] = fitSVMPosterior(___)

Description

ScoreSVMModel = fitSVMPosterior(SVMModel) returns ScoreSVMModel, which is a trained, support vector machine (SVM) classifier containing the optimal score-to-posterior-probability transformation function for two-class learning.

The software fits the appropriate score-to-posterior-probability transformation function using the SVM classifier SVMModel, and by cross validation using the stored predictor data (SVMModel.X) and the class labels (SVMModel.Y). The transformation function computes the posterior probability that an observation is classified into the positive class (SVMModel.Classnames(2)).

• If the classes are inseparable, then the transformation function is the sigmoid function.

• If the classes are perfectly separable, the transformation function is the step function.

• In two-class learning, if one of the two classes has a relative frequency of 0, then the transformation function is the constant function. fitSVMPosterior is not appropriate for one-class learning.

• If SVMModel is a ClassificationSVM classifier, then the software estimates the optimal transformation function by 10-fold cross validation as outlined in [1]. Otherwise, SVMModel must be a ClassificationPartitionedModel classifier. SVMModel specifies the cross-validation method.

• The software stores the optimal transformation function in ScoreSVMModel.ScoreTransform.

ScoreSVMModel = fitSVMPosterior(SVMModel,TBL,ResponseVarName) returns a trained support vector classifier containing the transformation function from the trained, compact SVM classifier SVMModel. The software estimates the score transformation function using predictor data in the table TBL and class labels TBL.ResponseVarName.

ScoreSVMModel = fitSVMPosterior(SVMModel,TBL,Y) returns a trained support vector classifier containing the transformation function from the trained, compact SVM classifier SVMModel. The software estimates the score transformation function using predictor data in the table TBL and class labels Y.

ScoreSVMModel = fitSVMPosterior(SVMModel,X,Y) returns a trained support vector classifier containing the transformation function from the trained, compact SVM classifier SVMModel. The software estimates the score transformation function using predictor data X and class labels Y.

ScoreSVMModel = fitSVMPosterior(___,Name,Value) uses additional options specified by one or more Name,Value pair arguments provided SVMModel is a ClassificationSVM classifier. For example, you can specify the number of folds to use in k-fold cross validation.

[ScoreSVMModel,ScoreTransform] = fitSVMPosterior(___) additionally returns the transformation function parameters (ScoreTransform) using any of the input arguments in the previous syntaxes.

Examples

Fit the Score-to-Posterior Probability Function for Separable Classes

Load Fisher's iris data set. Train the classifier using the petal lengths and widths, and remove the virginica species from the data.

load fisheriris
classKeep = ~strcmp(species,'virginica');
X = meas(classKeep,3:4);
y = species(classKeep);

gscatter(X(:,1),X(:,2),y);
title('Scatter Diagram of Iris Measurements')
xlabel('Petal length')
ylabel('Petal width')
legend('Setosa','Versicolor')

The classes are perfectly separable. Therefore, the score transformation function is a step function.

Train an SVM classifier using the data. Cross validate the classifier using 10-fold cross validation (the default).

rng(1);
CVSVMModel = fitcsvm(X,y,'CrossVal','on');

CVSVMModel is a trained ClassificationPartitionedModel SVM classifier.

Estimate the step function that transforms scores to posterior probabilities.

[ScoreCVSVMModel,ScoreParameters] = fitSVMPosterior(CVSVMModel);

Warning: Classes are perfectly separated. The optimal score-to-posterior transformation is a step function.

fitSVMPosterior does the following:

• Uses the data that the software stored in CVSVMModel to fit the transformation function

• Warns whenever the classes are separable

• Stores the step function in ScoreCSVMModel.ScoreTransform

Display the score function type and its parameter values.

ScoreParameters

ScoreParameters = struct with fields:
                        Type: 'step'
                  LowerBound: -0.8431
                  UpperBound: 0.6897
    PositiveClassProbability: 0.5000

ScoreParameters is a structure array with four fields:

• The score transformation function type (Type)

• The score corresponding to the negative class boundary (LowerBound)

• The score corresponding to the positive class boundary (UpperBound)

• The positive class probability (PositiveClassProbability)

Since the classes are separable, the step function transforms the score to either 0 or 1, which is the posterior probability that an observation is a versicolor iris.

Fit the Score-to-Posterior Probability Function for Inseparable Classes

Load the ionosphere data set.

load ionosphere

The classes of this data set are not separable.

Train an SVM classifier. Cross validate using 10-fold cross validation (the default). It is good practice to standardize the predictors and specify the class order.

rng(1) % For reproducibility
CVSVMModel = fitcsvm(X,Y,'ClassNames',{'b','g'},'Standardize',true,...
    'CrossVal','on');
ScoreTransform = CVSVMModel.ScoreTransform

ScoreTransform = 
'none'

CVSVMModel is a trained ClassificationPartitionedModel SVM classifier. The positive class is 'g'. The ScoreTransform property is none.

Estimate the optimal score function for mapping observation scores to posterior probabilities of an observation being classified as 'g'.

[ScoreCVSVMModel,ScoreParameters] = fitSVMPosterior(CVSVMModel);
ScoreTransform = ScoreCVSVMModel.ScoreTransform

ScoreTransform = 
'@(S)sigmoid(S,-9.481576e-01,-1.218300e-01)'

ScoreParameters

ScoreParameters = struct with fields:
         Type: 'sigmoid'
        Slope: -0.9482
    Intercept: -0.1218

ScoreTransform is the optimal score transform function. ScoreParameters contains the score transformation function, slope estimate, and the intercept estimate.

You can estimate test-sample, posterior probabilities by passing ScoreCVSVMModel to kfoldPredict.

Estimate Posterior Probabilities for Test Samples

Estimate positive class posterior probabilities for the test set of an SVM algorithm.

Load the ionosphere data set.

load ionosphere

Train an SVM classifier. Specify a 20% holdout sample. It is good practice to standardize the predictors and specify the class order.

rng(1) % For reproducibility
CVSVMModel = fitcsvm(X,Y,'Holdout',0.2,'Standardize',true,...
    'ClassNames',{'b','g'});

CVSVMModel is a trained ClassificationPartitionedModel cross-validated classifier.

Estimate the optimal score function for mapping observation scores to posterior probabilities of an observation being classified as 'g'.

ScoreCVSVMModel = fitSVMPosterior(CVSVMModel);

ScoreSVMModel is a trained ClassificationPartitionedModel cross-validated classifier containing the optimal score transformation function estimated from the training data.

Estimate the out-of-sample positive class posterior probabilities. Display the results for the first 10 out-of-sample observations.

[~,OOSPostProbs] = kfoldPredict(ScoreCVSVMModel);
indx = ~isnan(OOSPostProbs(:,2));
hoObs = find(indx); % Holdout observation numbers
OOSPostProbs = [hoObs, OOSPostProbs(indx,2)];
table(OOSPostProbs(1:10,1),OOSPostProbs(1:10,2),...
    'VariableNames',{'ObservationIndex','PosteriorProbability'})

ans=10×2 table
    ObservationIndex    PosteriorProbability
    ________________    ____________________

            6                   0.17378     
            7                   0.89637     
            8                 0.0076609     
            9                   0.91602     
           16                  0.026718     
           22                4.6081e-06     
           23                    0.9024     
           24                2.4129e-06     
           38                0.00042697     
           41                   0.86427     

Input Arguments

SVMModel — Trained SVM classifier

ClassificationSVM classifier | CompactClassificationSVM classifier | ClassificationPartitionedModel classifier

Trained SVM classifier, specified as a ClassificationSVM, CompactClassificationSVM, or ClassificationPartitionedModel classifier.

If SVMModel is a ClassificationSVM classifier, then you can set optional name-value pair arguments.

If SVMModel is a CompactClassificationSVM classifier, then you must input predictor data X and class labels Y.

TBL — Sample data

table

Sample data, specified as a table. Each row of TBL corresponds to one observation, and each column corresponds to one predictor variable. Optionally, TBL can contain additional columns for the response variable and observation weights. TBL must contain all of the predictors used to train SVMModel. Multicolumn variables and cell arrays other than cell arrays of character vectors are not allowed.

If TBL contains the response variable used to train SVMModel, then you do not need to specify ResponseVarName or Y.

If you trained SVMModel using sample data contained in a table, then the input data for fitSVMPosterior must also be in a table.

If you set 'Standardize',true in fitcsvm when training SVMModel, then the software standardizes the columns of the predictor data using the corresponding means in SVMModel.Mu and the standard deviations in SVMModel.Sigma.

Data Types: table

X — Predictor data

matrix

Predictor data used to estimate the score-to-posterior-probability transformation function, specified as a matrix.

Each row of X corresponds to one observation (also known as an instance or example), and each column corresponds to one variable (also known as a feature).

The length of Y and the number of rows in X must be equal.

If you set 'Standardize',true in fitcsvm when training SVMModel, then the software fits the transformation function parameter estimates using standardized data.

Data Types: double | single

ResponseVarName — Response variable name

name of variable in TBL

Response variable name, specified as the name of a variable in TBL. If TBL contains the response variable used to train SVMModel, then you do not need to specify ResponseVarName.

If you specify ResponseVarName, then you must do so as a character vector or string scalar. For example, if the response variable is stored as TBL.Response, then specify ResponseVarName as 'Response'. Otherwise, the software treats all columns of TBL, including TBL.Response, as predictors.

The response variable must be a categorical, character, or string array, logical or numeric vector, or cell array of character vectors. If the response variable is a character array, then each element must correspond to one row of the array.

Data Types: char | string

Y — Class labels

categorical array | character array | string array | logical vector | numeric vector | cell array of character vectors

Class labels used to estimate the score-to-posterior-probability transformation function, specified as a categorical, character, or string array, a logical or numeric vector, or a cell array of character vectors.

If Y is a character array, then each element must correspond to one class label.

The length of Y and the number of rows in X must be equal.

Data Types: categorical | char | string | logical | single | double | cell

Name-Value Pair Arguments

Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

Example: 'KFold',8 performs 8-fold cross validation when SVMModel is a ClassificationSVM classifier.

'CVPartition' — Cross-validation partition

[] (default) | cvpartition partition

Cross-validation partition used to compute the transformation function, specified as the comma-separated pair consisting of 'CVPartition' and a cvpartition partition object as created by cvpartition. You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

The crossval name-value pair argument of fitcsvm splits the data into subsets using cvpartition.

Example: Suppose you create a random partition for 5-fold cross-validation on 500 observations by using cvp = cvpartition(500,'KFold',5). Then, you can specify the cross-validated model by using 'CVPartition',cvp.

'Holdout' — Fraction of data for holdout validation

scalar value in the range (0,1)

Fraction of the data for holdout validation used to compute the transformation function, specified as the comma-separated pair consisting of 'Holdout' and a scalar value in the range (0,1). Holdout validation tests the specified fraction of the data and uses the remaining data for training.

You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

Example: 'Holdout',0.1

Data Types: double | single

'KFold' — Number of folds

10 (default) | positive integer value greater than 1

Number of folds to use when computing the transformation function, specified as the comma-separated pair consisting of 'KFold' and a positive integer value greater than 1.

You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

Example: 'KFold',8

Data Types: single | double

'Leaveout' — Leave-one-out cross-validation flag

'off' (default) | 'on'

Leave-one-out cross-validation flag indicating whether to use leave-one-out cross-validation to compute the transformation function, specified as the comma-separated pair consisting of 'Leaveout' and 'on' or 'off'. Use leave-one-out cross-validation by specifying 'Leaveout','on'.

You can use only one of these four options at a time for creating a cross-validated model: 'KFold', 'Holdout', 'Leaveout', or 'CVPartition'.

Example: 'Leaveout','on'

Output Arguments

ScoreSVMModel — Trained SVM classifier

ClassificationSVM classifier | CompactClassificationSVM classifier | ClassificationPartitionedModel classifier

Trained SVM classifier containing the estimated score transformation function, returned as a ClassificationSVM, CompactClassificationSVM, or ClassificationPartitionedModel classifier.

The ScoreSVMModel classifier type is the same as the SVMModel classifier type.

To estimate posterior probabilities, pass ScoreSVMModel and predictor data to predict. If you set 'Standardize',true in fitcsvm to train SVMModel, then predict standardizes the columns of X using the corresponding means in SVMModel.Mu and standard deviations in SVMModel.Sigma.

ScoreTransform — Optimal score-to-posterior-probability transformation function parameters

structure array

Optimal score-to-posterior-probability transformation function parameters, specified as a structure array. If field Type is:

• sigmoid, then ScoreTransform has these fields:

  • Slope — The value of A in the sigmoid function

  • Intercept — The value of B in the sigmoid function

• step, then ScoreTransform has these fields:

  • PositiveClassProbability: the value of π in the step function. π represents:

    • The probability that an observation is in the positive class.

    • The posterior probability that a score is in the interval (LowerBound,UpperBound).

  • LowerBound: the value $$\underset{y_n=-1}{\max }{s}_n$$ in the step function. It represents the lower bound of the interval that assigns the posterior probability of being in the positive class PositiveClassProbability to scores. Any observation with a score less than LowerBound has posterior probability of being the positive class 0.

  • UpperBound: the value $$\underset{y_n=+1}{\min }{s}_n$$ in the step function. It represents the upper bound of the interval that assigns the posterior probability of being in the positive class PositiveClassProbability. Any observation with a score greater than UpperBound has posterior probability of being the positive class 1.

• constant, then ScoreTransform.PredictedClass contains the name of the class prediction.

  This result is the same as SVMModel.ClassNames. The posterior probability of an observation being in ScoreTransform.PredictedClass is always 1.

More About

Sigmoid Function

The sigmoid function that maps score s_j corresponding to observation j to the positive class posterior probability is

$$P(s_j)=\frac{1}{1+\exp (A{s}_j+B)}.$$

If the value of the Type field of ScoreTransform is sigmoid, then parameters A and B correspond to the fields Scale and Intercept of ScoreTransform, respectively.

Step Function

The step function that maps score s_j corresponding to observation j to the positive class posterior probability is

$$P\left(s_j\right)=\{\begin{array}{l}\begin{array}{cc}0;& s<\underset{y_k=-1}{\max }{s}_k\end{array}\\ \begin{array}{cc}\pi ;& \underset{y_k=-1}{\max }{s}_k\le s_j\le \underset{y_k=+1}{\min }{s}_k\end{array}\\ \begin{array}{cc}1;& s_j>\underset{y_k=+1}{\min }{s}_k\end{array}\end{array},$$

where:

• s_j is the score of observation j.

• +1 and –1 denote the positive and negative classes, respectively.

• π is the prior probability that an observation is in the positive class.

If the value of the Type field of ScoreTransform is step, then the quantities $$\underset{y_k=-1}{\max }{s}_k$$ and $$\underset{y_k=+1}{\min }{s}_k$$ correspond to the fields LowerBound and UpperBound of ScoreTransform, respectively.

Constant Function

The constant function maps all scores in a sample to posterior probabilities 1 or 0.

If all observations have posterior probability 1, then they are expected to come from the positive class.

If all observations have posterior probability 0, then they are not expected to come from the positive class.

Tips

• This process describes one way to predict positive class posterior probabilities.

  1. Train an SVM classifier by passing the data to fitcsvm. The result is a trained SVM classifier, such as SVMModel, that stores the data. The software sets the score transformation function property (SVMModel.ScoreTransformation) to none.

  2. Pass the trained SVM classifier SVMModel to fitSVMPosterior or fitPosterior. The result, such as, ScoreSVMModel, is the same trained SVM classifier as SVMModel, except the software sets ScoreSVMModel.ScoreTransformation to the optimal score transformation function.

  3. Pass the predictor data matrix and the trained SVM classifier containing the optimal score transformation function (ScoreSVMModel) to predict. The second column in the second output argument of predict stores the positive class posterior probabilities corresponding to each row of the predictor data matrix.

     If you skip step 2, then predict returns the positive class score rather than the positive class posterior probability.

• After fitting posterior probabilities, you can generate C/C++ code that predicts labels for new data. Generating C/C++ code requires MATLAB^® Coder™. For details, see Introduction to Code Generation.

Algorithms

If you re-estimate the score-to-posterior-probability transformation function, that is, if you pass an SVM classifier to fitPosterior or fitSVMPosterior and its ScoreTransform property is not none, then the software:

• Displays a warning

• Resets the original transformation function to 'none' before estimating the new one

References

[1] Platt, J. “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods”. In: Advances in Large Margin Classifiers. Cambridge, MA: The MIT Press, 2000, pp. 61–74.