Main Content

shapley

Shapley values

    Description

    The Shapley value of a feature for a query point explains the deviation of the prediction for the query point from the average prediction, due to the feature. For each query point, the sum of the Shapley values for all features corresponds to the total deviation of the prediction from the average.

    You can create a shapley object for a machine learning model with a specified query point (queryPoint). The software creates an object and computes the Shapley values of all features for the query point.

    Use the Shapley values to explain the contribution of individual features to a prediction at the specified query point. Use the plot function to create a bar graph of the Shapley values. You can compute the Shapley values for another query point by using the fit function.

    Creation

    Description

    explainer = shapley(blackbox) creates a shapley object using a machine learning model object blackbox that contains predictor data. To compute Shapley values, use the fit function with explainer.

    example

    explainer = shapley(blackbox,X) creates a shapley object using the predictor data in X.

    example

    explainer = shapley(___,'QueryPoint',queryPoint) also computes the Shapley values for the query point queryPoint and stores the computed Shapley values in the ShapleyValues property of explainer. You can specify queryPoint in addition to any of the input argument combinations in the previous syntaxes.

    example

    explainer = shapley(___,Name,Value) specifies additional options using one or more name-value arguments. For example, specify 'UseParallel',true to compute Shapley values in parallel.

    Input Arguments

    expand all

    Machine learning model to be interpreted, specified as a full or compact regression or classification model object or a function handle.

    Predictor data, specified as a numeric matrix or table. Each row of X corresponds to one observation, and each column corresponds to one variable.

    • For a numeric matrix:

      • The variables that makes up the columns of X must have the same order as the predictor variables that trained blackbox, stored in blackbox.X.

      • If you trained blackbox using a table, then X can be a numeric matrix if the table contains all numeric predictor variables.

    • For a table:

      • If you trained blackbox using a table (for example, Tbl), then all predictor variables in X must have the same variable names and data types as those in Tbl. However, the column order of X does not need to correspond to the column order of Tbl.

      • If you trained blackbox using a numeric matrix, then the predictor names in blackbox.PredictorNames and the corresponding predictor variable names in X must be the same. To specify predictor names during training, use the 'PredictorNames' name-value argument. All predictor variables in X must be numeric vectors.

      • X can contain additional variables (response variables, observation weights, and so on), but shapley ignores them.

      • shapley does not support multicolumn variables or cell arrays other than cell arrays of character vectors.

    If blackbox is a model object that does not contain predictor data or a function handle, you must provide X. If blackbox is a full machine learning model object and you specify this argument, then shapley does not use the predictor data in blackbox; it uses the specified predictor data only.

    Data Types: single | double

    Query point at which shapley explains a prediction, specified as a row vector of numeric values or a single-row table.

    • For a row vector of numeric values:

      • The variables that makes up the columns of queryPoint must have the same order as X or the predictor variables that trained blackbox, stored in blackbox.X.

      • If you trained blackbox using a table, then queryPoint can be a numeric vector if the table contains all numeric variables.

    • For a single-row table:

      • If you trained blackbox using a table (for example, Tbl), then all predictor variables in queryPoint must have the same variable names and data types as those in Tbl. However, the column order of queryPoint does not need to correspond to the column order of Tbl.

      • If you trained blackbox using a numeric matrix, then the predictor names in blackbox.PredictorNames and the corresponding predictor variable names in queryPoint must be the same. To specify predictor names during training, use the 'PredictorNames' name-value argument. All predictor variables in queryPoint must be numeric vectors.

      • queryPoint can contain additional variables (response variables, observation weights, and so on), but shapley ignores them.

      • shapley does not support multicolumn variables or cell arrays other than cell arrays of character vectors.

    If queryPoint contains NaNs for continuous predictors and 'Method' is 'conditional-kernel', then the Shapley values (ShapleyValues) in the returned object are NaNs. Otherwise, shapley handles NaNs in queryPoint in the same way as blackbox (the predict object function of blackbox or the function handle specified by blackbox).

    Example: blackbox.X(1,:) specifies the query point as the first observation of the predictor data in the full machine learning model blackbox.

    Data Types: single | double | table

    Name-Value Pair Arguments

    Specify optional comma-separated pairs of Name,Value arguments. Name is the argument name and Value is the corresponding value. Name must appear inside quotes. You can specify several name and value pair arguments in any order as Name1,Value1,...,NameN,ValueN.

    Example: shapley(blackbox,'QueryPoint',q,'Method','conditional-kernel') creates a shapley object and computes the Shapley values for the query point q using the extension to the kernelSHAP algorithm.

    Categorical predictors list, specified as one of the values in this table.

    ValueDescription
    Vector of positive integers

    Each entry in the vector is an index value corresponding to the column of the predictor data that contains a categorical variable. The index values are between 1 and p, where p is the number of predictors used to train the model.

    If blackbox uses a subset of input variables as predictors, then the software indexes the predictors using only the subset. The 'CategoricalPredictors' values do not count the response variable, the observation weight variable, and any other variables that the function does not use.

    Logical vector

    A true entry means that the corresponding column of predictor data is a categorical variable. The length of the vector is p.

    Character matrixEach row of the matrix is the name of a predictor variable. The names must match the variable names of the predictor data in the form of a table. Pad the names with extra blanks so each row of the character matrix has the same length.
    String array or cell array of character vectorsEach element in the array is the name of a predictor variable. The names must match the variable names of the predictor data in the form of a table.
    'all'All predictors are categorical.

    • If you specify blackbox as a function handle, then shapley identifies categorical predictors from the predictor data X. If the predictor data is in a table, shapley assumes that a variable is categorical if it is a logical vector, unordered categorical vector, character array, string array, or cell array of character vectors. If the predictor data is a matrix, shapley assumes that all predictors are continuous. To identify any other predictors as categorical predictors, specify them by using the 'CategoricalPredictors' name-value argument.

    • If you specify blackbox as a regression or classification model object, then shapley identifies categorical predictors by using the CategoricalPredictors property of the model object.

    shapley supports an ordered categorical predictor when blackbox supports ordered categorical predictors and 'Method' is 'interventional-kernel'.

    Example: 'CategoricalPredictors','all'

    Data Types: single | double | logical | char | string | cell

    Maximum number of predictor subsets to use for Shapley value computation, specified as a positive integer.

    For details on how shapley chooses the subsets to use, see Complexity of Computing Shapley Values.

    Example: 'MaxNumSubsets',100

    Data Types: single | double

    Shapley value computation algorithm, specified as 'interventional-kernel' or 'conditional-kernel'.

    • 'interventional-kernel' (default) — shapley uses the kernelSHAP algorithm [1] with an interventional value function.

    • 'conditional-kernel'shapley uses the extension to the kernelSHAP algorithm [2] with a conditional value function.

    For details about these algorithms, see Shapley Value Computation Algorithms.

    Example: 'Method','conditional-kernel'

    Data Types: char | string

    Flag to run in parallel, specified as true or false. If you specify 'UseParallel',true, the shapley function executes for-loop iterations in parallel by using parfor. This option requires Parallel Computing Toolbox™.

    Example: 'UseParallel',true

    Data Types: logical

    Properties

    expand all

    This property is read-only.

    Machine learning model to be interpreted, specified as a regression or classification model object or a function handle.

    The blackbox argument sets this property.

    This property is read-only.

    Prediction for the query point computed by the machine learning model (BlackboxModel), specified as a scalar.

    • If BlackboxModel is a model object, then BlackboxFitted is a predicted response for regression or a classified label for classification.

    • If BlackboxModel is a function handle, then BlackboxFitted is a value returned by the function handle, either a predicted response for regression or a predicted score of a single class for classification.

    Data Types: single | double | categorical | logical | char | string | cell

    This property is read-only.

    Categorical predictor indices, specified as a vector of positive integers. CategoricalPredictors contains index values corresponding to the columns of the predictor data that contain categorical predictors. If none of the predictors are categorical, then this property is empty ([]).

    • If you specify blackbox using a function handle, then shapley identifies categorical predictors from the predictor data X. If you specify the 'CategoricalPredictors' name-value argument, then the argument sets this property.

    • If you specify blackbox as a regression or classification model object, then shapley determines this property by using the CategoricalPredictors property of the model object.

    shapley supports an ordered categorical predictor when blackbox supports ordered categorical predictors and 'Method' is 'interventional-kernel'.

    Data Types: single | double

    This property is read-only.

    Shapley value computation algorithm, specified as 'interventional-kernel' or 'conditional-kernel'.

    • 'interventional-kernel'shapley uses the kernelSHAP algorithm [1] with an interventional value function.

    • 'conditional-kernel'shapley uses the extension to the kernelSHAP algorithm [2] with a conditional value function.

    The 'Method' argument of shapley or the 'Method' argument of fit sets this property.

    For details about these algorithms, see Shapley Value Computation Algorithms.

    Data Types: char | string

    This property is read-only.

    Number of predictor subsets to use for Shapley value computation, specified as a positive integer.

    The 'MaxNumSubsets' argument of shapley or the 'MaxNumSubsets' argument of fit sets this property.

    For details on how shapley chooses the subsets to use, see Complexity of Computing Shapley Values.

    Data Types: single | double

    This property is read-only.

    Query point at which shapley explains a prediction using the Shapley values (ShapleyValues), specified as a row vector of numeric values or single-row table.

    The queryPoint argument of shapley or the queryPoint argument of fit sets this property.

    Data Types: single | double | table

    This property is read-only.

    Shapley values for the query point (QueryPoint), specified as a table.

    • For regression, the table has two columns. The first column contains the predictor variable names, and the second column contains the Shapley values of the predictors.

    • For classification, the table has two or more columns, depending on the number of classes in BlackboxModel. The first column contains the predictor variable names, and the rest of the columns contain the Shapley values of the predictors for each class.

    Data Types: table

    This property is read-only.

    Predictor data, specified as a numeric matrix or table.

    Each row of X corresponds to one observation, and each column corresponds to one variable.

    • If you specify the X argument, then it sets this property.

    • If you specify blackbox as a full machine learning model object and do not specify X, then this property value is the predictor data used to train blackbox.

    If an observation contains NaNs for continuous predictors and Method is 'conditional-kernel', then shapley does not use the observation for the Shapley value computation. Otherwise, shapley handles NaNs in X in the same way as BlackboxModel (the predict object function of BlackboxModel or the function handle specified by BlackboxModel).

    shapley stores all observations, including the rows with missing values, in this property.

    Data Types: single | double | table

    Object Functions

    fitCompute Shapley values for query point
    plotPlot Shapley values

    Examples

    collapse all

    Train a classification model and create a shapley object. When you create a shapley object, specify a query point so that the software computes the Shapley values for the query point. Then create a bar graph of the Shapley values by using the object function plot.

    Load the CreditRating_Historical data set. The data set contains customer IDs and their financial ratios, industry labels, and credit ratings.

    tbl = readtable('CreditRating_Historical.dat');

    Display the first three rows of the table.

    head(tbl,3)
    ans=3×8 table
         ID      WC_TA    RE_TA    EBIT_TA    MVE_BVTD    S_TA     Industry    Rating
        _____    _____    _____    _______    ________    _____    ________    ______
    
        62394    0.013    0.104     0.036      0.447      0.142       3        {'BB'}
        48608    0.232    0.335     0.062      1.969      0.281       8        {'A' }
        42444    0.311    0.367     0.074      1.935      0.366       1        {'A' }
    
    

    Train a blackbox model of credit ratings by using the fitcecoc function. Use the variables from the second through seventh columns in tbl as the predictor variables. A recommended practice is to specify the class names to set the order the classes.

    blackbox = fitcecoc(tbl,'Rating', ...
        'PredictorNames',tbl.Properties.VariableNames(2:7), ...
        'CategoricalPredictors','Industry', ...
        'ClassNames',{'AAA' 'AA' 'A' 'BBB' 'BB' 'B' 'CCC'});

    Create a shapley object that explains the prediction for the last observation. Specify a query point so that the software computes Shapley values and stores them in the ShapleyValues property.

    queryPoint = tbl(end,:)
    queryPoint=1×8 table
         ID      WC_TA    RE_TA    EBIT_TA    MVE_BVTD    S_TA    Industry    Rating
        _____    _____    _____    _______    ________    ____    ________    ______
    
        73104    0.239    0.463     0.065      2.924      0.34       2        {'AA'}
    
    
    explainer = shapley(blackbox,'QueryPoint',queryPoint)
    Warning: Computation can be slow because the predictor data has over 1000 observations. Use a smaller sample of the training set or specify 'UseParallel' as true for faster computation.
    
    explainer = 
      shapley with properties:
    
                BlackboxModel: [1x1 ClassificationECOC]
                   QueryPoint: [1x8 table]
               BlackboxFitted: {'AA'}
                ShapleyValues: [6x8 table]
                   NumSubsets: 64
                            X: [3932x6 table]
        CategoricalPredictors: 6
                       Method: 'interventional-kernel'
    
    

    As the warning message indicates, the computation can be slow because the predictor data has over 1000 observations. For faster computation, use a smaller sample of the training set or specify 'UseParallel' as true.

    For a classification model, shapley computes Shapley values using the predicted class score for each class. Display the values in the ShapleyValues property.

    explainer.ShapleyValues
    ans=6×8 table
        Predictor        AAA            AA             A             BBB            BB              B             CCC    
        __________    __________    __________    ___________    ___________    ___________    ___________    ___________
    
        "WC_TA"         0.014715      0.006439       0.002669     0.00048882     -0.0079015      -0.011841      -0.011395
        "RE_TA"         0.047918      0.026904       0.014759     -0.0031481       -0.02512      -0.059927       -0.08418
        "EBIT_TA"      0.0003427    0.00015023     0.00011977     3.3904e-05    -0.00018925    -0.00038136    -0.00033783
        "MVE_BVTD"       0.38334       0.37376        0.17563      -0.032136       -0.18729       -0.24831       -0.19585
        "S_TA"        -0.0037303    -0.0026019    -8.9059e-05    -0.00081782    -5.4905e-05      0.0004788    -0.00069003
        "Industry"     -0.028974     -0.013901      0.0010365       0.023298       0.026474       0.029887       0.045396
    
    

    The ShapleyValues property contains the Shapley values of all features for each class.

    Plot the Shapley values for the predicted class by using the plot function. To display an existing underscore in any predictor name, change the TickLabelInterpreter value of the axes to 'none'.

    f = figure;
    plot(explainer)
    f.CurrentAxes.TickLabelInterpreter = 'none';

    Figure contains an axes. The axes contains an object of type bar. This object represents AA.

    The horizontal bar graph shows the Shapley values for all variables, sorted by their absolute values. Each Shapley value explains the deviation of the score for the query point from the average score of the predicted class, due to the corresponding variable.

    Train a regression model and create a shapley object. When you create a shapley object, if you do not specify a query point, then the software does not compute Shapley values. Use the object function fit to compute the Shapley values for the specified query point. Then create a bar graph of the Shapley values by using the object function plot.

    Load the carbig data set, which contains measurements of cars made in the 1970s and early 1980s.

    load carbig

    Create a table containing the predictor variables Acceleration, Cylinders, and so on, as well as the response variable MPG.

    tbl = table(Acceleration,Cylinders,Displacement,Horsepower,Model_Year,Weight,MPG);

    Removing missing values in a training set can help reduce memory consumption and speed up training for the fitrkernel function. Remove missing values in tbl.

    tbl = rmmissing(tbl);

    Train a blackbox model of MPG by using the fitrkernel function

    rng('default') % For reproducibility
    mdl = fitrkernel(tbl,'MPG','CategoricalPredictors',[2 5]);

    Create a shapley object. Specify the data set tbl, because mdl does not contain training data.

    explainer = shapley(mdl,tbl)
    explainer = 
      shapley with properties:
    
                BlackboxModel: [1x1 RegressionKernel]
                   QueryPoint: []
               BlackboxFitted: []
                ShapleyValues: []
                   NumSubsets: 64
                            X: [392x7 table]
        CategoricalPredictors: [2 5]
                       Method: 'interventional-kernel'
    
    

    explainer stores the training data tbl in the X property.

    Compute the Shapley values of all predictor variables for the first observation in tbl.

    queryPoint = tbl(1,:)
    queryPoint=1×7 table
        Acceleration    Cylinders    Displacement    Horsepower    Model_Year    Weight    MPG
        ____________    _________    ____________    __________    __________    ______    ___
    
             12             8            307            130            70         3504     18 
    
    
    explainer = fit(explainer,queryPoint);

    For a regression model, shapley computes Shapley values using the predicted response, and stores them in the ShapleyValues property. Display the values in the ShapleyValues property.

    explainer.ShapleyValues
    ans=6×2 table
          Predictor       ShapleyValue
        ______________    ____________
    
        "Acceleration"       -0.1561  
        "Cylinders"         -0.18306  
        "Displacement"      -0.34203  
        "Horsepower"        -0.27291  
        "Model_Year"         -0.2926  
        "Weight"            -0.32402  
    
    

    Display the predicted response for the query point, and plot the Shapley values for the query point by using the plot function. To display an existing underscore in any predictor name, change the TickLabelInterpreter value of the axes to 'none'.

    explainer.BlackboxFitted
    ans = 21.0495
    
    f = figure; 
    plot(explainer)
    f.CurrentAxes.TickLabelInterpreter = 'none';

    Figure contains an axes. The axes contains an object of type bar.

    The horizontal bar graph shows the Shapley values for all variables, sorted by their absolute values. Each Shapley value explains the deviation of the prediction for the query point from the average, due to the corresponding variable.

    Train a regression model and create a shapley object using a function handle to the predict function of the model. Use the object function fit to compute the Shapley values for the specified query point. Then plot the Shapley values by using the object function plot.

    Load the carbig data set, which contains measurements of cars made in the 1970s and early 1980s.

    load carbig

    Create a table containing the predictor variables Acceleration, Cylinders, and so on.

    tbl = table(Acceleration,Cylinders,Displacement,Horsepower,Model_Year,Weight);

    Train a blackbox model of MPG by using the TreeBagger function.

    rng('default') % For reproducibility
    Mdl = TreeBagger(100,tbl,MPG,'Method','regression','CategoricalPredictors',[2 5]);

    shapley does not support a TreeBagger object directly, so you cannot specify the first input argument (blackbox model) of shapley as a TreeBagger object. Instead, you can use a function handle to the predict function. You can also specify options of the predict function using name-value arguments of the function.

    Create the function handle to the predict function of the TreeBagger object Mdl. Specify the array of tree indices to use as 1:50.

    f = @(tbl) predict(Mdl,tbl,'Trees',1:50);

    Create a shapley object using the function handle f. When you specify a blackbox model as a function handle, you must provide the predictor data. tbl includes categorical predictors (Cylinder and Model_Year) with the double data type. By default, shapley does not treat variables with the double data type as categorical predictors. Specify the second (Cylinder) and fifth (Model_Year) variables as categorical predictors.

    explainer = shapley(f,tbl,'CategoricalPredictors',[2 5]);
    explainer = fit(explainer,tbl(1,:));

    Plot the Shapley values.

    plot(explainer)

    Figure contains an axes. The axes contains an object of type bar.

    Display the predictor names in order of importance.

    tbl.Properties.VariableNames([3 2 6 4 5 1])
    ans = 1x6 cell
      Columns 1 through 4
    
        {'Displacement'}    {'Cylinders'}    {'Weight'}    {'Horsepower'}
    
      Columns 5 through 6
    
        {'Model_Year'}    {'Acceleration'}
    
    

    More About

    expand all

    References

    [1] Lundberg, Scott M., and S. Lee. "A Unified Approach to Interpreting Model Predictions." Advances in Neural Information Processing Systems 30 (2017): 4765–774.

    [2] Aas, Kjersti, Martin. Jullum, and Anders Løland. "Explaining Individual Predictions When Features Are Dependent: More Accurate Approximations to Shapley Values." arXiv:1903.10464 (2019).

    Extended Capabilities

    Introduced in R2021a