Metrics Handlers

This example shows how to manage Modelscape™ test metrics and their associated threshold objects using MetricsHandler objects.

MetricsHandler produces reports that summarize the metrics and the status of the metrics in the container relative to their thresholds.

For more information about test metrics and thresholds, see Credit Scorecard Validation Metrics and Fairness Metrics in Modelscape. To learn how to write your own metrics, see Test Metrics in Modelscape.

This example shows you how to set up some metrics and thresholds for mock data of a credit scoring model. This example creates a metrics handler object to visualize the metrics and summarize the results. It shows you how to set an overall status to the handler based on various metrics.

Set Up Test Metrics and Thresholds

Use the following random data as an example of training response data (defaultIndicators) and model predictions (scores).

rng('default');
scores = rand(1000,1);
defaultIndicators = double(scores + rand(1000,1) < 1);

Create three metrics:

1) Area under the receiver operating characteristic curve (AUROC),

2) Cumulative accuracy profile (CAP Accuracy) ratio, and

3) Kolmogorov-Smirnov statistic.

For AUROC and CAP Accuracy ratio, set values greater than 0.8 as Pass, values less than 0.7 as Fail, and values between these as Undecided, requiring further inspection. Set no thresholds for the Kolmogorov-Smirnov statistic.

import mrm.data.validation.TestThresholds
import mrm.data.validation.pd.*

auroc = AUROC(defaultIndicators, scores);
aurocThresholds = TestThresholds([0.7, 0.8], ["Fail", "Undecided", "Pass"]);

cap = CAPAccuracyRatio(defaultIndicators, scores);
capThresholds = TestThresholds([0.6, 0.7], ["Fail", "Undecided", "Pass"]);

ks = KSStatistic(defaultIndicators, scores);

Add metrics to a Metrics Handler Object

Add the metrics created in the previous section to a MetricsHandler object.

import mrm.data.validation.MetricsHandler
mh = MetricsHandler();
append(mh, auroc, aurocThresholds);
append(mh, cap, capThresholds);
append(mh, ks);
disp(mh)

  MetricsHandler with properties:

      CAP: [1x1 mrm.data.validation.pd.CAPAccuracyRatio]
    AUROC: [1x1 mrm.data.validation.pd.AUROC]
       KS: [1x1 mrm.data.validation.pd.KSStatistic]

The handler contains these three metrics that can be accessed as properties of this handler object. This allows you to access constituent metrics' diagnostics and visualizations.

visualize(mh.AUROC);

Interrogate Metrics Handlers

Use the report method to view the performance of the model relative to the given metrics.

summaryTable = report(mh);
disp(summaryTable)

               Metric                Value       Status       Diagnostic 
    ____________________________    _______    ___________    ___________

    Area under ROC curve            0.82905    Pass           (0.8, Inf) 
    Accuracy ratio                  0.65809    Undecided      (0.6, 0.7] 
    Kolmogorov-Smirnov statistic    0.51462    <undefined>    <undefined>

The model performs well on AUROC, whereas the "Undecided" status on the Accuracy Ratio suggests the model requires a closer look.

When the handler carries complex non-scalar metrics, use arguments Keys and Metrics arguments with report. For more information, see Fairness Metrics in Modelscape.

Set Overall Status to the Handler

For a handler with many metrics, set an overall status to the handler by associating a 'status interpreter' to the handler. This section shows how to use an interpreter supplied with Modelscape that is compatible with your threshold objects.

mh.StatusInterpreter = @mrm.data.validation.overallStatus;
summaryTable = report(mh);
disp(summaryTable)

               Metric                Value       Status       Diagnostic 
    ____________________________    _______    ___________    ___________

    Area under ROC curve            0.82905    Pass           (0.8, Inf) 
    Accuracy ratio                  0.65809    Undecided      (0.6, 0.7] 
    Kolmogorov-Smirnov statistic    0.51462    <undefined>    <undefined>
    Overall                             NaN    Undecided      <undefined>

The overall status is in general decided based on the status descriptions of the individual metrics. In the above case, the overall status is the 'worst' of the individual statuses - 'Undecided'.

Thresholding systems with other descriptive strings - for example "Red", "Amber", "Green" require a custom status interpreter to be implemented. To do this, see the instructions before the StatusInterpreter declaration in the MetricsHandler implementation.

edit mrm.data.validation.MetricsHandler

Alternatively, modify the interpreter as required.

edit mrm.data.validation.overallStatus

You can also set the StatusInterpreter for the handler immediately at construction:

mh2 = MetricsHandler('StatusInterpreter', @mrm.data.validation.overallStatus)

×

Abrir ejemplo

Tiene una versión modificada de este ejemplo. ¿Desea abrir este ejemplo con sus modificaciones?

No, sobrescribir la versión modificada Sí