evaluateAnomalyDetection

Evaluate anomaly detection results against ground truth

Since R2022b

collapse all in page

    Syntax

    metrics = evaluateAnomalyDetection(predLabels,gtLabels,anomalyLabels)
    metrics = evaluateAnomalyDetection(predLabels,gtLabels,anomalyLabels,Verbose=tf)

    Description

    metrics = evaluateAnomalyDetection(predLabels,gtLabels,anomalyLabels) calculates various metrics to evaluate the quality of the anomaly detection results predLabels against the ground truth gtLabels. anomalyLabels indicates which class labels in gtLabels belong to the anomaly (positive) class

    Note

    This functionality requires the Automated Visual Inspection Library for Computer Vision Toolbox™. You can install the Automated Visual Inspection Library for Computer Vision Toolbox from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons.

    example

    metrics = evaluateAnomalyDetection(predLabels,gtLabels,anomalyLabels,Verbose=tf) also displays the evaluation progress information in the command window.

    Examples

    collapse all

    This example uses:

    Open Live Script

    Load test images and corresponding labels, then create a datastore that reads the test data. The data set consists of grayscale images of handwritten digits 0–9.

    [Xtest,gtLabels] = digitTest4DArrayData;
dsTest = arrayDatastore(Xtest,IterationDimension=4);

    Load a pretrained FCDD anomaly detector. This detector has been trained to classify the digit 8 as normal and all other digits as anomalies. Therefore, specify the set of anomaly labels as the set of digits between 0 and 9, excluding 8.

    load("digit8AnomalyDetector.mat");
anomalyLabels = setdiff(string(0:9),"8");

    Classify each test image as normal or anomalous.

    predLabels = classify(detector,dsTest);

    Calculate metrics for the anomaly detection results.

    metrics = evaluateAnomalyDetection(predLabels,gtLabels,anomalyLabels)
    Evaluating anomaly detection results
------------------------------------
* Finalizing... Done.
* Data set metrics:

    GlobalAccuracy    MeanAccuracy    Precision    Recall    Specificity    F1Score    FalsePositiveRate    FalseNegativeRate
    ______________    ____________    _________    ______    ___________    _______    _________________    _________________

        0.7662           0.839         0.98971     0.748        0.93        0.85204          0.07                 0.252      

    metrics = 
  anomalyDetectionMetrics with properties:

              ConfusionMatrix: [2×2 table]
    NormalizedConfusionMatrix: [2×2 table]
               DataSetMetrics: [1×8 table]
                 ClassMetrics: [2×2 table]

    Display the confusion matrix.

    cm = metrics.ConfusionMatrix
    cm=2×2 table
               Normal    Anomaly
               ______    _______

    Normal       465        35  
    Anomaly     1134      3366

    Input Arguments

    collapse all

    Predicted labels, specified as a logical vector. The label value 0 (false) corresponds to the normal (negative) class. The value 1 (true) corresponds to the anomaly (positive) class. You can get the predicted labels using the classify function.

    Data Types: logical

    Ground truth labels for each image, specified as a numeric vector, logical vector, or categorical vector. The evaluateAnomalyDetection function converts the labels into a logical vector according to the set of anomaly labels in anomalyLabels.

    Anomaly labels, specified as a vector of the same data type as gtLabels. When gtLabels is categorical, anomalyLabels can be of data type string whose values correspond to categories in gtLabels.

    The evaluateAnomalyDetection function converts all ground truth labels in gtLabels that belong to the set of anomaly labels to a logical true, indicating an anomaly (positive detection). The function converts all other ground truth labels to a logical false, indicating normality (negative detection).

    Display evaluation progress information in the command window, specified as a numeric or logical 1 (true) or 0 (false). The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics.

    Data Types: logical

    Output Arguments

    collapse all

    Anomaly detection metrics, returned as an anomalyDetectionMetrics object.

    Version History

    Introduced in R2022b

    See Also

    Topics