Get Started with Anomaly Detection Using Deep Learning

R2026b

Anomaly detection using deep learning is an increasingly popular approach to automating visual inspection tasks. The goal of anomaly detection is to perform a binary classification of images as normal or anomalous. For example, in the manufacturing industry, anomaly detection can determine which manufactured units meet quality standards and which units are defective.

You can train an anomaly detector using semi-supervised training. A complete semi-supervised workflow consists of training a model on normal image data and determining an anomaly threshold that separates normal images from anomalous images. Anomaly detection techniques generally follow these steps:

Prepare training data and calibration data.
Train a model to produce anomaly scores.
Calibrate the model by determining an anomaly score threshold, and evaluate the performance of the model by classifying metrics and visualizing the classification results.
Classify test images using the trained model and the optimal anomaly threshold.
Deploy the model onto a production line.

Note

An alternative to anomaly detection is anomaly classification. Classification can differentiate between normal images and multiple known anomalous patterns. Classification can provide more insight on the type of anomaly, but it requires a greater amount of training data of the anomaly classes. For an example, see Classify Defects on Wafer Maps Using Deep Learning.

Prepare Training and Calibration Data

You can train an anomaly detector using one-class learning, which means that the network trains on data consisting only of normal images. Despite training on samples only of normal images, the model learns how to distinguish between normal and anomalous images. One-class learning offers many advantages for anomaly detection problems:

Representations of anomalies can be scarce.
Anomalies can represent expensive or catastrophic outcomes.
There can be many types of anomalies, and the types of anomalies can change over the lifetime of the model. Describing what a normal image looks like is often more feasible than providing data that represents all possible anomalies in real world settings.

You can use the Image Labeler app to interactively label images with scene labels. The app returns the ground truth labeled data as a groundTruth object. Convert the ground truth scene labels to an image datastore for training by using the sceneLabelTrainingData function. For more information, see Get Started with the Image Labeler.

In semi-supervised learning, you can tune the performance of the trained model using calibration data. The calibration data set consists of labeled samples of normal and anomalous images. Avoid overfitting the model by using different images in the training and calibration data sets. You can use functions such as splitAnomalyData to split a datastore into separate datastores for training and calibration.

Some techniques, such as outlier exposure, add a small number of anomalous images to the training data set. For example, the FCDD anomaly detector uses a small number of anomalous images for training. You can either provide anomalous images during training or let the FCDD anomaly detector generate anomalous data synthetically.

Choose the Anomaly Detection Model

The following table distinguishes the characteristics and performance of anomaly detection methods available through the Visual Inspection Toolbox™ during training and inference. The available anomaly detectors include fully convolutional data description (FCDD) [1], FastFlow [2], PatchCore [3], and Student-Teacher [4]. Choose an anomaly detector based on the requirements of your workflow.

Consider the training requirements.

Available Anomaly Detection Models: Training

Training Aspect	Student-Teacher	PatchCore	FastFlow	FCDD
Input image size	Medium to large images	Small to medium preferred (due to memory limitation for large images)	Small to medium preferred (due to memory limitation for large images)	Small to large (high-resolution images)
Model size	Moderate to large	Moderate to large (can vary with value of the compression ratio)	Moderate to large	Small (lightest weight model)
Base feature extraction network	Pretrained small or medium Teacher network	Flexible (supports pretrained ResNet-18 or ResNet-50, or custom pretrained network)	Flexible (supports custom pretrained network)	Flexible (supports custom pretrained network)
Performance speed	Generally faster than Patchcore and FastFlow	Fast	Fast	Fastest
Low-shot training regime	Not supported	Supported	Not supported	Not supported

Consider the inference and testing requirements.

Available Anomaly Detection Models: Inference

Inference Aspect	Student-Teacher	PatchCore	FastFlow	FCDD
Different image size at training and inference	Supported – performance is best when the inference image and training image have the same resolution even if image sizes are different	Not supported – image size at inference must match image size at training	Supported	Supported
Performance speed	Fast	Medium to fast	Fast	Fast

Train the Model

The Visual Inspection Toolbox provides a framework for training anomaly detection networks. Start by creating an untrained network object. Then, train the network by passing the network and the training data to the appropriate training function. The table shows the supported networks and training functions.

Network Description	Network Object	Training Function
Fully convolutional data description (FCDD) [1]	`fcddAnomalyDetector`	`trainFCDDAnomalyDetector`
FastFlow [2]	`fastFlowAnomalyDetector`	`trainFastFlowAnomalyDetector`
PatchCore [3]	`patchCoreAnomalyDetector`	`trainPatchCoreAnomalyDetector`
Student-Teacher [4]	`studentTeacherAnomalyDetector`	`trainStudentTeacherAnomalyDetector`

If you want to train an anomaly detection network that uses a different framework, then design a model that accepts images as input and returns scalar valued anomaly scores. The score represents the likelihood that an image is anomalous. Larger scores indicate a higher probability that an image in anomalous samples. Optionally, you can design a network that returns localized anomaly scores in addition to a scalar aggregate score for the image. Localized score information enables you to subjectively understand the network classification decisions.

Calibrate and Evaluate the Model

Calibrate the model by determining an anomaly score threshold that separates the scores of good samples from the scores of anomaly samples. Unfortunately, most data sets cannot be cleanly separated into these two categories, and will have misclassifications in the form of false positives (normal images classified as anomalous) and false negatives (anomalous images classified as normal). By adjusting the threshold, you can tune the false positive and false negative rates to satisfy your operating requirements.

To select a threshold, you can use the anomalyThreshold function. The function supports different optimization methods based on statistical techniques such as receiver operating characteristic (ROC) and precision-recall (PR) curves. The function optionally returns the performance metrics as an rocmetrics (Deep Learning Toolbox) function, which you can use to plot the ROC and PR curves.

Use the evaluateAnomalyDetection function to evaluate the quality of the anomaly detection results using metrics such as the confusion matrix and average precision. The anomalyDetectionMetrics object stores the metrics.

Normalize and Visualize Anomaly Maps

You can visualize the localized anomaly scores to understand the classification results using available functionality.

Display an anomaly score map as a heatmap overlaid on a calibration or test image using the anomalyMapOverlay function.
Normalize anomaly score maps using the percentileNormalizer object and the corresponding normalize function.
Display an interactive figure that displays the classification results and score maps for an entire data set using the viewAnomalyDetectionResults function.

Visualizing localized anomaly scores is useful for identifying patterns in false negatives and false positives. The patterns can indicate strategies to further improve the classifier performance.

Perform Classification

Classify test images as normal or anomalous by using the classify function. You can get the unnormalized anomaly scores for the test images using the predict function. You can also get the per-pixel anomaly score map using the anomalyMap function.

Note

For supervised defect classification workflows, fine-tune pretrained deep learning networks to classify parts or defect types. Use visionTransformer, or adapt networks such as ResNet or EfficientNet by using trainnet (Deep Learning Toolbox) and imagePretrainedNetwork (Deep Learning Toolbox).

Deploy the Model

The end goal of visual inspection is often to deploy trained models in real-time production systems on assembly lines. MATLAB^® Coder™ and GPU Coder™ enable deployment of visual inspection code, models, and applications to CPU and NVIDIA^® GPU targets. You can also export supported models to ONNX format for deployment in other environments. For more information, see Deploy Visual Inspection Code, Models, and Applications.

For examples, see:

References

[1] Liznerski, Philipp, Lukas Ruff, Robert A. Vandermeulen, Billy Joe Franks, Marius Kloft, and Klaus-Robert Müller. "Explainable Deep One-Class Classification." arXiv, March 18, 2021. https://arxiv.org/abs/2007.01760.

[2] Yu, Jiawei, Ye Zheng, Xiang Wang, Wei Li, Yushuang Wu, Rui Zhao, and Liwei Wu. "FastFlow: Unsupervised Anomaly Detection and Localization via 2D Normalizing Flows." arXiv, November 16, 2021. https://doi.org/10.48550/arXiv.2111.07677.

[3] Roth, Karsten, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. "Towards Total Recall in Industrial Anomaly Detection." arXiv, May 5, 2022. https://arxiv.org/abs/2106.08265.

[4] Li, Wenqiao, Bozhong Zheng, Xiaohao Xu, Jinye Gan, Fading Lu, Xiang Li, and Na Ni. "Multi-Sensor Object Anomaly Detection: Unifying Appearance, Geometry, and Internal Properties." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9984–9993. 2025.