Example of using Self attention layer in MATLAB R2023A

IN MATLAB 2023A, self-attention layer is intorduced.

can an example is provided to use it in image classication tasks?

[Mohamed Boushaki]

Interested!

[Kuo]

Same question, can there be an example about time series forecasting? Thanks !!

Answer 1

[Himanshu]

Hi Mahmoud,

I understand that you want to use "selfAttentionLayer" for image classification task in MATLAB.

A self-attention layer computes single-head or multihead self-attention of its input. For the following example, we will be using the "DigitDataset" in MATLAB.

% load digit dataset
digitDatasetPath = fullfile(matlabroot, 'toolbox', 'nnet', 'nndemos', 'nndatasets', 'DigitDataset');
imds = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders', true, 'LabelSource', 'foldernames');
[imdsTrain, imdsValidation] = splitEachLabel(imds, 0.7, 'randomized');

% define network architecture
layers = [
    imageInputLayer([28 28 1], 'Name', 'input')
    convolution2dLayer(3, 32, 'Padding', 'same', 'Name', 'conv1')
    batchNormalizationLayer('Name', 'bn1')
    reluLayer('Name', 'relu1')
    maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxpool1')
    convolution2dLayer(3, 64, 'Padding', 'same', 'Name', 'conv2')
    batchNormalizationLayer('Name', 'bn2')
    reluLayer('Name', 'relu2')
    maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxpool2')
    flattenLayer('Name', 'flatten')
    selfAttentionLayer(8, 64, 'Name', 'self_attention')
    fullyConnectedLayer(10, 'Name', 'fc')
    softmaxLayer('Name', 'softmax')
    classificationLayer('Name', 'output')]

% set training options
options = trainingOptions('sgdm', ...
    'InitialLearnRate', 0.01, ...
    'MaxEpochs', 5, ...
    'Shuffle', 'every-epoch', ...
    'ValidationData', imdsValidation, ...
    'ValidationFrequency', 30, ...
    'Verbose', false, ...
    'Plots', 'training-progress')

% training the network
net = trainNetwork(imdsTrain, layers, options);

Training Output:

In this code, the selfAttentionLayer is used to processes 28x28 grayscale images. The self-attention mechanism helps the model capture long-range dependencies in the input data, meaning it can learn to relate different parts of the image to each other. By introducing the selfAttentionLayer after a series of convolutional and pooling layers, the model can enhance its feature representation capabilities by considering spatial relationships between different regions of the input image.

You can refer to the below documentation to understand more about creating and training a simple convolutional neural network for deep learning classification.

Create Simple Deep Learning Neural Network for Classification - MATLAB & Simulink Example - MathWorks India

Self-attention layer - MATLAB - MathWorks India

[Mohamed Boushaki]

Dear Himanshu,

Many thanks for your help, could we use the self attention for the case of 1-D input signal (sequenceInputLayer)?

Regards

Mohamed

[Chih]

In https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.selfattentionlayer.html, it states that:

A self-attention layer computes single-head or multihead self-attention of its input.

The layer:

1. Computes the queries, keys, and values from the input
2. Computes the scaled dot-product attention across heads using the queries, keys, and values
3. Merges the results from the heads
4. Performs a linear transformation on the merged result

I wonder if the layer also apply softmax to the scaling (i.e. divide (Q*K) by sqrt(dim))? My understanding is that, within step 2, this softmax and scaling should happen.

Please clarify that for me or more general users.

Thanks.

[Muhammad Shoaib]

can an example is provided to use it in time sequence data? Means to select the more valuable features from the input. and also how to use it with LSTM.

[xingxingcui]

@Muhammad Shoaib ,@Himanshu I have tryed use selfAttentionLayer in time sequence data in R2023b,but faild! please see follow link, is there any idea?

https://ww2.mathworks.cn/matlabcentral/answers/2066601-selfattentionlayer-can-t-process-sequence-to-label-problem

[DGM]

Posted as a comment-as-flag by chang gao:

Useful answer.

[Philip Brown]

For time series data, you could take a look at this recent blog post and GitHub repo. That uses a transformer network containing selfAttentionLayer for time series prediction. The use case there is finance, but the DL techniques would be generally applicable.

[Tian,HCong]

This answer is very helpful to me, but if it is an RGB image, how should I adjust the program? Can you give me some guidance?