Main Content

Define Custom Deep Learning Layer with Learnable Parameters

If Deep Learning Toolbox™ does not provide the layer you require for your task, then you can define your own custom layer using this example as a guide. For a list of built-in layers, see List of Deep Learning Layers.

To define a custom deep learning layer, you can use the template provided in this example, which takes you through the following steps:

  1. Name the layer — Give the layer a name so that you can use it in MATLAB®.

  2. Declare the layer properties — Specify the properties of the layer including learnable parameters and state parameters.

  3. Create a constructor function (optional) — Specify how to construct the layer and initialize its properties. If you do not specify a constructor function, then at creation, the software initializes the Name, Description, and Type properties with [] and sets the number of layer inputs and outputs to 1.

  4. Create forward functions — Specify how data passes forward through the layer (forward propagation) at prediction time and at training time.

  5. Create reset state function (optional) — Specify how to reset state parameters.

  6. Create a backward function (optional) — Specify the derivatives of the loss with respect to the input data and the learnable parameters (backward propagation). If you do not specify a backward function, then the forward functions must support dlarray objects.

When defining the layer functions, you can use dlarray objects. Using dlarray objects makes working with high dimensional data easier by allowing you to label the dimensions. For example, you can label which dimensions correspond to spatial, time, channel, and batch dimensions using the "S", "T", "C", and "B" labels, respectively. For unspecified and other dimensions, use the "U" label. For dlarray object functions that operate over particular dimensions, you can specify the dimension labels by formatting the dlarray object directly, or by using the DataFormat option.

Using formatted dlarray objects in custom layers also allows you to define layers where the inputs and outputs have different formats, such as layers that permute, add, or remove dimensions. For example, you can define a layer that takes as input a mini-batch of images with format "SSCB" (spatial, spatial, channel, batch) and output a mini-batch of sequences with format "CBT" (channel, batch, time). Using formatted dlarray objects also allows you to define layers that can operate on data with different input formats, for example, layers that support inputs with formats "SSCB" (spatial, spatial, channel, batch) and "CBT" (channel, batch, time).

dlarray objects also enable support for automatic differentiation. This means that if your forward functions fully support dlarray objects, then defining the backward function is optional.

To enable support for using formatted dlarray objects in custom layer forward functions, also inherit from the nnet.layer.Formattable class when defining the custom layer. For an example, see Define Custom Deep Learning Layer with Formatted Inputs.

This example shows how to create a PReLU layer, which is a layer with a learnable parameter and use it in a convolutional neural network. A PReLU layer performs a threshold operation, where for each channel, any input value less than zero is multiplied by a scalar learned at training time.[1] For values less than zero, a PReLU layer applies scaling coefficients αi to each channel of the input. These coefficients form a learnable parameter, which the layer learns during training.

This figure from [1] compares the ReLU and PReLU layer functions.

Intermediate Layer Template

Copy the intermediate layer template into a new file in MATLAB. This template outlines the structure of an intermediate layer class definition. It outlines:

  • The optional properties blocks for the layer properties, learnable parameters, and state parameters.

  • The layer constructor function.

  • The predict function and the optional forward function.

  • The optional resetState function for layers with state properties.

  • The optional backward function.

classdef myLayer < nnet.layer.Layer % & nnet.layer.Formattable (Optional)

    properties
        % (Optional) Layer properties.

        % Declare layer properties here.
    end

    properties (Learnable)
        % (Optional) Layer learnable parameters.

        % Declare learnable parameters here.
    end

    properties (State)
        % (Optional) Layer state parameters.

        % Declare state parameters here.
    end

    properties (Learnable, State)
        % (Optional) Nested dlnetwork objects with both learnable
        % parameters and state.

        % Declare nested networks with learnable and state parameters here.
    end

    methods
        function layer = myLayer()
            % (Optional) Create a myLayer.
            % This function must have the same name as the class.

            % Define layer constructor function here.
        end

        function [Z,state] = predict(layer,X)
            % Forward input data through the layer at prediction time and
            % output the result and updated state.
            %
            % Inputs:
            %         layer - Layer to forward propagate through 
            %         X     - Input data
            % Outputs:
            %         Z     - Output of layer forward function
            %         state - (Optional) Updated layer state.
            %
            %  - For layers with multiple inputs, replace X with X1,...,XN, 
            %    where N is the number of inputs.
            %  - For layers with multiple outputs, replace Z with 
            %    Z1,...,ZM, where M is the number of outputs.
            %  - For layers with multiple state parameters, replace state 
            %    with state1,...,stateK, where K is the number of state 
            %    parameters.

            % Define layer predict function here.
        end

        function [Z,state,memory] = forward(layer,X)
            % (Optional) Forward input data through the layer at training
            % time and output the result, updated state, and a memory
            % value.
            %
            % Inputs:
            %         layer - Layer to forward propagate through 
            %         X     - Layer input data
            % Outputs:
            %         Z      - Output of layer forward function 
            %         state  - (Optional) Updated layer state 
            %         memory - (Optional) Memory value for custom backward
            %                  function
            %
            %  - For layers with multiple inputs, replace X with X1,...,XN, 
            %    where N is the number of inputs.
            %  - For layers with multiple outputs, replace Z with 
            %    Z1,...,ZM, where M is the number of outputs.
            %  - For layers with multiple state parameters, replace state 
            %    with state1,...,stateK, where K is the number of state 
            %    parameters.

            % Define layer forward function here.
        end

        function layer = resetState(layer)
            % (Optional) Reset layer state.

            % Define reset state function here.
        end

        function [dLdX,dLdW,dLdSin] = backward(layer,X,Z,dLdZ,dLdSout,memory)
            % (Optional) Backward propagate the derivative of the loss
            % function through the layer.
            %
            % Inputs:
            %         layer   - Layer to backward propagate through 
            %         X       - Layer input data 
            %         Z       - Layer output data 
            %         dLdZ    - Derivative of loss with respect to layer 
            %                   output
            %         dLdSout - (Optional) Derivative of loss with respect 
            %                   to state output
            %         memory  - Memory value from forward function
            % Outputs:
            %         dLdX   - Derivative of loss with respect to layer input
            %         dLdW   - (Optional) Derivative of loss with respect to
            %                  learnable parameter 
            %         dLdSin - (Optional) Derivative of loss with respect to 
            %                  state input
            %
            %  - For layers with state parameters, the backward syntax must
            %    include both dLdSout and dLdSin, or neither.
            %  - For layers with multiple inputs, replace X and dLdX with
            %    X1,...,XN and dLdX1,...,dLdXN, respectively, where N is
            %    the number of inputs.
            %  - For layers with multiple outputs, replace Z and dlZ with
            %    Z1,...,ZM and dLdZ,...,dLdZM, respectively, where M is the
            %    number of outputs.
            %  - For layers with multiple learnable parameters, replace 
            %    dLdW with dLdW1,...,dLdWP, where P is the number of 
            %    learnable parameters.
            %  - For layers with multiple state parameters, replace dLdSin
            %    and dLdSout with dLdSin1,...,dLdSinK and 
            %    dLdSout1,...dldSoutK, respectively, where K is the number
            %    of state parameters.

            % Define layer backward function here.
        end
    end
end

Name Layer and Specify Superclasses

First, give the layer a name. In the first line of the class file, replace the existing name myLayer with preluLayer.

classdef preluLayer < nnet.layer.Layer % & nnet.layer.Formattable (Optional)
    ...
end

If you do not specify a backward function, then the layer functions, by default, receive unformatted dlarray objects as input. To specify that the layer receives formatted dlarray objects as input and also outputs formatted dlarray objects, also inherit from the nnet.layer.Formattable class when defining the custom layer.

The layer does not require formattable inputs, so remove the optional nnet.layer.Formattable superclass.

classdef preluLayer < nnet.layer.Layer
    ...
end

Next, rename the myLayer constructor function (the first function in the methods section) so that it has the same name as the layer.

    methods
        function layer = preluLayer()           
            ...
        end

        ...
     end

Save the Layer

Save the layer class file in a new file named preluLayer.m. The file name must match the layer name. To use the layer, you must save the file in the current folder or in a folder on the MATLAB path.

Declare Properties and Learnable Parameters

Declare the layer properties in the properties section and declare learnable parameters by listing them in the properties (Learnable) section.

By default, custom intermediate layers have these properties. Do not declare these properties in the properties section.

PropertyDescription
NameLayer name, specified as a character vector or a string scalar. For Layer array input, the trainNetwork, assembleNetwork, layerGraph, and dlnetwork functions automatically assign names to layers with Name set to ''.
Description

One-line description of the layer, specified as a string scalar or a character vector. This description appears when the layer is displayed in a Layer array.

If you do not specify a layer description, then the software displays the layer class name.

Type

Type of the layer, specified as a character vector or a string scalar. The value of Type appears when the layer is displayed in a Layer array.

If you do not specify a layer type, then the software displays the layer class name.

NumInputsNumber of inputs of the layer, specified as a positive integer. If you do not specify this value, then the software automatically sets NumInputs to the number of names in InputNames. The default value is 1.
InputNamesInput names of the layer, specified as a cell array of character vectors. If you do not specify this value and NumInputs is greater than 1, then the software automatically sets InputNames to {'in1',...,'inN'}, where N is equal to NumInputs. The default value is {'in'}.
NumOutputsNumber of outputs of the layer, specified as a positive integer. If you do not specify this value, then the software automatically sets NumOutputs to the number of names in OutputNames. The default value is 1.
OutputNamesOutput names of the layer, specified as a cell array of character vectors. If you do not specify this value and NumOutputs is greater than 1, then the software automatically sets OutputNames to {'out1',...,'outM'}, where M is equal to NumOutputs. The default value is {'out'}.

If the layer has no other properties, then you can omit the properties section.

Tip

If you are creating a layer with multiple inputs, then you must set either the NumInputs or InputNames properties in the layer constructor. If you are creating a layer with multiple outputs, then you must set either the NumOutputs or OutputNames properties in the layer constructor. For an example, see Define Custom Deep Learning Layer with Multiple Inputs.

A PReLU layer does not require any additional properties, so you can remove the properties section.

A PReLU layer has only one learnable parameter, the scaling coefficient a. Declare this learnable parameter in the properties (Learnable) section and call the parameter Alpha.

    properties (Learnable)
        % Layer learnable parameters
            
        % Scaling coefficient
        Alpha
    end

Create Constructor Function

Create the function that constructs the layer and initializes the layer properties. Specify any variables required to create the layer as inputs to the constructor function.

The PReLU layer constructor function requires one input argument (the number of channels) and one optional argument (the layer name). The number of channels specifies the size of the learnable parameter Alpha. Specify two input arguments named numChannels and args in the preluLayer function that correspond to the number of channels and optional input arguments, respectively. Add a comment to the top of the function that explains the syntax of the function.

        function layer = preluLayer(numChannels,args)
            % layer = preluLayer(numChannels) creates a PReLU layer
            % with numChannels channels.
            %
            % layer = preluLayer(numChannels,Name=name) also specifies the
            % layer name

            ...
        end

Initialize Layer Properties

Initialize the layer properties, including learnable parameters in the constructor function. Replace the comment % Layer constructor function goes here with code that initializes the layer properties.

Parse the input arguments using an arguments block and set the Name property.

            arguments
                numChannels
                args.Name = "";
            end

            % Set layer name.
            layer.Name = args.Name;

Give the layer a one-line description by setting the Description property of the layer. Set the description to describe the type of layer and its size.

            % Set layer description.
            layer.Description = "PReLU with " + numChannels + " channels";

For a PReLU layer, when the input values are negative, the layer multiplies each channel of the input by the corresponding channel of Alpha. Initialize the learnable parameter Alpha to be a random vector of size 1-by-1-by-numChannels. With the third dimension specified as size numChannels, the layer can use element-wise multiplication of the input in the forward function. Alpha is a property of the layer object, so you must assign the vector to layer.Alpha.

            % Initialize scaling coefficient.
            layer.Alpha = rand([1 1 numChannels]);

View the completed constructor function.

        function layer = preluLayer(numChannels,args) 
            % layer = preluLayer(numChannels) creates a PReLU layer
            % with numChannels channels.
            %
            % layer = preluLayer(numChannels,Name=name) also specifies the
            % layer name.
    
            arguments
                numChannels
                args.Name = "";
            end
    
            % Set layer name.
            layer.Name = name;

            % Set layer description.
            layer.Description = "PReLU with " + numChannels + " channels";
        
            % Initialize scaling coefficient.
            layer.Alpha = rand([1 1 numChannels]); 
        end

With this constructor function, the command preluLayer(3,Name="prelu") creates a PReLU layer with three channels and the name "prelu".

Create Forward Functions

Create the layer forward functions to use at prediction time and training time.

Create a function named predict that propagates the data forward through the layer at prediction time and outputs the result.

The predict function syntax depends on the type of layer.

  • Z = predict(layer,X) forwards the input data X through the layer and outputs the result Z, where layer has a single input, a single output.

  • [Z,state] = predict(layer,X) also outputs the updated state parameter state, where layer has a single state parameter.

You can adjust the syntaxes for layers with multiple inputs, multiple outputs, or multiple state parameters:

  • For layers with multiple inputs, replace X with X1,...,XN, where N is the number of inputs. The NumInputs property must match N.

  • For layers with multiple outputs, replace Z with Z1,...,ZM, where M is the number of outputs. The NumOutputs property must match M.

  • For layers with multiple state parameters, replace state with state1,...,stateK, where K is the number of state parameters.

Tip

If the number of inputs to the layer can vary, then use varargin instead of X1,…,XN. In this case, varargin is a cell array of the inputs, where varargin{i} corresponds to Xi.

If the number of outputs can vary, then use varargout instead of Z1,…,ZN. In this case, varargout is a cell array of the outputs, where varargout{j} corresponds to Zj.

Tip

If the custom layer has a dlnetwork object for a learnable parameter, then in the predict function of the custom layer, use the predict function for the dlnetwork. Using the dlnetwork object predict function ensures that the software uses the correct layer operations for prediction.

Because a PReLU layer has only one input and one output, the syntax for predict for a PReLU layer is Z = predict(layer,X).

By default, the layer uses predict as the forward function at training time. To use a different forward function at training time, or retain a value required for a custom backward function, you must also create a function named forward.

The dimensions of the inputs depend on the type of data and the output of the connected layers:

Layer InputInput SizeObservation Dimension
Feature vectorsc-by-N, where c corresponds to the number of channels and N is the number of observations.2
2-D imagesh-by-w-by-c-by-N, where h, w, and c correspond to the height, width, and number of channels of the images, respectively, and N is the number of observations.4
3-D imagesh-by-w-by-d-by-c-by-N, where h, w, d, and c correspond to the height, width, depth, and number of channels of the 3-D images, respectively, and N is the number of observations.5
Vector sequencesc-by-N-by-S, where c is the number of features of the sequences, N is the number of observations, and S is the sequence length.2
2-D image sequencesh-by-w-by-c-by-N-by-S, where h, w, and c correspond to the height, width, and number of channels of the images, respectively, N is the number of observations, and S is the sequence length.4
3-D image sequencesh-by-w-by-d-by-c-by-N-by-S, where h, w, d, and c correspond to the height, width, depth, and number of channels of the 3-D images, respectively, N is the number of observations, and S is the sequence length.5

For layers that output sequences, the layers can output sequences of any length or output data with no time dimension. Note that when training a network that outputs sequences using the trainNetwork function, the lengths of the input and output sequences must match.

The forward function propagates the data forward through the layer at training time and also outputs a memory value.

The forward function syntax depends on the type of layer:

  • Z = forward(layer,X) forwards the input data X through the layer and outputs the result Z, where layer has a single input, a single output.

  • [Z,state] = forward(layer,X) also outputs the updated state parameter state, where layer has a single state parameter.

  • [__,memory] = forward(layer,X) also returns a memory value for a custom backward function using any of the previous syntaxes. If the layer has both a custom forward function and a custom backward function, then the forward function must return a memory value.

You can adjust the syntaxes for layers with multiple inputs, multiple outputs, or multiple state parameters:

  • For layers with multiple inputs, replace X with X1,...,XN, where N is the number of inputs. The NumInputs property must match N.

  • For layers with multiple outputs, replace Z with Z1,...,ZM, where M is the number of outputs. The NumOutputs property must match M.

  • For layers with multiple state parameters, replace state with state1,...,stateK, where K is the number of state parameters.

Tip

If the number of inputs to the layer can vary, then use varargin instead of X1,…,XN. In this case, varargin is a cell array of the inputs, where varargin{i} corresponds to Xi.

If the number of outputs can vary, then use varargout instead of Z1,…,ZN. In this case, varargout is a cell array of the outputs, where varargout{j} corresponds to Zj.

Tip

If the custom layer has a dlnetwork object for a learnable parameter, then in the forward function of the custom layer, use the forward function of the dlnetwork object. Using the dlnetwork object forward function ensures that the software uses the correct layer operations for training.

The PReLU operation is given by

f(xi)={xiif xi>0αixiif xi0

where xi is the input of the nonlinear activation f on channel i, and αi is the coefficient controlling the slope of the negative part. The subscript i in αi indicates that the nonlinear activation can vary on different channels.

Implement this operation in predict. In predict, the input X corresponds to x in the equation. The output Z corresponds to f(xi). The PReLU layer does not require memory or a different forward function for training, so you can remove the forward function from the class file. Add a comment to the top of the function that explains the syntaxes of the function.

Tip

If you preallocate arrays using functions such as zeros, then you must ensure that the data types of these arrays are consistent with the layer function inputs. To create an array of zeros of the same data type as another array, use the "like" option of zeros. For example, to initialize an array of zeros of size sz with the same data type as the array X, use Z = zeros(sz,"like",X).

        function Z = predict(layer, X)
            % Z = predict(layer, X) forwards the input data X through the
            % layer and outputs the result Z.
            
            Z = max(X,0) + layer.Alpha .* min(0,X);
        end

Because the predict function only uses functions that support dlarray objects, defining the backward function is optional. For a list of functions that support dlarray objects, see List of Functions with dlarray Support.

Completed Layer

View the completed layer class file.

classdef preluLayer < nnet.layer.Layer
    % Example custom PReLU layer.

    properties (Learnable)
        % Layer learnable parameters
            
        % Scaling coefficient
        Alpha
    end
    
    methods
        function layer = preluLayer(numChannels,args) 
            % layer = preluLayer(numChannels) creates a PReLU layer
            % with numChannels channels.
            %
            % layer = preluLayer(numChannels,Name=name) also specifies the
            % layer name.
    
            arguments
                numChannels
                args.Name = "";
            end
    
            % Set layer name.
            layer.Name = name;

            % Set layer description.
            layer.Description = "PReLU with " + numChannels + " channels";
        
            % Initialize scaling coefficient.
            layer.Alpha = rand([1 1 numChannels]); 
        end
        
        function Z = predict(layer, X)
            % Z = predict(layer, X) forwards the input data X through the
            % layer and outputs the result Z.
            
            Z = max(X,0) + layer.Alpha .* min(0,X);
        end
    end
end

GPU Compatibility

If the layer forward functions fully support dlarray objects, then the layer is GPU compatible. Otherwise, to be GPU compatible, the layer functions must support inputs and return outputs of type gpuArray (Parallel Computing Toolbox).

Many MATLAB built-in functions support gpuArray (Parallel Computing Toolbox) and dlarray input arguments. For a list of functions that support dlarray objects, see List of Functions with dlarray Support. For a list of functions that execute on a GPU, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox). To use a GPU for deep learning, you must also have a supported GPU device. For information on supported devices, see GPU Support by Release (Parallel Computing Toolbox). For more information on working with GPUs in MATLAB, see GPU Computing in MATLAB (Parallel Computing Toolbox).

In this example, the MATLAB functions used in predict all support dlarray objects, so the layer is GPU compatible.

Check Validity of Custom Layer Using checkLayer

Check the layer validity of the custom layer preluLayer.

The custom layer preluLayer, attached to this is example as a supporting file, applies the PReLU operation to the input data. To access this layer, open this example as a live script.

Create an instance of the layer and check its validity using checkLayer. Specify the valid input size to be the size of a single observation of typical input to the layer. The layer expects 4-D array inputs, where the first three dimensions correspond to the height, width, and number of channels of the previous layer output, and the fourth dimension corresponds to the observations.

Specify the typical size of the input of an observation and set the ObservationDimension option to 4.

layer = preluLayer(20);
validInputSize = [24 24 20];
checkLayer(layer,validInputSize,ObservationDimension=4)
Skipping GPU tests. No compatible GPU device found.
 
Skipping code generation compatibility tests. To check validity of the layer for code generation, specify the 'CheckCodegenCompatibility' and 'ObservationDimension' options.
 
Running nnet.checklayer.TestLayerWithoutBackward
.......... ........
Done nnet.checklayer.TestLayerWithoutBackward
__________

Test Summary:
	 18 Passed, 0 Failed, 0 Incomplete, 10 Skipped.
	 Time elapsed: 1.9312 seconds.

Here, the function does not detect any issues with the layer.

Include Custom Layer in Network

You can use a custom layer in the same way as any other layer in Deep Learning Toolbox. This section shows how to create and train a network for digit classification using the PReLU layer you created earlier.

Load the example training data.

[XTrain,YTrain] = digitTrain4DArrayData;

Create a layer array containing the custom layer preluLayer, attached to this is example as a supporting file. To access this layer, open this example as a live script.

layers = [ 
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    batchNormalizationLayer
    preluLayer(20)
    fullyConnectedLayer(10)
    softmaxLayer
    classificationLayer];

Set the training options and train the network.

options = trainingOptions("adam",MaxEpochs=10);
net = trainNetwork(XTrain,YTrain,layers,options);
Training on single CPU.
Initializing input data normalization.
|========================================================================================|
|  Epoch  |  Iteration  |  Time Elapsed  |  Mini-batch  |  Mini-batch  |  Base Learning  |
|         |             |   (hh:mm:ss)   |   Accuracy   |     Loss     |      Rate       |
|========================================================================================|
|       1 |           1 |       00:00:00 |       10.94% |       3.0526 |          0.0010 |
|       2 |          50 |       00:00:08 |       71.88% |       0.8377 |          0.0010 |
|       3 |         100 |       00:00:16 |       85.94% |       0.4878 |          0.0010 |
|       4 |         150 |       00:00:23 |       88.28% |       0.4068 |          0.0010 |
|       6 |         200 |       00:00:32 |       96.09% |       0.1691 |          0.0010 |
|       7 |         250 |       00:00:43 |       96.88% |       0.1369 |          0.0010 |
|       8 |         300 |       00:00:53 |       99.22% |       0.0744 |          0.0010 |
|       9 |         350 |       00:01:01 |       99.22% |       0.0589 |          0.0010 |
|      10 |         390 |       00:01:07 |      100.00% |       0.0469 |          0.0010 |
|========================================================================================|
Training finished: Max epochs completed.

Evaluate the network performance by predicting on new data and calculating the accuracy.

[XTest,YTest] = digitTest4DArrayData;
YPred = classify(net,XTest);
accuracy = mean(YTest==YPred)
accuracy = 0.9178

References

[1] "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." In 2015 IEEE International Conference on Computer Vision (ICCV), 1026–34. Santiago, Chile: IEEE, 2015. https://doi.org/10.1109/ICCV.2015.123.

See Also

| | | | | | | | |

Related Topics