ProjectedLayer

Compressed neural network layer using projection

Since R2023b

Description

Add-On Required: This feature requires the Deep Learning Toolbox Model Compression Library add-on.

A projected layer is a compressed neural network layer resulting from projection.

Creation

To compress a neural network using projection, use the compressNetworkUsingProjection function.

Properties

Projected

OriginalClass — Class of original layer

Read-only: character vector

This property is read-only.

Class of the original layer, returned as a character vector.

Example: 'nnet.cnn.layer.LSTMLayer'

Data Types: char

LearnablesReduction — Proportion of learnables removed in layer

Read-only: scalar in the interval [0,1]

This property is read-only.

Proportion of learnables removed in the layer, returned as a scalar in the interval [0,1].

Data Types: double

Network — Neural network that represents projection

dlnetwork object

Neural network that represents projection, returned as a dlnetwork object.

The neural network that represents the projection depends on the type of layer:

Original Layer	Network
convolution1dLayer (since R2024b)	Network containing two or three convolution1dLayer objects
convolution2dLayer	Network containing two or three convolution2dLayer objects
fullyConnectedLayer	Network containing two fullyConnectedLayer objects
lstmLayer	Network containing a single lstmProjectedLayer object
gruLayer	Network containing a single gruProjectedLayer object

For more information, see Projected Layer. To replace the ProjectedLayer objects in a neural network with the equivalent network that represents the projection, use the unpackProjectedLayers function.

InputSize — Number of input channels

Read-only: positive integer

This property is read-only.

Number of input channels, returned as a positive integer.

Data Types: double

OutputSize — Number of output channels

Read-only: positive integer

This property is read-only.

Number of output channels, returned as a positive integer.

Data Types: double

InputProjectorSize — Number of columns of input projector

Read-only: positive integer

This property is read-only.

Number of columns of the input projector, returned as a positive integer.

The input projector is the matrix Q used to project the layer input. For more information, see Projected Layer.

Data Types: double

OutputProjectorSize — Number of columns of output projector

Read-only: positive integer

This property is read-only.

Number of columns of the output projector, returned as a positive integer.

The output projector is the matrix Q used to project the layer output. For more information, see Projected Layer.

Data Types: double

Layer

Name — Layer name

character vector | string scalar

Layer name, specified as a character vector or a string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign new unique names to layers that have the name "".

When you compress a neural network using the compressNetworkUsingProjection, the function replaces projectable layers with ProjectedLayer objects with the same name.

The ProjectedLayer object stores this property as a character vector.

Data Types: char | string

NumInputs — Number of inputs

Read-only: positive integer

This property is read-only.

Number of inputs to the layer, returned as a positive integer.

Data Types: double

InputNames — Input names

cell array of character vectors

Input names, returned as a cell array of character vectors.

Data Types: cell

NumOutputs — Number of outputs

Read-only: positive integer

This property is read-only.

Number of outputs from the layer, returned as a positive integer.

Data Types: double

OutputNames — Output names

cell array of character vectors

Output names, returned as a cell array of character vectors.

Data Types: cell

Examples

Compress Network Using Projection

Load the pretrained network in dlnetJapaneseVowels and the training data in JapaneseVowelsTrainData.

load dlnetJapaneseVowels
load JapaneseVowelsTrainData

Compress the network. Specify the mini-batch size for the principal component analysis as 16. Specify the input data format as "CTB".

[netProjected,info] = compressNetworkUsingProjection(net,XTrain,MiniBatchSize=16,InputDataFormats="CTB");

Compressed network has 85.4% fewer learnable parameters.
Projection compressed 2 layers: "lstm","fc"

View the network layers.

netProjected.Layers

ans = 
  4×1 Layer array with layers:

     1   'sequenceinput'   Sequence Input    Sequence input with 12 channels
     2   'lstm'            Projected Layer   Projected LSTM with 100 hidden units
     3   'fc'              Projected Layer   Projected fully connected layer with output size 9
     4   'softmax'         Softmax           Softmax

View the projected LSTM layer.

netProjected.Layers(2)

ans = 
  ProjectedLayer with properties:

                   Name: 'lstm'
          OriginalClass: 'nnet.cnn.layer.LSTMLayer'
    LearnablesReduction: 0.8610
              InputSize: 12
             OutputSize: 100

   Hyperparameters
     InputProjectorSize: 7
    OutputProjectorSize: 6

   Learnable Parameters
                Network: [1×1 dlnetwork]

   State Parameters
                Network: [1×1 dlnetwork]

   Network Learnable Parameters
     Network/lstm/InputWeights      400×7 dlarray
     Network/lstm/RecurrentWeights  400×6 dlarray
     Network/lstm/Bias              400×1 dlarray
     Network/lstm/InputProjector    12×7  dlarray
     Network/lstm/OutputProjector   100×6 dlarray

   Network State Parameters
     Network/lstm/HiddenState  100×1 dlarray
     Network/lstm/CellState    100×1 dlarray

  Show all properties

Unpack Projected Layers

Load the pretrained network in dlnetProjectedJapaneseVowels.

load dlnetProjectedJapaneseVowels

View the network properties.

net

net = 
  dlnetwork with properties:

         Layers: [4×1 nnet.cnn.layer.Layer]
    Connections: [3×2 table]
     Learnables: [9×3 table]
          State: [2×3 table]
     InputNames: {'sequenceinput'}
    OutputNames: {'softmax'}
    Initialized: 1

  View summary with summary.

View the network layers. The network has two projected layers.

net.Layers

ans = 
  4×1 Layer array with layers:

     1   'sequenceinput'   Sequence Input    Sequence input with 12 channels
     2   'lstm'            Projected Layer   Projected LSTM with 100 hidden units
     3   'fc'              Projected Layer   Projected fully connected layer with output size 9
     4   'softmax'         Softmax           Softmax

Unpack the projected layers.

netUnpacked = unpackProjectedLayers(net)

netUnpacked = 
  dlnetwork with properties:

         Layers: [5×1 nnet.cnn.layer.Layer]
    Connections: [4×2 table]
     Learnables: [9×3 table]
          State: [2×3 table]
     InputNames: {'sequenceinput'}
    OutputNames: {'softmax'}
    Initialized: 1

  View summary with summary.

View the unpacked network layers. The unpacked network has a projected LSTM layer and two fully connected layers in place of the projected layers.

netUnpacked.Layers

ans = 
  5×1 Layer array with layers:

     1   'sequenceinput'   Sequence Input    Sequence input with 12 channels
     2   'lstm'            Projected LSTM    Projected LSTM with 100 hidden units, an output projector size of 7, and an input projector size of 8
     3   'fc_proj_in'      Fully Connected   Fully connected layer with output size 4
     4   'fc_proj_out'     Fully Connected   Fully connected layer with output size 9
     5   'softmax'         Softmax           Softmax

Algorithms

Projected Layer

To compress a deep learning network, you can use projected layers. A projected layer is a type of deep learning layer that enables compression by reducing the number of stored learnable parameters. The layer introduces learnable projector matrices Q, replaces multiplications of the form $$Wx$$, where W is a learnable matrix, with the multiplication $$WQ{Q}^{\top }x$$, and stores Q and $$W\prime =WQ$$ instead of storing W. Projecting x into a lower dimensional space using Q typically requires less memory to store the learnable parameters and can have similarly strong prediction accuracy.

For some types of layers, you can represent a projected layer as a neural network containing two or more layers with fewer learnable parameters. For example, you can represent a projected convolution layer as three convolution layers that perform the input projection, convolution, and the output projection operations independently. When you compress a network using the compressNetworkUsingProjection function, the software replaces layers that support projection with ProjectedLayer objects that contain the equivalent neural network. To replace ProjectedLayer objects in a neural network with the equivalent neural network that represents the projection, use the unpackProjectedLayers function or set the UnpackProjectedLayers option of the compressNetworkUsingProjection function to 1 (true).

In rare cases, layers can increase in size due to projection. For example, this can happen when:

• A layer is small. Adding projector matrices increases the number of learnable parameters. Usually, the decrease in number of learnable parameters of the original layer is large enough that the total number of learnable parameters decreases. However, when the original layer is already small, then the decrease in learnable parameters may not be sufficient and the overall number of learnable parameters can increase.

• The ExplainedVarianceGoal argument is too large, or if the layer does not contain much redundant information. In this case, the projector matrices are added, but not truncated enough to decrease the total number of learnable parameters.

The compressNetworkUsingProjection and neuronPCA functions support these layers:

• convolution1dLayer (since R2024b)

• convolution2dLayer

• fullyConnectedLayer

• lstmLayer

• gruLayer

Note

Layers that share learnable parameters with other layers through weight tying do not support compression using projection.

Since R2026a, compressNetworkUsingProjection and neuronPCA support compressing layers contained inside a networkLayer object.

The compressNetworkUsingProjection function replaces projectable layers with ProjectedLayer objects. A ProjectedLayer object contains information about the projection operation and contains the neural network that represents the projection.

The neural network that represents the projection depends on the type of layer:

Original Layer	Network
convolution1dLayer (since R2024b)	Network containing two or three convolution1dLayer objects
convolution2dLayer	Network containing two or three convolution2dLayer objects
fullyConnectedLayer	Network containing two fullyConnectedLayer objects
lstmLayer	Network containing a single lstmProjectedLayer object
gruLayer	Network containing a single gruProjectedLayer object

Layer Input and Output Formats

Most layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray objects. The format of a dlarray object is a string of characters in which each character describes the corresponding dimension of the data. The format consists of one or more of these characters:

• "S" — Spatial

• "C" — Channel

• "B" — Batch

• "T" — Time

• "U" — Unspecified

For example, you can describe 2-D image data that is represented as a 4-D array, where the first two dimensions correspond to the spatial dimensions of the images, the third dimension corresponds to the channels of the images, and the fourth dimension corresponds to the batch dimension, as having the format "SSCB" (spatial, spatial, channel, batch).

You can interact with these dlarray objects in automatic differentiation workflows, such as those for:

• developing a custom layer

• using a functionLayer object

• using the forward and predict functions with dlnetwork objects

To learn more about the supported input and output formats of a ProjectedLayer object, see the documentation for the layer given by the OriginalClass property.

References

[1] "Compressing Neural Networks Using Network Projection." Accessed July 20, 2023. https://www.mathworks.com/company/technical-articles/compressing-neural-networks-using-network-projection.html.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.

Version History

Introduced in R2023b

R2026a: Code generation support

Generate C or C++ code using MATLAB^® Coder™.