How do I use trainNetwork for a sequence-to-one regression?

Hi there,

I'm trying to create a model for a sequence-to-one regression. But sadly I receive an error:

Error using trainNetwork

Invalid training data. If the network outputs sequences, then regression responses must be a cell array of numeric sequences, or a single numeric sequence.

My Dataset: (a snapshot with 128 samples is attached - see .mat file)

load dataset.mat
whos X_train y_train

  Name           Size              Bytes  Class     Attributes

  X_train      128x1             6034432  cell                
  y_train      128x1                1024  double              

size(X_train) % 128x1 cell (128 samples)

ans = 1×2

128 1

size(X_train{1}) % 35x168 double (35 features with 168 time steps each)

ans = 1×2

35 168

size(y_train) % 128x1 double (1 numeric output for each sample)

ans = 1×2

128 1

My Network:

layers = [ 
    sequenceInputLayer(35)  % number of features
    fullyConnectedLayer(42) % dummy value 
    tanhLayer 
    fullyConnectedLayer(1)  % number of responses per sample 
    regressionLayer
];

options = trainingOptions('adam', 'MaxEpochs', 1); 

trainNetwork(X_train, y_train, layers, options)

Error using trainNetwork (line 184)
Invalid training data. If the network outputs sequences, then regression responses must be a cell array of numeric sequences, or a single numeric sequence.

After reading the docs of Train deep learning neural network - MATLAB trainNetwork - MathWorks Deutschland I don't see my mistake.

From my point of view I'm doing the same thing as shown in the example from Sequence input layer - MATLAB - MathWorks Deutschland except I'm doing a regression instead of categorization.

openExample('matlab/DivideArrayAndReturnSubarraysInCellArrayExample')

Answer 1

[Nikhil Reddy]

I understand that you are trying to create a sequence-to-one regression model. However, you are recieving an error regarding the responses (y_train) input parameter to the "trainNetwork" function. I see that since you are trying to acheive a single response for each sample(X_train), y_train has a dimentionality of n x 1.

It is my understanding that since you are providing sequence data, we also need to add an additional layer in the "layers" array which learns long term dependencies between time steps in the sequence data. We can add a "lstmLayer" with desired number of hidden memory units (depending on how much information the model should remember) along with the desired 'OutputMode' (in this case the 'OutputMode' is set to 'last' since the desired configuration of the regression model is sequence-to-one which is an extension of the sequence-to-sequence model). Find the redefined code for layers array below:

layers = [ 
    sequenceInputLayer(35)  % number of features
    lstmLayer(10, 'OutputMode',"last")
    fullyConnectedLayer(42) % dummy value 
    tanhLayer 
    fullyConnectedLayer(1)  % number of responses per sample 
    regressionLayer
];

Please refer to the following links for more information:

lstmLayer:

Long short-term memory (LSTM) layer - MATLAB (mathworks.com)

Network architecture in sequence-to-sequence models:

https://www.mathworks.com/help/deeplearning/ug/sequence-to-sequence-regression-using-deep-learning.html#SequencetoSequenceRegressionUsingDeepLearningExample-3

[Paul Krüger]

You are absoluty right!

I wanted to compare a lstm network with hidden state to a more "basic" network.

I solved my task by providing a modified dataset containing only the last time step to my basic network (performing one-to-one regression) and the sequential dataset to a network similar to your layers.

[aryan ramesh]

Hello, Paul.

It seems i am struggling with the same problem here. Tks for your question.

Could you show the final size of you X_train data.

How did you modify your data :

I solved my task by providing a modified dataset containing only the last time step

Best Regards,

[Paul Krüger]

Hi @aryan ramesh,

maybe I can help you.

As @Nikhil Reddy in the accepted answer points out:

"It is my understanding that since you are providing sequence data, we also need to add an additional layer in the "layers" array which learns long term dependencies between time steps in the sequence data."

My solution was not to use sequential data anymore.

I changed my sequence-to-one regression to a one-to-one regression using only the last time step of my 168 time steps. My X_train is a 128x36 double array. (number of data x features)

In my example you need change sequenceInputLayer to featureInputLayer as well.

layers = [

featureInputLayer(35) % number of features

fullyConnectedLayer(42) % dummy value

tanhLayer

fullyConnectedLayer(1) % number of responses per sample

regressionLayer

];

If you still want a sequence-to-one regression you need a lstm layer after the sequenceInputLayer. See the accepted answer.

Let me know if you need anything else!

[Md Zahidul Islam]

Recently I faced same problem while using 1dimentional CNN model. However, one way to solve the problem is to use globalAveragePooling1dLayer for such case. This layer take the average of all sequence in a channel and give single output.