how to install a network structure with two input,one is imageinput,the other is sequence input?

I want to improve the performance of a CNN network. So I add a sequencial input(which is match with the CNN image input), and training it with LSTM network ,then join them together in their fullyconnected layers, and use softmax to classify. But the deepNetworkDesigner don't support these two kinds of input. How can I solve this problem?

Answer 1

[Krishna]

Hi Zhengyun,

It seems you're looking to enhance the performance of a CNN network by incorporating a sequential layer for your image data, which will then be fed into an LSTM network. The output from this would subsequently be merged and passed through a fully connected layer for classification purposes.

You can do this in MATLAB using ‘dlnetwork’. You can pass the data as sequence to an LSTM architecture and as an image to CNN architecture which are two different networks then combine using ‘additionLayer.’ Make sure your dimension of the output are same for both the network. Then pass this combined output through fully connected layer and then a SoftMax function. This is how the architecture should like.

Please go through the following documentation to learn more,

https://in.mathworks.com/help/deeplearning/ref/dlnetwork.html

https://in.mathworks.com/help/deeplearning/ug/list-of-deep-learning-layers.html

https://in.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.additionlayer.html

Also please go through this example to learn more about using CNN and RNN together,

https://in.mathworks.com/help/deeplearning/ug/sequence-classification-using-cnn-lstm-network.html

Also, for this type of combined networks you need to use Datastores to train the network in MATLAB, please go through the following documentation to learn more,

https://in.mathworks.com/help/matlab/datastore.html

Hope this helps.