How RL + lstm work

I'm using MATLAB for reinforcement learning. I've activated the RNN of the agent and noticed that a layer of LSTM has been added to the network. Now I want to know whether this LSTM uses the parameters of the previous network output at the current time as the time series, or uses the observations at different times or the previous layer of the LSTM network as the time series. Also, are there any relevant literatures on RL + LSTM?

Answer 1

[Shantanu]

Hi Jin,

As for the LSTM input, it uses the activations (not parameters) from the previous network layer at the current time as its main input. It handles the "time series" aspect by combining this with its internal hidden state (its memory) from the previous time step.

Therefore, it processes observations one by one, not all at once. As for the RL agents that can use LSTMs, the main ones are DQN, PPO, A2C, DDPG, SAC, and TD3.

Some resources and examples that may be helpful

https://www.mathworks.com/help/reinforcement-learning/ug/create-agents-for-reinforcement-learning.html?requestedDomain=

https://www.mathworks.com/help/reinforcement-learning/ug/train-dqn-to-control-house-heating.html