How to build an Actor-Critic model with shared layers?

3 visualizaciones (últimos 30 días)
Heesu Kim
Heesu Kim el 8 de Mzo. de 2021
Comentada: Bradley Fourie el 13 de Ag. de 2022
Hi.
I'm trying to build an Actor-Critic model uisng Reinforcement Learning Toolbox.
What I'm currently intending is to share low level features over Actor and Critic, like this.
-----------------------------
inputImage
|
backboneNet
| |
actorNet criticNet
-------------------------------
So, I created three LayerGraphs for the backbone, as well as actor and critic, and connected them.
And then, the next steps are the same as the guideline of generating AC agent, like this.
%% Backbone Network
% ...(Some codes to define the backbone layers)...
backboneNet = createLgraphUsingConnections(all_layers, all_connections) ;
%% Critic Network
% ...(Some codes to define the critic layers)...
criticNet = layerGraph(fullyConnectedLayer(1, 'Name', 'critic_FC')) ;
all_layers = [...
backbone.Layers
criticNet.Layers] ;
all_connections = [...
backbone.Connections
criticNet.Connections] ;
connection = subgraphConnections(backbone.Layers(end), criticNet.Layers(1));
all_connections = [all_connections; connection];
criticNet = createLgraphUsingConnections(all_layers, all_connections) ;
critic = rlValueRepresentation(...
criticNet, obsInfo,...
'Observation', {'inputImage'},...
criticOpts);
%% Actor Network
% ...(Some codes to define the actor layers)...
actorNet_concatOut = layerGraph(concatenationLayer(1, 2, 'Name', 'mean&sdev'));
all_layers = [...
backbone.Layers
actorNet_meanPath.Layers
actorNet_sdevPath.Layers
actorNet_concatOut.Layers] ;
all_connections = [...
backbone.Connections
actorNet_meanPath.Connections
actorNet_sdevPath.Connections
actorNet_concatOut.Connections] ;
connection1 = subgraphConnections(...
backbone.Layers(end),...
[actorNet_meanPath.Layers(1) actorNet_sdevPath.Layers(1)]);
connection2 = subgraphConnections(...
[actorNet_meanPath.Layers(end) actorNet_sdevPath.Layers(end)],...
actorNet_concatOut.Layers(1));
all_connections = [all_connections; connection1; connection2];
actorNet = createLgraphUsingConnections(all_layers, all_connections) ;
actor = rlStochasticActorRepresentation(...
actorNet, obsInfo, actInfo,...
'Observation', {'inputImage'},...
actorOpts);
%% Agent
agent = rlACAgent(actor,critic,agentOpts);
But If I put in the backboneNet as above, the actorNet gradients appear as zero so the actorNet doen't learn anything.
When I was not using the backboneNet, everything was fine.
I guess I'm doing something wrong or misunderstand how to define shared layers.
Is there anybody know how to define shared layers over the actor and critic?
Thanks
  1 comentario
Bradley Fourie
Bradley Fourie el 13 de Ag. de 2022
Hi Heesu,
I am attempting something similar, have you perhaps figured out what the problem was?
Kind regards,
Brad

Iniciar sesión para comentar.

Respuestas (0)

Productos


Versión

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by