How to build an Actor-Critic model with shared layers?

4 views (last 30 days)
Hi.
I'm trying to build an Actor-Critic model uisng Reinforcement Learning Toolbox.
What I'm currently intending is to share low level features over Actor and Critic, like this.
-----------------------------
inputImage
|
backboneNet
| |
actorNet criticNet
-------------------------------
So, I created three LayerGraphs for the backbone, as well as actor and critic, and connected them.
And then, the next steps are the same as the guideline of generating AC agent, like this.
%% Backbone Network
% ...(Some codes to define the backbone layers)...
backboneNet = createLgraphUsingConnections(all_layers, all_connections) ;
%% Critic Network
% ...(Some codes to define the critic layers)...
criticNet = layerGraph(fullyConnectedLayer(1, 'Name', 'critic_FC')) ;
all_layers = [...
backbone.Layers
criticNet.Layers] ;
all_connections = [...
backbone.Connections
criticNet.Connections] ;
connection = subgraphConnections(backbone.Layers(end), criticNet.Layers(1));
all_connections = [all_connections; connection];
criticNet = createLgraphUsingConnections(all_layers, all_connections) ;
critic = rlValueRepresentation(...
criticNet, obsInfo,...
'Observation', {'inputImage'},...
criticOpts);
%% Actor Network
% ...(Some codes to define the actor layers)...
actorNet_concatOut = layerGraph(concatenationLayer(1, 2, 'Name', 'mean&sdev'));
all_layers = [...
backbone.Layers
actorNet_meanPath.Layers
actorNet_sdevPath.Layers
actorNet_concatOut.Layers] ;
all_connections = [...
backbone.Connections
actorNet_meanPath.Connections
actorNet_sdevPath.Connections
actorNet_concatOut.Connections] ;
connection1 = subgraphConnections(...
backbone.Layers(end),...
[actorNet_meanPath.Layers(1) actorNet_sdevPath.Layers(1)]);
connection2 = subgraphConnections(...
[actorNet_meanPath.Layers(end) actorNet_sdevPath.Layers(end)],...
actorNet_concatOut.Layers(1));
all_connections = [all_connections; connection1; connection2];
actorNet = createLgraphUsingConnections(all_layers, all_connections) ;
actor = rlStochasticActorRepresentation(...
actorNet, obsInfo, actInfo,...
'Observation', {'inputImage'},...
actorOpts);
%% Agent
agent = rlACAgent(actor,critic,agentOpts);
But If I put in the backboneNet as above, the actorNet gradients appear as zero so the actorNet doen't learn anything.
When I was not using the backboneNet, everything was fine.
I guess I'm doing something wrong or misunderstand how to define shared layers.
Is there anybody know how to define shared layers over the actor and critic?
Thanks
  1 Comment
Bradley Fourie
Bradley Fourie on 13 Aug 2022
Hi Heesu,
I am attempting something similar, have you perhaps figured out what the problem was?
Kind regards,
Brad

Sign in to comment.

Answers (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by