In DDPG algorithm to control Vehicle HVAC system, the agent is not able to start its traning. No error or no termination.

I am developing the DDPG algorithm to control the vehicle HVAC system. I have 4 output ranging within 0 and 1, each. While running the simulation, the code runs and it keep on executing but the training is not started. The training episode manager opens as blank white screen and the code is ruuning with any update. Even the Simulink model remains Ideal.

[Akshat]

Hi Konguvel,

Could you help me with the model that you are running so that I can reproduce the issue at my end?

[Konguvel Rajeswari Subramaniam]

Hi Akshat,

Thank you for your response. How can I produce my model. Either I need to attach the complete model or just the DDPG algorithm?

[Akshat]

In order to better understand and address the issue you are facing, it would be helpful if you could provide your model and the DDPG algorithm you are using.

[Konguvel Rajeswari Subramaniam]

Hi Akshat,

I have attached the required files for your reference.

Answer 1

[Ayush Aniket]

Hi Konguvel,

As per my understanding the training process for your RL model keeps executing without any updates or any errors. The following is my observation after going through the code you provided:

The reset function “localResetFcn(in)” has an invalid condition for resetting the “PMV” parameter. The condition for the while loop seems redundant:

% Randomize initial PMV index Values 
blk = sprintf('Cooling/pmv');
v = rand*3; 
while v >= 0 || v >= 3 %The issue lies in this statement.
    v = rand*3; 
end
in = setBlockParameter(in,blk,'Value',num2str(v)); 

It should be modified as follows:

while v <= 0 || v >= 3

This statement might be putting the training function in an infinite loop causing the issue that you mentioned. However, after correcting this part, I am getting an error with reference to the path of the “setBlockParameter()” function. You should find the actual path of the parameter that you want to reset and provide that information in the function. For instance, the path of the "PMV" parameter is:

blk = sprintf('Cooling/pmv');

whereas the actual path will be as shown below:

blk = sprintf('Cooling/VHVAC/PMV');

You can read more about the “setBlockParameter()” function by referring to the following link:

https://www.mathworks.com/help/simulink/slref/simulink.simulationinput.setblockparameter.html#bvqkv10-BlockPath

Also, to get the full path of a block in a model you can use the following command after interactively selecting that block in the model:

blkpath = getfullname(gcb); 

You can read more about finding path of a block on the following link:

https://www.mathworks.com/help/simulink/slref/getfullname.html

Hope it helps.