Reinforcement Learning with Simulink: timeout an episode

Setup:

• Using RL train function in a parfor loop to optimize hyperparameters
• Simulink model with RL Agent

Problem:

• Some hyperparameters do not produce sensible Agents, some of which might lock the simulation in a timestep forever (happens occassionally) due to solver adjustments

Desired solution:

• Have a timeout for individual epsidodes, e.g., 10min, that just kills the simulation if it does not finish in the specified time
• Usually, the way to do this is to pass "TimeOut" to the Simulink simulation object
• Is there a way to pass the simulink object into train or any other way to to set a timeout in the model parameters?

Answer 1

[Shubham]

Hi Tobias,

To manage simulations that run indefinitely due to certain hyperparameter configurations in a parallel loop (parfor) while training RL agents with MATLAB and Simulink, implementing a timeout mechanism for your training episodes is essential. Although there isn't a direct way to pass a "TimeOut" parameter through the train function that applies to the Simulink simulation time, there are several strategies you could explore to achieve a similar outcome:

Custom Training Loop with Timeout

Crafting a custom training loop gives you the flexibility to include a timeout mechanism. Within this loop, you can monitor the elapsed time and terminate the simulation if it exceeds your predefined limit. This method requires a good grasp of how the RL toolbox operates but offers the most control.

Utilizing the Simulation Pace Block

For simulations running in normal or accelerator mode, incorporating a Simulation Pace block from the Simulink library might help. Although this block doesn't directly stop the simulation after a certain period, it can be used alongside a MATLAB function block designed to halt the simulation based on elapsed time.

External Script or Function to Monitor and Terminate Simulations

Creating an external MATLAB script or function that oversees the Simulink process and terminates it if it runs beyond the specified timeout can be effective. This approach involves identifying the process ID (PID) of the simulation when it starts and using a timer or loop within MATLAB to check the elapsed time, killing the process if necessary. This method is somewhat brute-force and requires precision to avoid terminating the wrong processes or causing data issues.

Modifying the Agent or Environment for Timeout

Adjusting the RL environment or the agent to include a timeout mechanism is another viable strategy. This could involve adding a step counter to the environment that ends the episode once a maximum number of steps, calculated based on the expected duration of each step and the total allowed simulation time, is reached. Alternatively, modifying the reward function to penalize or conclude the episode under certain conditions could also work.

Using MATLAB's timeout Function

For MATLAB versions R2021a and later, the timeout function allows executing a function with a time limit. Although originally designed for async operations, it's possible to wrap your training call within a function that executes with a timeout. This method may require significant adjustments to your training setup and isn't guaranteed to integrate flawlessly with all configurations of the RL toolbox and Simulink simulations.

Conclusion

Each method offers different levels of complexity, reliability, and integration with the MATLAB and Simulink ecosystem. Modifying the RL environment or agent to include a timeout mechanism is often the most straightforward and robust approach, keeping all controls within the MATLAB and Simulink environment and avoiding the need for external process management.