Error in retraining multi-agent reinforcement learning

Hi,

I'm trying to stop and resume a multi-agent training. I'm using two RL policy gradient agents. However, it throws following error.

Index exceeds the number of array elements. Index must not exceed 1.

Error in rl.train.rlTrainingResult/checkStopTrainOptionChanged (line 172)
                this.TrainingOptions.StopTrainingValue(agentIdx) ~= str2double(this.Information.StopTrainingValue);

Error in rl.internal.episodeManager.EpisodeManager/resumeTraining (line 242)
                if checkManualStopTrainOrMaxEpisodes(checkpoint(idx)) || checkStopTrainOptionChanged(checkpoint(idx),idx)

Error in rl.train.TrainingManager/initializeEpisodeManager (line 560)
                    resumeTraining(this.EpisodeMgr,checkpoint);

Error in rl.train.TrainingManager/preTrain (line 287)
            initializeEpisodeManager(this,checkpoint);

Error in rl.train.TrainingManager/run (line 217)
            preTrain(this,checkpoint);

Error in rl.agent.AbstractAgent/train (line 83)
trainingResult = run(trainMgr,checkpoint);

I followed material provided here.

https://au.mathworks.com/help/reinforcement-learning/ref/rl.agent.rlqagent.train.html#mw_8a9c91b8-bd46-46ed-8968-ee203489c8f3

I updated the EntropyLossWeight parameter in agent options, not sure if that will cause this error.The max steps and the stop training value are both updated. Still unable to resume the training.

Thanks for your help

Answer 1

[Aditya]

Hey Venkata

I understand the error that was occurred and here are few ways to resolve it.

The error you encountered during the stop and resume of multi-agent training in RL policy gradient agents is related to the `StopTrainingValue` option. The error message suggests that the index used to access the `StopTrainingValue` array exceeds its size, which should not happen.

To troubleshoot and resolve this issue, you can try the following steps:

1. Verify `StopTrainingValue` array: Double-check the `StopTrainingValue` option for each agent. Ensure that the array size matches the number of agents being trained. Each agent should have a corresponding value in the `StopTrainingValue` array.

2. Check `EntropyLossWeight` update: Updating the `EntropyLossWeight` parameter in the agent options should not directly cause this error. However, if you made any other changes or updates to the agent options, ensure that they are correctly set and consistent across training and resume stages.

3. Confirm `MaxEpisodes` and `MaxStepsPerEpisode`: Ensure that both `MaxEpisodes` and `MaxStepsPerEpisode` are updated correctly for the resumed training. These parameters control the maximum number of episodes and steps per episode, respectively. Make sure they match the desired values for the resumed training.

4. Use `rlTrainingOptions`: Instead of directly modifying the agent options, consider using `rlTrainingOptions` to set the training options. This allows for more flexibility and easier management of training parameters. You can specify `MaxEpisodes`, `MaxStepsPerEpisode`, `StopTrainingCriteria`, and `StopTrainingValue` within `rlTrainingOptions` to ensure consistency.

Here's an example of how to use `rlTrainingOptions`:

trainingOpts = rlTrainingOptions('MaxEpisodes', maxEpisodes, 'MaxStepsPerEpisode', maxStepsPerEpisode, ...
    'StopTrainingCriteria', 'AverageReward', 'StopTrainingValue', stopTrainingValue);

trainingStats = train(agent, env, trainingOpts);

By using `rlTrainingOptions`, you can ensure that the training options are correctly set and consistent throughout the training and resume stages.

If the issue persists, it would be helpful to provide more details about your code and the specific values used for the options. This will allow for a more in-depth analysis of the problem and provide more targeted assistance in resolving the issue.