FileLogger

Log reinforcement learning training data to MAT files

Since R2022b

expand all in page

    Description

    Use a FileLogger object to log data to MAT files, within the train function or inside a custom training loop. To log data when using the train function, specify appropriate callback functions in FileLogger, as shown in the examples. These callbacks are executed at different stages of training, for example, EpisodeFinishedFcn is executed after the completion of an episode. The output of a callback function is a structure containing the data to log at that stage of training.

    Note

    Using a FileLogger object to log data when using the train function does not affect (and is not affected by) any option to save agents during training specified within a training option object.

    Note

    When using a FileLogger object to log data using a built-in training function and no data is available to be logged for an episode, then no MAT file for that episode will be created. For example, if you are logging the training losses using the AgentLearnFinishedFcn callback then data is created only for only the episodes in which that the agent learn. If, for example, the first episode does not contain an adequate number of steps for the agent to learn, then no MAT file is created for that episode.

    Note

    FileLogger is a handle object. If you assign an existing FileLogger object to a new FileLogger object, both the new object and the original one refer to the same underlying object in memory. To preserve the original object parameters for later use, save the object to a MAT file. For more information about handle objects, see Handle Object Behavior.

    Creation

    Create a FileLogger object using rlDataLogger without any input arguments.

    Properties

    expand all

    Object containing a set of logging options, returned as a MATFileLoggingOptions object. A MATFileLoggingOptions object has the following properties, which you can access using dot notation after creating the FileLogger object.

    Name or fully qualified path of the logging directory, specified as a string or a character vector. This is the name of the directory where the MAT files containing the logged data are saved. As a default, a subdirectory called logs is created in the current folder during setup and files are saved there during training.

    Example: LoggingDirectory=mylogs

    Rule to name the MAT files, specified as a string or a character vector. For example, the naming rule "episode<id>" results in the file names episode001.mat, episode002.mat and so on.

    Example: FileNameRule="ThirdRun<id>"

    MAT file version, specified as a string or character vector. The default is "-v7". For more information, see MAT-File Versions.

    Example: Version="-v7.3"

    Option to use compression when saving data to a MAT file, specified as a logical variable. The default is true. For more information, see MAT-File Versions.

    Example: UseCompression=false

    Disk data write period, specified as a positive integer. It is the number of episodes after which data is saved to disk. For example, if DataWriteFrequency is 5 then data from episodes 1 to 5 will be cached in memory and be written to disk at the end of the 5-th episode. This improves performance in some cases. The default is 1.

    Example: DataWriteFrequency=10

    This property is read-only.

    Maximum number of episodes, specified as a positive integer. When using train, the value is automatically initialized. Set this value when using the logger object in a custom training loop. The default is 500.

    Example: MaxEpisodes=1000

    This property is read-only.

    Callback to log data after episode completion, specified as a function handle object. The specified function is automatically called by the training loop at the end of each episode, and must return a structure containing the data to log, such as experiences, simulation information, or initial observations.

    Your function must have the following signature.

    function dataToLog = myEpisodeFinishedFcn(data)

    Here, data is a structure that contains the following fields:

    • EpisodeCount — current episode number

    • Environment — environment object

    • Agent — agent object

    • Experience — structure array containing the experiences. Each element of this array corresponds to a step and is a structure containing the fields NextObservation, Observation, Action, Reward and IsDone.

    • EpisodeInfo — structure containing the fields CumulativeReward, StepsTaken and InitialObservation.

    • SimulationInfo — contains simulation information from the episode. For MATLAB environments this is a structure with the field SimulationError, and for Simulink® environments it is a Simulink.SimulationOutput object.

    The function output dataToLog is the structure containing the data to be logged to disk.

    Example: EpisodeFinishedFcn=@myEpLoggingFcn

    This property is read-only.

    Callback to log data after training step completion within an episode, specified as a function handle object. The specified function is automatically called by the training loop at the end of each training step, and must return a structure containing the data to log, such as for example the state of the agent's exploration policy.

    Your function must have the following signature.

    function dataToLog = myAgentStepFinishedFcn(data)

    Here, data is a structure that contains the following fields:

    • EpisodeCount — current episode number

    • AgentStepCount — cumulative number of steps taken by the agent

    • SimulationTime — current simulation time in the environment

    • Agent — agent object

    The function output dataToLog is the structure containing the data to be logged to disk.

    For multi agent training, AgentStepFinishedFcn can be a cell array of function handles with as many elements as number of agent groups.

    Note

    Logging data using the AgentStepFinishedFcn callback is not supported when training agents in parallel with the train function.

    Example: AgentStepFinishedFcn=@myAgtStepLoggingFcn

    This property is read-only.

    Callback to log data after completion of the learn subroutine, specified as a function handle object. The specified function is automatically called by the training loop at the end of each learning subroutine, and must return a structure containing the data to log, such as the training losses of the actor and critic networks, or, for a model-based agent, the environment model training losses.

    Your function must have the following signature.

    function dataToLog = myAgentLearnFinishedFcn(data)

    Here, data is a structure that contains the following fields:

    • EpisodeCount — current episode number

    • AgentStepCount — cumulative number of steps taken by the agent

    • AgentLearnCount — cumulative number of learning steps taken by the agent

    • EnvModelTrainingInfo — structure containing model-based agents related fields TransitionFcnLoss, RewardFcnLoss and IsDoneFcnLoss.

    • Agent — agent object

    • ActorLoss — value of the actor loss

    • CriticLoss — value of the critic loss

    • ActorGradientStepCount — cumulative number of gradient calculations for the actor (if the agent has an actor)

    • CriticGradientStepCount — cumulative number of gradient calculations for the critic (if the agent has a critic)

    Depending on the agent, the data structure also contains the following fields:

    • TDTarget — temporal-difference target value (for DQN, DDPG, TD3, SAC, PPO and TRPO agents)

    • TDError — temporal-difference target error (for DQN, DDPG, TD3, SAC, PPO and TRPO agents)

    • SampleIndex — indices of the minibatch experiences sampled for the current gradient step (for DQN, DDPG, TD3, and SAC agents)

    • MaskIndex — sequence padding mask (for DQN, DDPG, TD3, and SAC agents that use RNNs)

    • Advantage — advantage value (for PPO and TRPO agents)

    • PolicyRatio — policy ratio value (for PPO agents)

    • AdvantageLoss — advantage loss value (for PPO agents)

    • EntropyLoss — entropy loss value (for PPO agents)

    The function output dataToLog is the structure containing the data to be logged to disk.

    For multi agent training, AgentLearnFinishedFcn can be a cell array of function handles with as many elements as number of agent groups.

    Example: AgentLearnFinishedFcn=@myAgtLearnLoggingFcn

    Object Functions

    setupSet up reinforcement learning environment or initialize data logger object
    cleanupClean up reinforcement learning environment or data logger object

    Examples

    collapse all

    Open Live Script

    This example shows how to log data to disk when using train.

    Create a FileLogger object using rlDataLogger.

    logger = rlDataLogger();

    Specify a directory to save logged data.

    logger.LoggingOptions.LoggingDirectory = "myDataLog";

    Create callback functions to log the data (for this example, see the helper function section), and specify the appropriate callback functions in the logger object. 

    logger.EpisodeFinishedFcn    = @myEpisodeFinishedFcn;
logger.AgentStepFinishedFcn  = @myAgentStepFinishedFcn;
logger.AgentLearnFinishedFcn = @myAgentLearnFinishedFcn;

    To train the agent, you can now call train, passing logger as an argument such as in the following command.

    trainResult = train(agent, env, trainOpts, Logger=logger);

    While the training progresses, data will be logged to the specified directory, according to the rule specified in the FileNameRule property of logger.LoggingOptions.

    logger.LoggingOptions.FileNameRule
    ans = 
"loggedData<id>"

    Example Logging Functions

    Episode completion logging function. This function is automatically called by the training loop at the end of each episode, and must return a structure containing the episode-related data to log, such as experiences, simulation information, or initial observations.

    function dataToLog = myEpisodeFinishedFcn(data)
    dataToLog.Experience = data.Experience;
end

    Agent step completion logging function. This function is automatically called by the training loop at the end of each training step, and must return a structure containing the step-related data to log, such as for example the state of the agent exploration policy.

    function dataToLog = myAgentStepFinishedFcn(data)
    dataToLog.State = getState(getExplorationPolicy(data.Agent));
end

    Learn subroutine completion logging function. This function is automatically called by the training loop at the end of each learning subroutine, and must return a structure containing the learning-related data to log, such as the training losses of the actor and critic networks, or, for a model-based agent, the environment model training losses.

    function dataToLog = myAgentLearnFinishedFcn(data)
    dataToLog.ActorLoss  = data.ActorLoss;
    dataToLog.CriticLoss = data.CriticLoss;
end

    For the specific function signatures and more information on the function input structure, see the corresponding properties of FileLogger. For a related example, see Log Training Data to Disk.

    Open Live Script

    This example shows how to log data to disk when training an agent using a custom training loop.

    Create a FileLogger object using rlDataLogger.

    flgr = rlDataLogger();

    Set up the logger object. This operation initializes the object performing setup tasks such as, for example, creating the directory to save the data files. 

    setup(flgr);

    Within a custom training loop, you can now store data to the logger object memory and write data to file.

    For this example, store random numbers to the file logger object, grouping them in the variables Context1 and Context2. When you issue a write command, a MAT file corresponding to an iteration and containing both variables is saved with the name specified in flgr.LoggingOptions.FileNameRule, in the folder specified by flgr.LoggingOptions.LoggingDirectory.

    for iter = 1:10

    % Store three random numbers in memory 
    % as elements of the variable "Context1"
    for ct = 1:3
        store(flgr, "Context1", rand, iter);
    end

    % Store a random number in memory 
    % as the variable "Context2"
    store(flgr, "Context2", rand, iter);

    % Write data to file every 4 iterations
    if mod(iter,4)==0
        write(flgr);
    end

end

    Clean up the logger object. This operation performs clean up tasks like for example writing to file any data still in memory.

    cleanup(flgr);

    Limitations

    • Logging data using the AgentStepFinishedFcn callback is not supported when training agents in parallel with the train function.

    Version History

    Introduced in R2022b

    See Also

    Functions

    Objects

    Topics