Main Content

Experiment Manager

Design and run experiments to train and compare deep learning networks

Description

The Experiment Manager app enables you to create deep learning experiments to train networks under multiple initial conditions and compare the results. For example, you can use deep learning experiments to:

  • Sweep through a range of hyperparameter values or use Bayesian optimization to find optimal training options. Bayesian optimization requires Statistics and Machine Learning Toolbox™.

  • Use the built-in function trainNetwork or define your own custom training function.

  • Compare the results of using different data sets or test different deep network architectures.

To set up your experiment quickly, you can start with a preconfigured template. The experiment templates support workflows that include image classification, image regression, sequence classification, semantic segmentation, and custom training loops.

Experiment Manager provides visualization tools such as training plots and confusion matrices, filters to refine your experiment results, and annotations to record your observations. To improve reproducibility, every time that you run an experiment, Experiment Manager stores a copy of the experiment definition. You can access past experiment definitions to keep track of the hyperparameter combinations that produce each of your results.

Experiment Manager organizes your experiments and results in a project.

  • You can store several experiments in the same project.

  • Each experiment contains a set of results for each time that you run the experiment.

  • Each set of results consists of one or more trials corresponding to a different combination of hyperparameters.

By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox™, you can configure your experiment to run multiple trials at the same time or to run a single trial at a time on multiple GPUs, on a cluster, or in the cloud. For more information, see Use Experiment Manager to Train Networks in Parallel.

The Experiment Browser pane displays the hierarchy of experiments and results in the project. For instance, this project has two experiments, each of which has several sets of results.

Experiment Browser showing two experiments. Experiment1 is a built-in training experiment with four results. Experiment2 is a custom training experiment with two results.

The blue flask indicates a built-in training experiment that uses the trainNetwork function. The green beaker indicates a custom training experiment that relies on a different training function. To open the configuration for an experiment and view its results, double-click the name of the experiment or a set of results.

Experiment Manager app

Open the Experiment Manager App

  • MATLAB® Toolstrip: On the Apps tab, under Machine Learning and Deep Learning, click the app icon.

  • MATLAB command prompt: Enter experimentManager.

Examples

expand all

This example shows how to use the experiment template for image classification by sweeping hyperparameters. With this template, you can quickly set up a built-in training experiment that uses the trainNetwork function. For more examples of solving image classification problems with Experiment Manager, see Create a Deep Learning Experiment for Classification and Use Experiment Manager to Train Networks in Parallel. For more information on an alternative strategy to sweeping hyperparameters, see Tune Experiment Hyperparameters by Using Bayesian Optimization.

Open the example to load a project with a preconfigured experiment that you can inspect and run. To open the experiment, in the Experiment Browser pane, double-click the name of the experiment (Experiment1).

Alternatively, you can configure the experiment yourself by following these steps.

1. Open Experiment Manager. A dialog box provides links to the getting started tutorials and your recent projects, as well as buttons to create a new project or open an example from the documentation.

2. Under New, select Project. A dialog box lists several templates that support workflows including image classification, image regression, sequence classification, semantic segmentation, and custom training loops.

3. Under Image Classification Experiments, select Image Classification by Sweeping Hyperparameters.

4. Specify the name and location for the new project. Experiment Manager opens a new experiment in the project. The Experiment pane displays the description, hyperparameters, setup function, and metrics that define the experiment.

5. In the Description field, enter a description of the experiment:

Classification of digits, using various initial learning rates.

6. Under Hyperparameters, replace the value of myInitialLearnRate with 0.0025:0.0025:0.015. Verify that Strategy is set to Exhaustive Sweep.

7. Under Setup Function, click Edit. The setup function opens in MATLAB Editor. The setup function specifies the training data, network architecture, and training options for the experiment. In this experiment, the setup function has three sections.

  • Load Training Data defines image datastores containing the training and validation data for the experiment. The experiment uses the Digits data set, which consists of 10,000 28-by-28 pixel grayscale images of digits from 0 to 9, categorized by the digit they represent. For more information on this data set, see Image Data Sets.

  • Define Network Architecture defines the architecture for a simple convolutional neural network for deep learning classification.

  • Specify Training Options defines a trainingOptions object for the experiment. In this experiment, the setup function loads the values for the initial learning rate from the myInitialLearnRate entry in the hyperparameter table.

When you run the experiment, Experiment Manager trains the network defined by the setup function six times. Each trial uses one of the learning rates specified in the hyperparameter table. By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox, you can run multiple trials at the same time. For best results, before you run your experiment, start a parallel pool with as many workers as GPUs. For more information, see Use Experiment Manager to Train Networks in Parallel and GPU Support by Release (Parallel Computing Toolbox).

  • To run one trial of the experiment at a time, on the Experiment Manager toolstrip, click Run.

  • To run multiple trials at the same time, click Use Parallel and then Run. If there is no current parallel pool, Experiment Manager starts one using the default cluster profile. Experiment Manager then executes multiple simultaneous trials, depending on the number of parallel workers available.

A table of results displays the accuracy and loss for each trial.

While the experiment is running, click Training Plot to display the training plot and track the progress of each trial. You can also monitor the training progress in the MATLAB Command Window.

Click Confusion Matrix to display the confusion matrix for the validation data in each completed trial.

When the experiment finishes, you can sort the table by column or filter trials by using the Filters pane. You can also record observations by adding annotations to the results table. For more information, see Sort, Filter, and Annotate Experiment Results.

To test the performance of an individual trial, export the trained network or the training information for the trial. On the Experiment Manager toolstrip, select Export > Trained Network or Export > Training Information, respectively. For more information, see net and info.

To close the experiment, in the Experiment Browser pane, right-click the name of the project and select Close Project. Experiment Manager closes all of the experiments and results contained in the project.

This example shows how to use the experiment template for image regression by sweeping hyperparameters. With this template, you can quickly set up a built-in training experiment that uses the trainNetwork function. For another example of solving a regression problem with Experiment Manager, see Create a Deep Learning Experiment for Regression. For more information on an alternative strategy to sweeping hyperparameters, see Tune Experiment Hyperparameters by Using Bayesian Optimization.

Open the example to load a project with a preconfigured experiment that you can inspect and run. To open the experiment, in the Experiment Browser pane, double-click the name of the experiment (Experiment1).

Alternatively, you can configure the experiment yourself by following these steps.

1. Open Experiment Manager. A dialog box provides links to the getting started tutorials and your recent projects, as well as buttons to create a new project or open an example from the documentation.

2. Under New, select Project. A dialog box lists several templates that support workflows including image classification, image regression, sequence classification, semantic segmentation, and custom training loops.

3. Under Image Regression Experiments, select Image Regression by Sweeping Hyperparameters.

4. Specify the name and location for the new project. Experiment Manager opens a new experiment in the project. The Experiment pane displays the description, hyperparameters, setup function, and metrics that define the experiment.

5. In the Description field, enter a description of the experiment:

Regression to predict angles of rotation of digits, using various initial learning rates.

6. Under Hyperparameters, replace the value of myInitialLearnRate with 0.001:0.001:0.006. Verify that Strategy is set to Exhaustive Sweep.

7. Under Setup Function, click Edit. The setup function opens in MATLAB Editor. The setup function specifies the training data, network architecture, and training options for the experiment. In this experiment, the setup function has three sections.

  • Load Training Data defines the training and validation data for the experiment as 4-D arrays. The training and validation data each consist of 5000 images from the Digits data set. Each image shows a digit from 0 to 9, rotated by a certain angle. The regression values correspond to the angles of rotation. For more information on this data set, see Image Data Sets.

  • Define Network Architecture defines the architecture for a simple convolutional neural network for deep learning regression.

  • Specify Training Options defines a trainingOptions object for the experiment. In this experiment, the setup function loads the values for the initial learning rate from the myInitialLearnRate entry in the hyperparameter table.

When you run the experiment, Experiment Manager trains the network defined by the setup function six times. Each trial uses one of the learning rates specified in the hyperparameter table. By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox, you can run multiple trials at the same time. For best results, before you run your experiment, start a parallel pool with as many workers as GPUs. For more information, see Use Experiment Manager to Train Networks in Parallel and GPU Support by Release (Parallel Computing Toolbox).

  • To run one trial of the experiment at a time, on the Experiment Manager toolstrip, click Run.

  • To run multiple trials at the same time, click Use Parallel and then Run. If there is no current parallel pool, Experiment Manager starts one using the default cluster profile. Experiment Manager then executes multiple simultaneous trials, depending on the number of parallel workers available.

A table of results displays the root mean squared error (RMSE) and loss for each trial.

While the experiment is running, click Training Plot to display the training plot and track the progress of each trial. You can also monitor the training progress in the MATLAB Command Window.

When the experiment finishes, you can sort the table by column or filter trials by using the Filters pane. You can also record observations by adding annotations to the results table. For more information, see Sort, Filter, and Annotate Experiment Results.

To test the performance of an individual trial, export the trained network or the training information for the trial. On the Experiment Manager toolstrip, select Export > Trained Network or Export > Training Information, respectively. For more information, see net and info.

To close the experiment, in the Experiment Browser pane, right-click the name of the project and select Close Project. Experiment Manager closes all of the experiments and results contained in the project.

This example shows how to use the training experiment template for image classification using a custom training loop. With this template, you can quickly set up a custom training experiment.

Open the example to load a project with a preconfigured experiment that you can inspect and run. To open the experiment, in the Experiment Browser pane, double-click the name of the experiment (Experiment1).

Alternatively, you can configure the experiment yourself by following these steps.

1. Open Experiment Manager. A dialog box provides links to the getting started tutorials and your recent projects, as well as buttons to create a new project or open an example from the documentation.

2. Under New, select Project. A dialog box lists several templates that support workflows including image classification, image regression, sequence classification, semantic segmentation, and custom training loops.

3. Under Image Classification Experiments, select Image Classification Using Custom Training Loop.

4. Select the location and name for a new project. Experiment Manager opens a new experiment in the project. The Experiment pane displays the description, hyperparameters, and training function that define the experiment.

3. In the Description field, enter a description of the experiment:

Classification of digits, using various initial learning rates.

4. Under Hyperparameters, replace the value of myInitialLearnRate with 0.0025:0.0025:0.015. Verify that Strategy is set to Exhaustive Sweep.

5. Under Training Function, click Edit. The training function opens in MATLAB Editor. The training function specifies the training data, network architecture, training options, and training procedure used by the experiment. In this experiment, the training function has four sections.

  • Load Training Data defines the training data for the experiment as 4-D arrays. The experiment uses the Digits data set, which consists of 5,000 28-by-28 pixel grayscale images of digits from 0 to 9, categorized by the digit they represent. For more information on this data set, see Image Data Sets.

  • Define Network Architecture defines the architecture for a simple convolutional neural network for deep learning classification. To train the network with a custom training loop, the training function represents the network as a dlnetwork object.

  • Specify Training Options defines the training options used by the experiment. In this experiment, the training function loads the values for the initial learning rate from the myInitialLearnRate entry in the hyperparameter table.

  • Train Model defines the custom training loop used by the experiment. For each epoch, the custom training loop shuffles the data and iterates over mini-batches of data. For each mini-batch, the custom training loop evaluates the model gradients, state, and loss, determines the learning rate for the time-based decay learning rate schedule, and updates the network parameters. To track the progress of the training and record the value of the training loss, the training function uses the experiments.Monitor object monitor.

When you run the experiment, Experiment Manager trains the network defined by the training function six times. Each trial uses one of the learning rates specified in the hyperparameter table. By default, Experiment Manager runs one trial at a time. If you have Parallel Computing Toolbox, you can run multiple trials at the same time. For best results, before you run your experiment, start a parallel pool with as many workers as GPUs. For more information, see Use Experiment Manager to Train Networks in Parallel and GPU Support by Release (Parallel Computing Toolbox).

  • To run one trial of the experiment at a time, on the Experiment Manager toolstrip, click Run.

  • To run multiple trials at the same time, click Use Parallel and then Run. If there is no current parallel pool, Experiment Manager starts one using the default cluster profile. Experiment Manager then executes multiple simultaneous trials, depending on the number of parallel workers available.

A table of results displays the training loss for each trial.

While the experiment is running, click Training Plot to display the training plot and track the progress of each trial.

When the experiment finishes, you can sort the table by column or filter trials by using the Filters pane. You can also record observations by adding annotations to the results table. For more information, see Sort, Filter, and Annotate Experiment Results.

To test the performance of an individual trial, export the training output for the trial. On the Experiment Manager toolstrip, select Export. In this experiment, the training output is a structure that contains the values of the training loss and the trained network.

To close the experiment, in the Experiment Browser pane, right-click the name of the project and select Close Project. Experiment Manager closes all of the experiments and results contained in the project.

This example shows how to set up a built-in training experiment using the Experiment Manager app. Built-in training experiments rely on the trainNetwork function and support workflows such as image classification, image regression, sequence classification, and semantic segmentation.

Built-in training experiments consist of a description, a table of hyperparameters, a setup function, and a collection of metric functions to evaluate the results of the experiment.

In the Description field, enter a description of the experiment.

Under Hyperparameters, select the strategy to use for your experiment.

  • To sweep through a range of hyperparameter values, set Strategy to Exhaustive Sweep. In the hyperparameter table, specify the values of the hyperparameters used in the experiment. You can specify hyperparameter values as scalars or vectors with numeric, logical, or string values. For example, these are valid hyperparameter specifications:

    • 0.01

    • 0.01:0.01:0.05

    • [0.01 0.02 0.04 0.08]

    • ["sgdm" "rmsprop" "adam"]

    When you run the experiment, Experiment Manager trains the network using every combination of the hyperparameter values specified in the table.

  • To find optimal training options by using Bayesian optimization, set Strategy to Bayesian Optimization. In the hyperparameter table, specify these properties of the hyperparameters used in the experiment:

    • Range — Enter a two-element vector that gives the lower bound and upper bound of a real- or integer-valued hyperparameter, or a string array or cell array that lists the possible values of a categorical hyperparameter.

    • Type — Select real (real-valued hyperparameter), integer (integer-valued hyperparameter), or categorical (categorical hyperparameter).

    • Transform — Select none (no transform) or log (logarithmic transform). For log, the hyperparameter must be real or integer and positive. With this option, the hyperparameter is searched and modeled on a logarithmic scale.

    When you run the experiment, Experiment Manager searches for the best combination of hyperparameters. Each trial in the experiment uses a new combination of hyperparameter values based on the results of the previous trials.

    To specify the duration of your experiment, under Bayesian Optimization Options, enter the maximum time (in seconds) and the maximum number of trials to run. Note that the actual run time and number of trials in your experiment can exceed these settings because Experiment Manager checks these options only when a trial finishes executing.

    Bayesian optimization requires Statistics and Machine Learning Toolbox. For more information, see Tune Experiment Hyperparameters by Using Bayesian Optimization.

The Setup Function configures the training data, network architecture, and training options for the experiment. The input to the setup function is a structure with fields from the hyperparameter table. The output of the setup function must match the input of the trainNetwork function. This table lists the supported signatures for the setup function.

Goal of ExperimentSetup Function Signature
Train a network for image classification and regression tasks using the images and responses specified by images and the training options defined by options.
function [images,layers,options] = Experiment_setup(params)
...
end
Train a network using the images specified by images and responses specified by responses.
function [images,responses,layers,options] = Experiment_setup(params)
...
end
Train a network for sequence or time-series classification and regression tasks (for example, an LSTM or GRU network) using the sequences and responses specified by sequences.
function [sequences,layers,options] = Experiment_setup(params)
...
end
Train a network using the sequences specified by sequences and responses specified by responses.
function [sequences,reponses,layers,options] = Experiment_setup(params)
...
end
Train a network for feature classification or regression tasks (for example, a multilayer perceptron, or MLP, network) using the feature data and responses specified by features.
function [features,layers,options] = Experiment_setup(params)
...
end
Train a network using the feature data specified by features and responses specified by responses.
function [features,responses,layers,options] = Experiment_setup(params)
...
end

Note

Experiment Manager does not support the execution of multiple trials in parallel when you set the training option ExecutionEnvironment to "multi-gpu" or "parallel" or when you enable the training option DispatchInBackground. Use these options to speed up your training only if you intend to run one trial of your experiment at a time. For more information, see Use Experiment Manager to Train Networks in Parallel.

The Metrics section specifies functions to evaluate the results of the experiment. The input to a metric function is a structure with three fields:

The output of a metric function must be a scalar number, a logical value, or a string.

If your experiment uses Bayesian optimization, select a metric to optimize from the Optimize list. In the Direction list, specify that you want to Maximize or Minimize this metric. Experiment Manager uses this metric to determine the best combination of hyperparameters for your experiment. You can choose a standard training or validation metric (such as accuracy, RMSE, or loss) or a custom metric from the table.

This example shows how to set up a custom training experiment using the Experiment Manager app. Custom training experiments support workflows that require a training function other than trainNetwork. These workflows include:

  • Training a network that is not defined by a layer graph.

  • Training a network using a custom learning rate schedule.

  • Updating the learnable parameters of a network by using a custom function.

  • Training a generative adversarial network (GAN).

  • Training a Siamese network.

Custom training experiments consist of a description, a table of hyperparameters, and a training function.

In the Description field, enter a description of the experiment.

Under Hyperparameters, select the strategy to use for your experiment.

  • To sweep through a range of hyperparameter values, set Strategy to Exhaustive Sweep. In the hyperparameter table, specify the values of the hyperparameters used in the experiment. You can specify hyperparameter values as scalars or vectors with numeric, logical, or string values. For example, these are valid hyperparameter specifications:

    • 0.01

    • 0.01:0.01:0.05

    • [0.01 0.02 0.04 0.08]

    • ["sgdm" "rmsprop" "adam"]

    When you run the experiment, Experiment Manager trains the network using every combination of the hyperparameter values specified in the table.

  • To find optimal training options by using Bayesian optimization, set Strategy to Bayesian Optimization. In the hyperparameter table, specify these properties of the hyperparameters used in the experiment:

    • Range — Enter a two-element vector that gives the lower bound and upper bound of a real- or integer-valued hyperparameter, or a string array or cell array that lists the possible values of a categorical hyperparameter.

    • Type — Select real (real-valued hyperparameter), integer (integer-valued hyperparameter), or categorical (categorical hyperparameter).

    • Transform — Select none (no transform) or log (logarithmic transform). For log, the hyperparameter must be real or integer and positive. With this option, the hyperparameter is searched and modeled on a logarithmic scale.

    When you run the experiment, Experiment Manager searches for the best combination of hyperparameters. Each trial in the experiment uses a new combination of hyperparameter values based on the results of the previous trials.

    To specify the duration of your experiment, under Bayesian Optimization Options, enter the maximum time (in seconds) and the maximum number of trials to run. Note that the actual run time and number of trials in your experiment can exceed these settings because Experiment Manager checks these options only when a trial finishes executing.

    Bayesian optimization requires Statistics and Machine Learning Toolbox. For more information, see Use Bayesian Optimization in Custom Training Experiments.

The Training Function specifies the training data, network architecture, training options, and training procedure used by the experiment. The inputs to the training function are:

  • A structure with fields from the hyperparameter table

  • An experiments.Monitor object that you can use to track the progress of the training, update information fields in the results table, record values of the metrics used by the training, and produce training plots

Experiment Manager saves the output of the training function, so you can export it to the MATLAB workspace when the training is complete.

Note

Both information and metric columns display numerical values in the results table for your experiment. Additionally, metric values are recorded in the training plot. Use information columns for values that you want to display in the results table but not in the training plot.

If your experiment uses Bayesian optimization, in the Metrics section, under Optimize, enter the name of a metric to optimize. In the Direction list, specify that you want to Maximize or Minimize this metric. Experiment Manager uses this metric to determine the best combination of hyperparameters for your experiment. You can choose any metric that you define using the experiments.Monitor object for the training function.

Experiment Manager provides two options for interrupting experiments:

  • Stop marks any running trials as Stopped and saves their results. When the experiment stops, you can display the training plot and export the training output for these trials.

  • Cancel marks any running trials as Canceled and discards their results. When the experiment stops, you cannot display the training plot or export the training output for these trials.

Both options save the results of any completed trials and cancel any queued trials. Typically, Cancel is faster than Stop.

Instead of stopping an experiment, you can stop an individual trial that is running or cancel an individual queued trial. In the Progress column of the results table, click the Stop or Cancel button for the trial.

Results table showing Stop icon for a running trial.

When the training is complete, you can restart a trial that you stopped or canceled. In the Progress column of the results table, click the Restart button for the trial.

Results table showing Restart icon for a running trial.

Alternatively, to restart all the trials that you canceled, in the Experiment Manager toolstrip, click Restart All Canceled .

Note

Experiments that use Bayesian optimization support only the Cancel option. In addition, these experiments do not support restarting of canceled trials.

Custom training experiments that use exhaustive sweep support only the Stop option.

This example shows how to compare your results and record your observations after running an experiment.

When you run an experiment, Experiment Manager trains the network defined by the setup function multiple times. Each trial uses a different combination of hyperparameters. When the experiment finishes, a table displays training and validation metrics (such as accuracy, RMSE, and loss) for each trial. To compare the results of an experiment, you can use these metrics to sort the results table and filter trials.

To sort the trials in the results table, use the drop-down menu for the column corresponding to a training or validation metric.

  1. Point to the header of a column by which you want to sort.

  2. Click the triangle icon.

  3. Select Sort in Ascending Order or Sort in Descending Order.

    Results table showing drop down menu for the Validation Accuracy column.

To filter trials from the results table, use the Filters pane.

  1. On the Experiment Manager toolstrip, select Filters.

    The Filters pane shows histograms for the numeric metrics in the results table. To remove a histogram from the Filters pane, in the results table, open the drop-down menu for the corresponding column and clear the Show Filter check box.

  2. Adjust the sliders under the histogram for the training or validation metric by which you want to filter.

    Histogram for Validation Accuracy, with filter sliders set to 62 and 64.88.

    The results table shows only the trials with a metric value in the selected range.

    Results table showing only trials with Validation Accuracy greater than 62.

  3. To restore all of the trials in the results table, close the Experiment Result pane and reopen the results from the Experiment Browser pane.

To record observations about the results of your experiment, add an annotation.

  1. Right-click a cell in the results table and select Add Annotation. Alternatively, select a cell in the results table and, on the Experiment Manager toolstrip, select Annotations > Add Annotation.

    Results table showing drop down menu for the Validation Accuracy cell with the largest value.

  2. In the Annotations pane, enter your observations in the text box.

    Annotation text says, "Best accuracy so far. Train for longer to improve this value."

    You can add multiple annotations for each cell in the results table. Each annotation is marked with a time stamp.

  3. To highlight the cell that corresponds to an annotation, click the link above the annotation.

To open the Annotations pane and view all of your annotations, on the Experiment Manager toolstrip, select Annotations > View Annotations.

This example shows how to inspect the configuration of an experiment that produced a given result.

After you run an experiment, you can open the Experiment Source pane to see a read-only copy of the experiment description and hyperparameter table, as well as links to all of the functions used by the experiment. You can use the information in this pane to track the configuration of data, network, and training options that produces each of your results.

For instance, suppose that you run an experiment multiple times. Each time that you run the experiment, you change the contents of the setup function but always use the same function name. The first time that you run the experiment, you use the default network provided by the experiment template for image classification. The second time that you run the experiment, you modify the setup function to load a pretrained GoogLeNet network, replacing the final layers with new layers for transfer learning. For an example that uses these two network architectures, see Create a Deep Learning Experiment for Classification.

On the first Experiment Result pane, click the View Experiment Source link. Experiment Manager opens an Experiment Source pane that contains the experiment definition that produced the first set of results. Click the link at the bottom of the pane to open the setup function that you used the first time you ran the experiment. You can copy this setup function to rerun the experiment using the simple classification network.

On the second Experiment Result pane, click the View Experiment Source link. Experiment Manager opens an Experiment Source pane that contains the experiment definition that produced the second set of results. Click the link at the bottom of the pane to open the setup function that you used the second time you ran the experiment. You can copy this setup function to rerun the experiment using transfer learning.

Experiment Manager stores a copy of all the functions that you use, so you do not have to manually rename these functions when you modify and rerun an experiment.

Related Examples

Tips

Introduced in R2020a