Run Tests in Parallel with Custom Plugin

This example shows how to create a custom plugin that supports running tests in parallel. The custom plugin counts the number of passing and failing assertions for a test suite. To extend the TestRunner, the plugin overrides select methods of the matlab.unittest.plugins.TestRunnerPlugin class. Additionally, to support running tests in parallel, the plugin subclasses the matlab.unittest.plugins.Parallelizable interface. To run tests in parallel, you need Parallel Computing Toolbox™.

Create Plugin Class

In a file in your current folder, create the parallelizable plugin class AssertionCountingPlugin, which inherits from both the TestRunnerPlugin and Parallelizable classes. For the complete code for AssertionCountingPlugin, see Plugin Class Definition Summary.

To keep track of the number of passing and failing assertions, define four read-only properties within a properties block. Each MATLAB^® worker on the current parallel pool uses NumPassingAssertions and NumFailingAssertions to track the number of passing and failing assertions when running a portion of the TestSuite array. The MATLAB client uses FinalizedNumPassingAssertions and FinalizedNumFailingAssertions to aggregate the results from different workers and to report the total number of passing and failing assertions at the end of the test session.

    properties (SetAccess = private)
        NumPassingAssertions
        NumFailingAssertions
        FinalizedNumPassingAssertions
        FinalizedNumFailingAssertions
    end

Extend Running of Test Session

To extend the running of the entire TestSuite array, override the runSession method of TestRunnerPlugin in a methods block with protected access. The testing framework evaluates this method one time on the client.

    methods (Access = protected)
        function runSession(plugin, pluginData)
            suiteSize = numel(pluginData.TestSuite);
            fprintf('## Running a total of %d tests\n\n', suiteSize);
            plugin.FinalizedNumPassingAssertions = 0;
            plugin.FinalizedNumFailingAssertions = 0;

            runSession@matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);

            fprintf('## Done running tests\n')
            plugin.printAssertionSummary()
        end
    end

runSession displays information about the total number of Test elements, initializes the properties used by the plugin to generate text output, and invokes the superclass method to trigger the entire test run. After the framework completes evaluating the superclass method, runSession displays the assertion count summary by calling the helper method printAssertionSummary (see Define Helper Methods).

Extend Creation of Shared Test Fixtures and TestCase Instances

Add listeners to AssertionPassed and AssertionFailed events to count the assertions. To add these listeners, extend the methods used by the testing framework to create the test content. The test content includes TestCase instances for each Test element, class-level TestCase instances for the TestClassSetup and TestClassTeardown method blocks, and Fixture instances used when a TestCase class has the SharedTestFixtures attribute.

Invoke the corresponding superclass method when you override the creation methods. The creation methods return the content that the testing framework creates for each of their respective contexts. When implementing one of these methods using the incrementPassingAssertionsCount and incrementFailingAssertionsCount helper methods, add the listeners required by the plugin to the returned Fixture or TestCase instance.

Add these creation methods to a methods block with protected access.

    methods (Access = protected)
        function fixture = createSharedTestFixture(plugin, pluginData)
            fixture = createSharedTestFixture@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);

            fixture.addlistener('AssertionPassed', ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            fixture.addlistener('AssertionFailed', ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function testCase = createTestClassInstance(plugin, pluginData)
            testCase = createTestClassInstance@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);

            testCase.addlistener('AssertionPassed', ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener('AssertionFailed', ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end

        function testCase = createTestMethodInstance(plugin, pluginData)
            testCase = createTestMethodInstance@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);

            testCase.addlistener('AssertionPassed', ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener('AssertionFailed', ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end
    end

Extend Running of Test Suite Portion

The testing framework divides the entire TestSuite array into different groups and assigns them to workers for processing. Each worker can run one or more test suite portions. To customize the behavior of workers, override the runTestSuite method of TestRunnerPlugin in a methods block with protected access.

Extend the TestRunner to display the identifier of each test group that a worker runs along with the number of Test elements within the group. Additionally, store the number of passing and failing assertions in a buffer so that the client can retrieve these values to produce the finalized results. Like all plugin methods, the runTestSuite method requires you to invoke the corresponding superclass method at an appropriate point. In this case, invoke the superclass method after initializing the properties and before storing the worker data. The testing framework evaluates runTestSuite on the workers as many times as the number of test suite portions.

    methods (Access = protected)
        function runTestSuite(plugin, pluginData)
            suiteSize = numel(pluginData.TestSuite);
            groupNumber = pluginData.Group;
            fprintf('### Running a total of %d tests in group %d\n', ...
                suiteSize, groupNumber);
            plugin.NumPassingAssertions = 0;
            plugin.NumFailingAssertions = 0;

            runTestSuite@matlab.unittest.plugins.TestRunnerPlugin(...
                plugin, pluginData);

            assertionStruct = struct('Passing', plugin.NumPassingAssertions, ...
                'Failing', plugin.NumFailingAssertions);
            plugin.storeIn(pluginData.CommunicationBuffer, assertionStruct);
        end
    end

To store test-specific data, the implementation of runTestSuite contains a call to the storeIn method of the Parallelizable interface. Use storeIn along with retrieveFrom when workers must report to the client. In this example, after returning from the superclass method, NumPassingAssertions and NumFailingAssertions contain the number of passing and failing assertions corresponding to a group of tests. Because storeIn accepts the worker data as only one input argument, assertionStruct groups the assertion counts using two fields.

Extend Reporting of Finalized Test Suite Portion

Extend reportFinalizedSuite to aggregate the assertion counts by retrieving test data for each finalized test suite portion. To retrieve the stored assertionStruct for a test suite portion, invoke the retrieveFrom method within the scope of reportFinalizedSuite. Add the field values to the corresponding class properties, and invoke the superclass method. The testing framework evaluates this method on the client as many times as the number of test suite portions.

    methods (Access = protected)
        function reportFinalizedSuite(plugin, pluginData)
            assertionStruct = plugin.retrieveFrom(pluginData.CommunicationBuffer);
            plugin.FinalizedNumPassingAssertions = ...
                plugin.FinalizedNumPassingAssertions + assertionStruct.Passing;
            plugin.FinalizedNumFailingAssertions = ...
                plugin.FinalizedNumFailingAssertions + assertionStruct.Failing;

            reportFinalizedSuite@matlab.unittest.plugins.TestRunnerPlugin(...
                plugin, pluginData);
        end
    end

Define Helper Methods

In a methods block with private access, define three helper methods. These methods increment the number of passing or failing assertions within each running test suite portion, and print the assertion count summary.

    methods (Access = private)
        function incrementPassingAssertionsCount(plugin)
            plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1;
        end

        function incrementFailingAssertionsCount(plugin)
            plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1;
        end

        function printAssertionSummary(plugin)
            fprintf('%s\n', repmat('_', 1, 30))
            fprintf('Total Assertions: %d\n', plugin.FinalizedNumPassingAssertions + ...
                plugin.FinalizedNumFailingAssertions)
            fprintf('\t%d Passed, %d Failed\n', plugin.FinalizedNumPassingAssertions, ...
                plugin.FinalizedNumFailingAssertions)
        end
    end

Plugin Class Definition Summary

The following code provides the complete contents of AssertionCountingPlugin.

classdef AssertionCountingPlugin < ...
        matlab.unittest.plugins.TestRunnerPlugin & ...
        matlab.unittest.plugins.Parallelizable
    
    properties (SetAccess = private)
        NumPassingAssertions
        NumFailingAssertions
        FinalizedNumPassingAssertions
        FinalizedNumFailingAssertions
    end
    
    methods (Access = protected)
        function runSession(plugin, pluginData)
            suiteSize = numel(pluginData.TestSuite);
            fprintf('## Running a total of %d tests\n\n', suiteSize);
            plugin.FinalizedNumPassingAssertions = 0;
            plugin.FinalizedNumFailingAssertions = 0;
            
            runSession@matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            
            fprintf('## Done running tests\n')
            plugin.printAssertionSummary()
        end
        
        function fixture = createSharedTestFixture(plugin, pluginData)
            fixture = createSharedTestFixture@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            
            fixture.addlistener('AssertionPassed', ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            fixture.addlistener('AssertionFailed', ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end
        
        function testCase = createTestClassInstance(plugin, pluginData)
            testCase = createTestClassInstance@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            
            testCase.addlistener('AssertionPassed', ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener('AssertionFailed', ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end
        
        function testCase = createTestMethodInstance(plugin, pluginData)
            testCase = createTestMethodInstance@...
                matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            
            testCase.addlistener('AssertionPassed', ...
                @(~,~)plugin.incrementPassingAssertionsCount);
            testCase.addlistener('AssertionFailed', ...
                @(~,~)plugin.incrementFailingAssertionsCount);
        end
        
        function runTestSuite(plugin, pluginData)
            suiteSize = numel(pluginData.TestSuite);
            groupNumber = pluginData.Group;
            fprintf('### Running a total of %d tests in group %d\n', ...
                suiteSize, groupNumber);
            plugin.NumPassingAssertions = 0;
            plugin.NumFailingAssertions = 0;
            
            runTestSuite@matlab.unittest.plugins.TestRunnerPlugin(...
                plugin, pluginData);
            
            assertionStruct = struct('Passing', plugin.NumPassingAssertions, ...
                'Failing', plugin.NumFailingAssertions);
            plugin.storeIn(pluginData.CommunicationBuffer, assertionStruct);
        end
        
        
        function reportFinalizedSuite(plugin, pluginData)
            assertionStruct = plugin.retrieveFrom(pluginData.CommunicationBuffer);
            plugin.FinalizedNumPassingAssertions = ...
                plugin.FinalizedNumPassingAssertions + assertionStruct.Passing;
            plugin.FinalizedNumFailingAssertions = ...
                plugin.FinalizedNumFailingAssertions + assertionStruct.Failing;
            
            reportFinalizedSuite@matlab.unittest.plugins.TestRunnerPlugin(...
                plugin, pluginData);
        end
    end
    
    methods (Access = private)
        function incrementPassingAssertionsCount(plugin)
            plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1;
        end
        
        function incrementFailingAssertionsCount(plugin)
            plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1;
        end
        
        function printAssertionSummary(plugin)
            fprintf('%s\n', repmat('_', 1, 30))
            fprintf('Total Assertions: %d\n', plugin.FinalizedNumPassingAssertions + ...
                plugin.FinalizedNumFailingAssertions)
            fprintf('\t%d Passed, %d Failed\n', plugin.FinalizedNumPassingAssertions, ...
                plugin.FinalizedNumFailingAssertions)
        end
    end
end

Create Example Test Class

In your current folder, create a file named ExampleTest.m containing the following parameterized test class. This class results in 300 Test elements, 100 of which are assertion tests that compare pseudorandom integers between 1 and 10.

classdef ExampleTest < matlab.unittest.TestCase
    
    properties (TestParameter)
        num1 = repmat({@()randi(10)}, 1, 10);
        num2 = repmat({@()randi(10)}, 1, 10);
    end
    
    methods(Test)
        function testAssert(testCase, num1, num2)
            testCase.assertNotEqual(num1(), num2())
        end
        function testVerify(testCase, num1, num2)
            testCase.verifyNotEqual(num1(), num2())
        end
        function testAssume(testCase, num1, num2)
            testCase.assumeNotEqual(num1(), num2())
        end
    end
end

Add Plugin to TestRunner and Run Tests

At the command prompt, create a test suite from the ExampleTest class.

import matlab.unittest.TestSuite
import matlab.unittest.TestRunner

suite = TestSuite.fromClass(?ExampleTest);

Create a TestRunner instance with no plugins. This code creates a silent runner and gives you control over the installed plugins.

runner = TestRunner.withNoPlugins;

Add AssertionCountingPlugin to the runner and run the tests in parallel. You can also run the same tests in serial mode if you invoke the run method on the runner.

runner.addPlugin(AssertionCountingPlugin)
result = runner.runInParallel(suite);

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 6).
## Running a total of 300 tests

Split tests into 18 groups and running them on 6 workers.
----------------
Finished Group 6
----------------
### Running a total of 18 tests in group 6

----------------
Finished Group 1
----------------
### Running a total of 20 tests in group 1

----------------
Finished Group 2
----------------
### Running a total of 20 tests in group 2

----------------
Finished Group 3
----------------
### Running a total of 19 tests in group 3

----------------
Finished Group 4
----------------
### Running a total of 19 tests in group 4

----------------
Finished Group 5
----------------
### Running a total of 18 tests in group 5

----------------
Finished Group 7
----------------
### Running a total of 18 tests in group 7

----------------
Finished Group 8
----------------
### Running a total of 17 tests in group 8

----------------
Finished Group 9
----------------
### Running a total of 17 tests in group 9

-----------------
Finished Group 10
-----------------
### Running a total of 17 tests in group 10

-----------------
Finished Group 11
-----------------
### Running a total of 16 tests in group 11

-----------------
Finished Group 12
-----------------
### Running a total of 16 tests in group 12

-----------------
Finished Group 15
-----------------
### Running a total of 15 tests in group 15

-----------------
Finished Group 14
-----------------
### Running a total of 15 tests in group 14

-----------------
Finished Group 17
-----------------
### Running a total of 14 tests in group 17

-----------------
Finished Group 16
-----------------
### Running a total of 14 tests in group 16

-----------------
Finished Group 13
-----------------
### Running a total of 15 tests in group 13

-----------------
Finished Group 18
-----------------
### Running a total of 12 tests in group 18

## Done running tests
______________________________
Total Assertions: 100
	88 Passed, 12 Failed