Run Tests in Parallel with Custom Plugin
This example shows how to create a custom plugin that supports
running tests in parallel. The custom plugin counts the number of passing and
failing assertions for a test suite. To extend a TestRunner instance, the plugin
overrides select methods of the matlab.unittest.plugins.TestRunnerPlugin class. Additionally, to
support running tests in parallel, the plugin subclasses the matlab.unittest.plugins.Parallelizable interface. To run tests in
parallel, you need Parallel Computing Toolbox™.
Create Plugin Class
In a file named AssertionCountingPlugin.m in your current
folder, create the parallelizable plugin class
AssertionCountingPlugin, which inherits from both the
TestRunnerPlugin and Parallelizable classes. For
the complete code of AssertionCountingPlugin, see Plugin Class Definition.
To keep track of the number of passing and failing assertions, define four
read-only properties within a properties block. Each MATLAB® worker on the current parallel pool uses
NumPassingAssertions and
NumFailingAssertions to track the number of passing and
failing assertions when running a portion of the TestSuite array. The MATLAB client uses FinalizedNumPassingAssertions and
FinalizedNumFailingAssertions to aggregate the results from
different workers and to report the total number of passing and failing assertions
at the end of the test session.
properties (SetAccess = private) NumPassingAssertions NumFailingAssertions FinalizedNumPassingAssertions FinalizedNumFailingAssertions end
Extend Running of Test Session
To extend the running of the entire TestSuite array, override the
runSession method in a methods block with
protected access. runSession displays
information about the total number of Test elements, initializes the
properties that the plugin uses to generate text output, and invokes the superclass
method to trigger the entire test run. After the testing framework completes
evaluating the superclass method, runSession displays the assertion
count summary by calling the helper method printAssertionSummary
(see Define Helper Methods). The
framework evaluates this method one time on the client.
methods (Access = protected) function runSession(plugin,pluginData) suiteSize = numel(pluginData.TestSuite); fprintf("## Running a total of %d tests\n\n",suiteSize) plugin.FinalizedNumPassingAssertions = 0; plugin.FinalizedNumFailingAssertions = 0; runSession@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData) fprintf("## Done running tests\n") plugin.printAssertionSummary end end
Extend Creation of Shared Test Fixtures and Test Cases
Add listeners to AssertionPassed and AssertionFailed
events to count the assertions. To add these listeners, extend the methods that the
testing framework uses to create the test content. The test content includes TestCase instances for each
Test element, class-level TestCase instances for
the TestClassSetup and TestClassTeardown
methods blocks, and Fixture instances used when a
TestCase class has the SharedTestFixtures
attribute. Add these creation methods to a methods block with
protected access.
Invoke the corresponding superclass method when you override the creation methods.
The creation methods return the content that the testing framework creates for each
of their respective contexts. When implementing each of these methods using the
incrementPassingAssertionsCount and
incrementFailingAssertionsCount
helper
methods, add the listeners required by the plugin to the returned
Fixture or TestCase instance.
methods (Access = protected) function fixture = createSharedTestFixture(plugin,pluginData) fixture = createSharedTestFixture@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData); fixture.addlistener("AssertionPassed", ... @(~,~)plugin.incrementPassingAssertionsCount); fixture.addlistener("AssertionFailed", ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestClassInstance(plugin,pluginData) testCase = createTestClassInstance@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData); testCase.addlistener("AssertionPassed", ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener("AssertionFailed", ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestMethodInstance(plugin,pluginData) testCase = createTestMethodInstance@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData); testCase.addlistener("AssertionPassed", ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener("AssertionFailed", ... @(~,~)plugin.incrementFailingAssertionsCount); end end
Extend Running of Test Suite Portion
The testing framework divides the entire TestSuite array into
groups and assigns them to workers for processing. Each worker can run one or more
test suite portions. To customize the behavior of workers, override the runTestSuite method in a methods block with
protected access.
Extend the test runner to display the identifier of each test group that a worker
runs along with the number of Test elements within the group.
Additionally, store the number of passing and failing assertions in a communication
buffer so that the client can retrieve these values to produce the finalized
results. Like all plugin methods, the runTestSuite method requires
you to invoke the corresponding superclass method at an appropriate point. In this
case, invoke the superclass method after initializing the properties and before
storing the worker data. The testing framework evaluates
runTestSuite on the workers as many times as the number of test
suite portions.
methods (Access = protected) function runTestSuite(plugin,pluginData) suiteSize = numel(pluginData.TestSuite); groupNumber = pluginData.Group; fprintf("### Running a total of %d tests in group %d\n", ... suiteSize,groupNumber) plugin.NumPassingAssertions = 0; plugin.NumFailingAssertions = 0; runTestSuite@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData) assertionStruct = struct( ... "Passing",plugin.NumPassingAssertions, ... "Failing",plugin.NumFailingAssertions); plugin.storeIn(pluginData.CommunicationBuffer,assertionStruct) end end
To store test-specific data, the implementation of runTestSuite
contains a call to the storeIn
method of the Parallelizable interface. Use storeIn
along with retrieveFrom when workers must report to the client. In this example,
after returning from the superclass method,
NumPassingAssertions and
NumFailingAssertions contain the number of passing and
failing assertions corresponding to a group of tests. Because
storeIn accepts the worker data as only one input argument, the
structure assertionStruct groups the assertion counts using two
fields.
Extend Reporting of Finalized Test Suite Portion
Extend reportFinalizedSuite to aggregate the assertion counts by retrieving
test data for each finalized test suite portion. To retrieve the stored structure
for a test suite portion, invoke the retrieveFrom method within the
scope of reportFinalizedSuite. To
return default data in case the buffer is empty (since R2024a), such as when a fatal
assertion failure prevents all workers from writing to the buffer, specify
DefaultData when invoking the retrieveFrom
method. Add the field values to the corresponding class properties, and invoke the
superclass method. The testing framework evaluates this method on the client as many
times as the number of test suite portions.
methods (Access = protected) function reportFinalizedSuite(plugin,pluginData) assertionStruct = plugin.retrieveFrom( ... pluginData.CommunicationBuffer, ... DefaultData=struct("Passing",0,"Failing",0)); plugin.FinalizedNumPassingAssertions = ... plugin.FinalizedNumPassingAssertions + assertionStruct.Passing; plugin.FinalizedNumFailingAssertions = ... plugin.FinalizedNumFailingAssertions + assertionStruct.Failing; reportFinalizedSuite@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData) end end
Define Helper Methods
In a methods block with private access,
define three helper methods. These methods increment the number of passing or
failing assertions within each running test suite portion and print the assertion
count summary.
methods (Access = private) function incrementPassingAssertionsCount(plugin) plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1; end function incrementFailingAssertionsCount(plugin) plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1; end function printAssertionSummary(plugin) fprintf("%s\n",repmat('_',1,30)) fprintf("Total Assertions: %d\n", ... plugin.FinalizedNumPassingAssertions + ... plugin.FinalizedNumFailingAssertions) fprintf("\t%d Passed, %d Failed\n", ... plugin.FinalizedNumPassingAssertions, ... plugin.FinalizedNumFailingAssertions) end end
Plugin Class Definition
This code provides the complete contents of the
AssertionCountingPlugin class.
classdef AssertionCountingPlugin < ... matlab.unittest.plugins.TestRunnerPlugin & ... matlab.unittest.plugins.Parallelizable properties (SetAccess = private) NumPassingAssertions NumFailingAssertions FinalizedNumPassingAssertions FinalizedNumFailingAssertions end methods (Access = protected) function runSession(plugin,pluginData) suiteSize = numel(pluginData.TestSuite); fprintf("## Running a total of %d tests\n\n",suiteSize) plugin.FinalizedNumPassingAssertions = 0; plugin.FinalizedNumFailingAssertions = 0; runSession@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData) fprintf("## Done running tests\n") plugin.printAssertionSummary end function fixture = createSharedTestFixture(plugin,pluginData) fixture = createSharedTestFixture@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData); fixture.addlistener("AssertionPassed", ... @(~,~)plugin.incrementPassingAssertionsCount); fixture.addlistener("AssertionFailed", ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestClassInstance(plugin,pluginData) testCase = createTestClassInstance@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData); testCase.addlistener("AssertionPassed", ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener("AssertionFailed", ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestMethodInstance(plugin,pluginData) testCase = createTestMethodInstance@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData); testCase.addlistener("AssertionPassed", ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener("AssertionFailed", ... @(~,~)plugin.incrementFailingAssertionsCount); end function runTestSuite(plugin,pluginData) suiteSize = numel(pluginData.TestSuite); groupNumber = pluginData.Group; fprintf("### Running a total of %d tests in group %d\n", ... suiteSize,groupNumber) plugin.NumPassingAssertions = 0; plugin.NumFailingAssertions = 0; runTestSuite@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData) assertionStruct = struct( ... "Passing",plugin.NumPassingAssertions, ... "Failing",plugin.NumFailingAssertions); plugin.storeIn(pluginData.CommunicationBuffer,assertionStruct) end function reportFinalizedSuite(plugin,pluginData) assertionStruct = plugin.retrieveFrom( ... pluginData.CommunicationBuffer, ... DefaultData=struct("Passing",0,"Failing",0)); plugin.FinalizedNumPassingAssertions = ... plugin.FinalizedNumPassingAssertions + assertionStruct.Passing; plugin.FinalizedNumFailingAssertions = ... plugin.FinalizedNumFailingAssertions + assertionStruct.Failing; reportFinalizedSuite@ ... matlab.unittest.plugins.TestRunnerPlugin(plugin,pluginData) end end methods (Access = private) function incrementPassingAssertionsCount(plugin) plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1; end function incrementFailingAssertionsCount(plugin) plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1; end function printAssertionSummary(plugin) fprintf("%s\n",repmat('_',1,30)) fprintf("Total Assertions: %d\n", ... plugin.FinalizedNumPassingAssertions + ... plugin.FinalizedNumFailingAssertions) fprintf("\t%d Passed, %d Failed\n", ... plugin.FinalizedNumPassingAssertions, ... plugin.FinalizedNumFailingAssertions) end end end
Create Test Class
In a file named ExampleTest.m in your current folder, create
the ExampleTest parameterized test class. This class results in 300
tests, 100 of which are assertions to compare pseudorandom integers between 1 and
10.
classdef ExampleTest < matlab.unittest.TestCase properties (TestParameter) num1 = repmat({@()randi(10)},1,10) num2 = repmat({@()randi(10)},1,10) end methods (Test) function testAssert(testCase,num1,num2) testCase.assertNotEqual(num1(),num2()) end function testVerify(testCase,num1,num2) testCase.verifyNotEqual(num1(),num2()) end function testAssume(testCase,num1,num2) testCase.assumeNotEqual(num1(),num2()) end end end
Add Plugin to Test Runner and Run Tests
Create a test suite from the ExampleTest class.
suite = testsuite("ExampleTest");Create a test runner with no plugins. This code creates a silent runner that produces no output.
runner = testrunner("minimal");You can now add any plugins you choose. Add an instance of the
AssertionCountingPlugin class to the runner, and run the tests in
parallel. (You can also run the same tests in serial mode if you invoke the
run method on the runner.)
runner.addPlugin(AssertionCountingPlugin) runner.runInParallel(suite);
## Running a total of 300 tests Split tests into 18 groups and running them on 6 workers. ---------------- Finished Group 1 ---------------- ### Running a total of 20 tests in group 1 ---------------- Finished Group 2 ---------------- ### Running a total of 20 tests in group 2 ---------------- Finished Group 3 ---------------- ### Running a total of 19 tests in group 3 ---------------- Finished Group 4 ---------------- ### Running a total of 19 tests in group 4 ---------------- Finished Group 5 ---------------- ### Running a total of 18 tests in group 5 ---------------- Finished Group 6 ---------------- ### Running a total of 18 tests in group 6 ---------------- Finished Group 7 ---------------- ### Running a total of 18 tests in group 7 ---------------- Finished Group 8 ---------------- ### Running a total of 17 tests in group 8 ---------------- Finished Group 9 ---------------- ### Running a total of 17 tests in group 9 ----------------- Finished Group 10 ----------------- ### Running a total of 17 tests in group 10 ----------------- Finished Group 11 ----------------- ### Running a total of 16 tests in group 11 ----------------- Finished Group 12 ----------------- ### Running a total of 16 tests in group 12 ----------------- Finished Group 13 ----------------- ### Running a total of 15 tests in group 13 ----------------- Finished Group 14 ----------------- ### Running a total of 15 tests in group 14 ----------------- Finished Group 16 ----------------- ### Running a total of 14 tests in group 16 ----------------- Finished Group 15 ----------------- ### Running a total of 15 tests in group 15 ----------------- Finished Group 17 ----------------- ### Running a total of 14 tests in group 17 ----------------- Finished Group 18 ----------------- ### Running a total of 12 tests in group 18 ## Done running tests ______________________________ Total Assertions: 100 88 Passed, 12 Failed
See Also
Functions
Classes
matlab.unittest.plugins.TestRunnerPlugin|matlab.unittest.plugins.Parallelizable|matlab.unittest.TestRunner|matlab.unittest.TestSuite|matlab.unittest.TestCase|matlab.unittest.fixtures.Fixture
