Run Tests in Parallel with Custom Plugin
This example shows how to create a custom plugin that supports
running tests in parallel. The custom plugin counts the number of passing and
failing assertions for a test suite. To extend the TestRunner
,
the plugin overrides select methods of the
matlab.unittest.plugins.TestRunnerPlugin
class. Additionally,
to support running tests in parallel, the plugin subclasses the matlab.unittest.plugins.Parallelizable
interface. To run tests in
parallel, you need Parallel Computing Toolbox™.
Create Plugin Class
In a file in your current folder, create the parallelizable plugin class
AssertionCountingPlugin
, which inherits from both the
TestRunnerPlugin
and Parallelizable
classes. For the complete code for AssertionCountingPlugin
,
see Plugin
Class Definition Summary.
To keep track of the number of passing and failing assertions, define four
read-only properties within a properties
block. Each
MATLAB® worker on the current parallel pool uses
NumPassingAssertions
and
NumFailingAssertions
to track the number of passing and
failing assertions when running a portion of the TestSuite
array. The MATLAB client uses FinalizedNumPassingAssertions
and
FinalizedNumFailingAssertions
to aggregate the results
from different workers and to report the total number of passing and failing
assertions at the end of the test session.
properties (SetAccess = private) NumPassingAssertions NumFailingAssertions FinalizedNumPassingAssertions FinalizedNumFailingAssertions end
Extend Running of Test Session
To extend the running of the entire TestSuite
array,
override the runSession
method of
TestRunnerPlugin
in a methods
block
with protected
access. The testing framework evaluates this
method one time on the client.
methods (Access = protected) function runSession(plugin, pluginData) suiteSize = numel(pluginData.TestSuite); fprintf('## Running a total of %d tests\n\n', suiteSize); plugin.FinalizedNumPassingAssertions = 0; plugin.FinalizedNumFailingAssertions = 0; runSession@matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); fprintf('## Done running tests\n') plugin.printAssertionSummary() end end
runSession
displays information about the total number of
Test
elements, initializes the properties used by the
plugin to generate text output, and invokes the superclass method to trigger the
entire test run. After the framework completes evaluating the superclass method,
runSession
displays the assertion count summary by
calling the helper method printAssertionSummary
(see Define
Helper Methods).
Extend Creation of Shared Test Fixtures and TestCase Instances
Add listeners to AssertionPassed
and
AssertionFailed
events to count the assertions. To add
these listeners, extend the methods used by the testing framework to create the
test content. The test content includes TestCase
instances
for each Test
element, class-level
TestCase
instances for the
TestClassSetup
and TestClassTeardown
method blocks, and Fixture
instances used when a
TestCase
class has the
SharedTestFixtures
attribute.
Invoke the corresponding superclass method when you override the creation
methods. The creation methods return the content that the testing framework
creates for each of their respective contexts. When implementing one of these
methods using the incrementPassingAssertionsCount
and
incrementFailingAssertionsCount
helper
methods, add the listeners required by the plugin to the returned
Fixture
or TestCase
instance.
Add these creation methods to a methods
block with
protected
access.
methods (Access = protected) function fixture = createSharedTestFixture(plugin, pluginData) fixture = createSharedTestFixture@... matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); fixture.addlistener('AssertionPassed', ... @(~,~)plugin.incrementPassingAssertionsCount); fixture.addlistener('AssertionFailed', ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestClassInstance(plugin, pluginData) testCase = createTestClassInstance@... matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); testCase.addlistener('AssertionPassed', ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener('AssertionFailed', ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestMethodInstance(plugin, pluginData) testCase = createTestMethodInstance@... matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); testCase.addlistener('AssertionPassed', ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener('AssertionFailed', ... @(~,~)plugin.incrementFailingAssertionsCount); end end
Extend Running of Test Suite Portion
The testing framework divides the entire TestSuite
array
into different groups and assigns them to workers for processing. Each worker
can run one or more test suite portions. To customize the behavior of workers,
override the runTestSuite
method of
TestRunnerPlugin
in a methods
block
with protected
access.
Extend the TestRunner
to display the identifier of each
test group that a worker runs along with the number of Test
elements within the group. Additionally, store the number of passing and failing
assertions in a buffer so that the client can retrieve these values to produce
the finalized results. Like all plugin methods, the
runTestSuite
method requires you to invoke the
corresponding superclass method at an appropriate point. In this case, invoke
the superclass method after initializing the properties and before storing the
worker data. The testing framework evaluates runTestSuite
on
the workers as many times as the number of test suite portions.
methods (Access = protected) function runTestSuite(plugin, pluginData) suiteSize = numel(pluginData.TestSuite); groupNumber = pluginData.Group; fprintf('### Running a total of %d tests in group %d\n', ... suiteSize, groupNumber); plugin.NumPassingAssertions = 0; plugin.NumFailingAssertions = 0; runTestSuite@matlab.unittest.plugins.TestRunnerPlugin(... plugin, pluginData); assertionStruct = struct('Passing', plugin.NumPassingAssertions, ... 'Failing', plugin.NumFailingAssertions); plugin.storeIn(pluginData.CommunicationBuffer, assertionStruct); end end
To store test-specific data, the implementation of
runTestSuite
contains a call to the storeIn
method of the Parallelizable
interface.
Use storeIn
along with retrieveFrom
when workers must report to the client. In this
example, after returning from the superclass method,
NumPassingAssertions
and
NumFailingAssertions
contain the number of passing and
failing assertions corresponding to a group of tests. Because
storeIn
accepts the worker data as only one input
argument, assertionStruct
groups the assertion counts using
two fields.
Extend Reporting of Finalized Test Suite Portion
Extend reportFinalizedSuite
to aggregate the assertion
counts by retrieving test data for each finalized test suite portion. To
retrieve the stored assertionStruct
for a test suite portion,
invoke the retrieveFrom
method within the scope of
reportFinalizedSuite
. Add the field values to the
corresponding class properties, and invoke the superclass method. The testing
framework evaluates this method on the client as many times as the number of
test suite portions.
methods (Access = protected) function reportFinalizedSuite(plugin, pluginData) assertionStruct = plugin.retrieveFrom(pluginData.CommunicationBuffer); plugin.FinalizedNumPassingAssertions = ... plugin.FinalizedNumPassingAssertions + assertionStruct.Passing; plugin.FinalizedNumFailingAssertions = ... plugin.FinalizedNumFailingAssertions + assertionStruct.Failing; reportFinalizedSuite@matlab.unittest.plugins.TestRunnerPlugin(... plugin, pluginData); end end
Define Helper Methods
In a methods
block with private
access,
define three helper methods. These methods increment the number of passing or
failing assertions within each running test suite portion, and print the
assertion count summary.
methods (Access = private) function incrementPassingAssertionsCount(plugin) plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1; end function incrementFailingAssertionsCount(plugin) plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1; end function printAssertionSummary(plugin) fprintf('%s\n', repmat('_', 1, 30)) fprintf('Total Assertions: %d\n', plugin.FinalizedNumPassingAssertions + ... plugin.FinalizedNumFailingAssertions) fprintf('\t%d Passed, %d Failed\n', plugin.FinalizedNumPassingAssertions, ... plugin.FinalizedNumFailingAssertions) end end
Plugin Class Definition Summary
The following code provides the complete contents of
AssertionCountingPlugin
.
classdef AssertionCountingPlugin < ... matlab.unittest.plugins.TestRunnerPlugin & ... matlab.unittest.plugins.Parallelizable properties (SetAccess = private) NumPassingAssertions NumFailingAssertions FinalizedNumPassingAssertions FinalizedNumFailingAssertions end methods (Access = protected) function runSession(plugin, pluginData) suiteSize = numel(pluginData.TestSuite); fprintf('## Running a total of %d tests\n\n', suiteSize); plugin.FinalizedNumPassingAssertions = 0; plugin.FinalizedNumFailingAssertions = 0; runSession@matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); fprintf('## Done running tests\n') plugin.printAssertionSummary() end function fixture = createSharedTestFixture(plugin, pluginData) fixture = createSharedTestFixture@... matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); fixture.addlistener('AssertionPassed', ... @(~,~)plugin.incrementPassingAssertionsCount); fixture.addlistener('AssertionFailed', ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestClassInstance(plugin, pluginData) testCase = createTestClassInstance@... matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); testCase.addlistener('AssertionPassed', ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener('AssertionFailed', ... @(~,~)plugin.incrementFailingAssertionsCount); end function testCase = createTestMethodInstance(plugin, pluginData) testCase = createTestMethodInstance@... matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData); testCase.addlistener('AssertionPassed', ... @(~,~)plugin.incrementPassingAssertionsCount); testCase.addlistener('AssertionFailed', ... @(~,~)plugin.incrementFailingAssertionsCount); end function runTestSuite(plugin, pluginData) suiteSize = numel(pluginData.TestSuite); groupNumber = pluginData.Group; fprintf('### Running a total of %d tests in group %d\n', ... suiteSize, groupNumber); plugin.NumPassingAssertions = 0; plugin.NumFailingAssertions = 0; runTestSuite@matlab.unittest.plugins.TestRunnerPlugin(... plugin, pluginData); assertionStruct = struct('Passing', plugin.NumPassingAssertions, ... 'Failing', plugin.NumFailingAssertions); plugin.storeIn(pluginData.CommunicationBuffer, assertionStruct); end function reportFinalizedSuite(plugin, pluginData) assertionStruct = plugin.retrieveFrom(pluginData.CommunicationBuffer); plugin.FinalizedNumPassingAssertions = ... plugin.FinalizedNumPassingAssertions + assertionStruct.Passing; plugin.FinalizedNumFailingAssertions = ... plugin.FinalizedNumFailingAssertions + assertionStruct.Failing; reportFinalizedSuite@matlab.unittest.plugins.TestRunnerPlugin(... plugin, pluginData); end end methods (Access = private) function incrementPassingAssertionsCount(plugin) plugin.NumPassingAssertions = plugin.NumPassingAssertions + 1; end function incrementFailingAssertionsCount(plugin) plugin.NumFailingAssertions = plugin.NumFailingAssertions + 1; end function printAssertionSummary(plugin) fprintf('%s\n', repmat('_', 1, 30)) fprintf('Total Assertions: %d\n', plugin.FinalizedNumPassingAssertions + ... plugin.FinalizedNumFailingAssertions) fprintf('\t%d Passed, %d Failed\n', plugin.FinalizedNumPassingAssertions, ... plugin.FinalizedNumFailingAssertions) end end end
Create Example Test Class
In your current folder, create a file named ExampleTest.m
containing the following parameterized test class. This class results in 300
Test
elements, 100 of which are assertion tests that
compare pseudorandom integers between 1 and 10.
classdef ExampleTest < matlab.unittest.TestCase properties (TestParameter) num1 = repmat({@()randi(10)}, 1, 10); num2 = repmat({@()randi(10)}, 1, 10); end methods(Test) function testAssert(testCase, num1, num2) testCase.assertNotEqual(num1(), num2()) end function testVerify(testCase, num1, num2) testCase.verifyNotEqual(num1(), num2()) end function testAssume(testCase, num1, num2) testCase.assumeNotEqual(num1(), num2()) end end end
Add Plugin to TestRunner and Run Tests
At the command prompt, create a test suite from the
ExampleTest
class.
import matlab.unittest.TestSuite import matlab.unittest.TestRunner suite = TestSuite.fromClass(?ExampleTest);
Create a TestRunner
instance with no plugins. This code
creates a silent runner and gives you control over the installed plugins.
runner = TestRunner.withNoPlugins;
Add AssertionCountingPlugin
to the runner and run the tests
in parallel. You can also run the same tests in serial mode if you invoke the
run
method on the runner.
runner.addPlugin(AssertionCountingPlugin) result = runner.runInParallel(suite);
Starting parallel pool (parpool) using the 'local' profile ... Connected to the parallel pool (number of workers: 6). ## Running a total of 300 tests Split tests into 18 groups and running them on 6 workers. ---------------- Finished Group 6 ---------------- ### Running a total of 18 tests in group 6 ---------------- Finished Group 1 ---------------- ### Running a total of 20 tests in group 1 ---------------- Finished Group 2 ---------------- ### Running a total of 20 tests in group 2 ---------------- Finished Group 3 ---------------- ### Running a total of 19 tests in group 3 ---------------- Finished Group 4 ---------------- ### Running a total of 19 tests in group 4 ---------------- Finished Group 5 ---------------- ### Running a total of 18 tests in group 5 ---------------- Finished Group 7 ---------------- ### Running a total of 18 tests in group 7 ---------------- Finished Group 8 ---------------- ### Running a total of 17 tests in group 8 ---------------- Finished Group 9 ---------------- ### Running a total of 17 tests in group 9 ----------------- Finished Group 10 ----------------- ### Running a total of 17 tests in group 10 ----------------- Finished Group 11 ----------------- ### Running a total of 16 tests in group 11 ----------------- Finished Group 12 ----------------- ### Running a total of 16 tests in group 12 ----------------- Finished Group 15 ----------------- ### Running a total of 15 tests in group 15 ----------------- Finished Group 14 ----------------- ### Running a total of 15 tests in group 14 ----------------- Finished Group 17 ----------------- ### Running a total of 14 tests in group 17 ----------------- Finished Group 16 ----------------- ### Running a total of 14 tests in group 16 ----------------- Finished Group 13 ----------------- ### Running a total of 15 tests in group 13 ----------------- Finished Group 18 ----------------- ### Running a total of 12 tests in group 18 ## Done running tests ______________________________ Total Assertions: 100 88 Passed, 12 Failed
See Also
matlab.unittest.plugins.TestRunnerPlugin
| matlab.unittest.TestCase
| matlab.unittest.TestRunner
| matlab.unittest.fixtures.Fixture
| matlab.unittest.plugins.Parallelizable
| matlab.unittest.TestSuite
| addlistener
| runInParallel
| matlab.unittest.TestResult