polyspace.test.TestCaseResult Class

R2026b

Namespace: polyspace.test

(Python) Review test results for a specific test case in a suite

Since R2024b

expand all in page

Description

This Python® class contains test results for a specific test case in a C/C++ test suite.

Creation

Syntax

testCaseResult = testResults.getResults(SuiteName, TestName)[0]
testCaseResult = testSuiteResult.TestCaseResults[testCaseIndex]

Description

testCaseResult = testResults.getResults(SuiteName, TestName)[0] returns the result of the test case named TestName in suite SuiteName:

Here, SuiteName is the name of the test suite and TestName is the name of the test case, both specified as strings.

If the test with name TestName is a parametrized test, getResults returns multiple results as a list of polyspace.test.TestCaseResult objects. You can access the results for each iteration of the parametrized test by indexing into this list.

example

testCaseResult = testSuiteResult.TestCaseResults[testCaseIndex] loads the results of test case number testCaseIndex:

Here, testCaseIndex is the index of the test case specified as an integer starting from 0.

example

Input Arguments

expand all

Name of the test suite containing the test case, specified as a string.

  • To find the names of all test suites in a results object res, enter the following:

    suite_names = [s.Name for s in res.TestSuiteResults]

  • To find the names of all test suites whose names begin with "Sensor", enter the following:

    suite_names = [s.Name for s in res.TestSuiteResults if s.Name.startswith("Sensor")]

Name of the test case whose results you want to retrieve, specified as a string.

  • To find the names of all test cases along with their suite names in a results object res, enter the following:

    suite_and_test_names = [(s.Name, tc.Name) for s in res.TestSuiteResults for tc in s.TestCaseResults]
for suite_name, test_name in suite_and_test_names:
    print(suite_name, test_name)

  • To find the names of all test cases whose names begin with "Sensor" along with their suite names, enter the following:

    suite_and_test_names = [(s.Name, tc.Name) for s in res.TestSuiteResults for tc in s.TestCaseResults if tc.Name.startswith("Sensor")]
for suite_name, test_name in suite_and_test_names:
    print(suite_name, test_name)

Properties

expand all

Name of test case.

Overall execution status of the test case, specified as a polyspace.test.DetailedStatusInfo object with the following properties:

PropertyValuesDescription
FailedTrue or False

The value of this property is True if the test fails or encounters a runt-time error or exception. The value can also be True if the test is expected to fail but passes instead.

In all other cases, the value is False.

IncompleteTrue or False

The value of this property is True if the test does not run to completion because of an assertion or assumption failure. For more information on assertions and assumptions, see Assessment Macros in Polyspace Test API for C/C++ Code.

In all other cases, the value is False.

VerifyFailTrue or False

The value of this property is True if the test contains a VERIFY assessment failure, otherwise the value is False.

For more information on VERIFY assessments, see Assessment Macros in Polyspace Test API for C/C++ Code.

AssertFailTrue or False

The value of this property is True if the test contains an ASSERT assessment failure, otherwise the value is False.

For more information on ASSERT assessments, see Assessment Macros in Polyspace Test API for C/C++ Code.

AssumeFailTrue or False

The value of this property is True if the test contains an ASSUME assessment failure, otherwise the value is False.

For more information on ASSUME assessments, see Assessment Macros in Polyspace Test API for C/C++ Code.

ExceptionTrue or False

The value of this property is True if the test aborted because of an exception being thrown (C++ only), otherwise the value is False.

NotRunTrue or False

The value of this property is True if the test did not start. A test might not start for many reasons. For instance, there might be an assertion or assumption failure or runtime error in the suite setup.

Otherwise, the value is False.

CrashTrue or False

The value of this property is True if the test aborted because of a run-time error, otherwise the value is False.

ExpectFailTrue or False

The value of this property is True if the test was expected to fail, otherwise the value is False. For more information on expected failures, see Write C/C++ Tests with Known Errors or Failures to Support Test-Driven Development.

File containing test definition.

Line number where test definition begins. This is the line containing the PST_SIMPLE_TEST, PST_TEST, or PST_TEST_CONFIG macro (depending on whether your test case is a simple test, a test in a named suite without a configuration, or a test in a named suite with a configuration).

Timing of test case execution and timestamp of execution, specified as a polyspace.test.TimingInfo object with the following properties:

PropertyValueDescription
Timestampdatetime.datetime value

Time stamp of the beginning of test case execution, for example, 2024-Jul-04 10:38:21.

ExecutionTimeFloating-point value

Time taken for test case execution in seconds (up to millisecond precision).

Total number of assessments in the test. This number includes passing and failing assessments.

Number of passing assessments in the test case.

Number of failing assessments in the test case.

Result details for assessments in the test case, returned as a list of polyspace.test.AssessmentResult objects with one object per assessment.

For more information, see polyspace.test.AssessmentResult.

Result details for passing assessments in the test case, returned as a list of polyspace.test.AssessmentResult objects with one object per passing assessment.

For more information, see polyspace.test.AssessmentResult.

Result details for failing assessments in the test case, returned as a list of polyspace.test.AssessmentResult objects with one object per failing assessment.

For more information, see polyspace.test.AssessmentResult.

Messages emitted during test execution that are not related to assessments. For instance, an issue unrelated to assessments can occur during test execution and a warning message might be emitted The message is specified as a polyspace.test.Message object with the following properties:

PropertyDescription
Phase

The value can be one of the following:

  • TestPhase.NONE

  • TestPhase.TEST

  • TestPhase.TEST_CONFIG

  • TestPhase.TEST_SETUP

  • TestPhase.TEST_TEARDOWN

  • TestPhase.SUITE_TEST_SETUP

  • TestPhase.SUITE_TEST_TEARDOWN

Type

The value can be one of the following:

  • MessageType.NONE

  • MessageType.WARNING

  • MessageType.ERROR

  • MessageType.FATAL_ERROR

  • MessageType.INFO

Msg

Actual content of the message.

Events that occurred during test execution specified as a heterogeneous list of polyspace.test.Message and polyspace.test.AssessmentResult objects. The sequence of objects in the list represents the actual sequence in which the events occurred.

For instance, suppose that the body of a test contains one assessment but during test execution, a warning message is emitted before the assessment runs. The Events property contains two objects:

  • Events[0] is a polyspace.test.Message object that contains details of the warning message. For more information on this object, see Messages.

  • Events[1] is a polyspace.test.AssessmentResult object that contains details of the assessment result. For more information on this object, see AssessmentResults.

The Events property allows you to see the sequence in which the events occurred. If you are not interested in the sequence, use the Messages property to check for warnings and other messages, and the AssessmentResults property to retrieve the results of assessments.

Examples

collapse all

This example shows how to retrieve results for a specific test case by name using the getResults method.

In general, you generate and manage Polyspace® Test™ results by using classes from the polyspace.project and polyspace.test modules. Before starting, make sure you can import these modules on a Python shell or in a Python script without errors. For more information, see Set Up Python API for Polyspace.

Open the shipped example project, build, and run tests:

import polyspace.project
import polyspace.test
import os

proj = polyspace.project.Project(os.path.join(
    polyspace.__install_path__, "polyspace", "examples",
    "pstest", "Getting_Started_Example", "Demo_C_PS_Test.psprjx"))
polyspace.test.build(proj)
res = polyspace.test.run(proj)

Retrieve the result of the test case named "addLatestReading_test_1" in suite "setStateTests":

tcRes = res.getResults("setStateTests", "addLatestReading_test_1")[0]
print(f"Test Case: {tcRes.Name}, Failed: {tcRes.Status.Failed}")

This example shows to print the names of failing test cases after running tests in a project.

In general, you generate and manage Polyspace Test results by using classes from the polyspace.project and polyspace.test modules. Before starting, make sure you can import these modules on a Python shell or in a Python script without errors. For more information, see Set Up Python API for Polyspace.

  1. Import the required modules:

    import polyspace.project
import polyspace.test
import os

  2. Add source files and xUnit test files to a project. This example uses some example source and test files available with a Polyspace Test installation. Instead, you can use your own sources and tests.

    examples_path = os.path.join(polyspace.__install_path__, "polyspace", 
                            "examples", "pstest", "Getting_Started_Example")
polyspaceProject = polyspace.project.Project("newProject")
polyspaceProject.Code.Files.add(os.path.join(examples_path, "sources", "utils.c"))
polyspaceProject.IncludePaths.add(os.path.join(examples_path, "includes"))
polyspaceProject.Tests.Files.add(os.path.join(examples_path, "tests", "test.c"))

  3. Run the tests added to the project.

    res = polyspace.test.run(polyspaceProject)

  4. Print the names of failing test cases (along with their suite names).

    for suiteIndex in range(len(res.TestSuiteResults)):
    # Read result of test suite
    resTestSuite = res.TestSuiteResults[suiteIndex]
    if(resTestSuite.Status.Failed):
        for testIndex in range(len(resTestSuite.TestCaseResults)):
            # Read result of test case
            resTestCase = resTestSuite.TestCaseResults[testIndex]
            if(resTestCase.Status.Failed):
                print(resTestSuite.Name + "/" + resTestCase.Name)

    Alternatively, you can simply iterate through the test suites and cases as follows:

    for resTestSuite in res.TestSuiteResults:
    if resTestSuite.Status.Failed:
        for resTestCase in resTestSuite.TestCaseResults:
            if resTestCase.Status.Failed:
                print(f"{resTestSuite.Name}/{resTestCase.Name}")

    If you use the example source and test files, you should see the suite and test name updateStateTests/addLatestReading_test_1_fail printed.

Version History

Introduced in R2024b

See Also

| |

Topics