Automate Testing for Scenario Variants of AEB System
This example shows how to assess the functionality of an autonomous emergency braking (AEB) application by varying the scenario inspired by European New Car Assessment Programme (Euro NCAP) Car-to-Pedestrian Nearside Child (CPNC) driving scenario. This example builds on the Autonomous Emergency Braking with Sensor Fusion example.
Autonomous emergency braking is an advanced active safety system that helps drivers avoid or mitigate collisions with other vehicles. The test space for autonomous driving functions is incredibly large, and it is not feasible to validate large numbers of scenarios with real-world driving tests. This example enables you to perform scenario-based testing of an AEB system by generating multiple scenario variants from a seed scenario.
The example shows how to vary ego speed and collision point parameters to generate multiple variants of the CPNC test scenario. The example also shows how to use iterative testing methodology in Simulink® Test Manager to test the generated variants. In this example, you:
Explore the test bench model — The model contains modules for the sensors and environment, sensor fusion and tracking, decision logic, controls, and vehicle dynamics.
Review seed scenario and generate variants — Review the CPNC seed scenario and generate variants of the seed scenario by varying the ego speed and the collision point between the ego vehicle and pedestrian.
Perform iterative testing with scenario variants — Configure Test Manager to simulate the AEB test bench model for each scenario variant using scripted iterative testing with Simulink Test™, assess success criteria, and report results.
Visualize the results and review the generated report — Plot a grid-based visualization to review and analyze the results from iterative testing. You can also export the simulation results to a PDF and review the generated report.
Explore Test Bench Model
This example reuses the
AEBTestBench model from the Autonomous Emergency Braking with Sensor Fusion example.
To explore the test bench model, load the autonomous emergency braking project.
To reduce Command Window output, turn off model predictive controller (MPC) update messages.
Open the system-level simulation test bench model.
This test bench model has these modules:
Sensors and Environment— Subsystem that specifies the road, actors, camera, and radar sensor used for simulation.
Sensor Fusion and Tracking— Algorithm model that fuses vehicle detections from the camera to those from the radar sensor.
AEB Decision Logic— Algorithm model that specifies the lateral and longitudinal decision logic that provides most important object (MIO) related information and ego vehicle reference path information to the controller.
AEB Controller— Algorithm model that specifies the steering angle and acceleration controls.
Vehicle Dynamics— Subsystem that specifies the dynamic model of the ego vehicle.
Metrics Assessment— Subsystem that assesses system-level behavior.
For more details on these components and simulating the test bench model, see the Autonomous Emergency Braking with Sensor Fusion example.
In this example, your focus is on automating the simulation runs for multiple variants of a seed scenario. You create the seed scenario using the Driving Scenario Designer app.
Review Seed Scenario and Generate Variants
helperCreateSeedScenario function generates a cuboid scenario that is compatible with the
AEBTestBench model. This is an open-loop scenario containing two stationary target vehicles on a straight road and a child pedestrian. This scenario is inspired by Euro NCAP CPNC scenario as per the Euro NCAP Test Protocol - AEB VRU systems version 2.0 in which the ego vehicle collides with a pedestrian child who is running behind the obstruction from the nearside. The frontal structure of the vehicle hits the pedestrian.
Plot the open-loop scenario to see the interactions of the ego vehicle and the pedestrian child target.
hFigSeedScenario = helperPlotScenario("helperCreateSeedScenario");
The ego vehicle, which is not under closed-loop control, collides with the pedestrian child. The goal of the closed-loop system is to avoid or mitigate the collision with the pedestrian actor. In the
AEBTestBench model, the ego vehicle has the same initial speed and initial position as in the open-loop scenario.
Close the figure.
Generate Variants of Seed Scenario The
helperCreateSeedScenario function creates a seed scenario in which the speed of the ego vehicle is 60 km/h, and the collision point is at the front-left corner of the ego vehicle. To test this AEB CPNC scenario per Euro NCAP test protocols, you create multiple variants of the seed scenario by varying the ego speed and collision point. Use the
helperCreateSeedScenario function to create the seed scenario.
[seedScenario,egoVehicle] = helperCreateSeedScenario;
getScenarioDescriptor function to create a
scenarioDescriptor object that has the properties of the seed scenario. You can modify the properties of this object to generate scenario variations.
Get the scenario descriptor from the seed scenario
scenarioDescriptor = getScenarioDescriptor(seedScenario);
Define the IDs of the ego vehicle and the target actor that collides with the ego vehicle.
egoID = egoVehicle.ActorID; targetID = 2;
To generate scenario variations, the ego vehicle in the seed scenario must satisfy these criteria:
The ego vehicle must collide with the target vehicle.
The ego vehicle must have consistent motion properties.
The ego trajectory must have at least three waypoints.
The ego vehicle must travel at a constant speed.
To generate variants of the seed scenario, use the
generateVariants function. The function uses these input arguments: the
scenarioDescriptor object, the ego vehicle ID, the target actor ID, a new speed for the ego vehicle, and a new collision point.
Specify the new ego speed in m/s. To convert 20 km/h to m/s, divide by 3.6. This results in a speed value of approximately 5.5 m/s.
egoNewSpeed = 20 * 1/3.6; % m/s
Define the new collision point, as specified in the Generate Scenario Variants for Testing AEB Pedestrian Systems example, which is required for scenario variant generation. For a collision point at the center of the front of the ego vehicle front, specify
newCollisionPoint = 0.5;
To generate a scenario variant, create an empty
variationProperties object, which stores actor and coliision property variations. Then, add a speed variation to the ego actor by using the
varyActorProperties function of the variationProperties object.
variation = variationProperties; varyActorProperties(variation,egoID,Speed=egoNewSpeed);
Add a variation for a collision point between the ego and target actors by using the
varyCollisionProperties function of the variationProperties object.
Generate a variant scenario descriptor from the seed scenario descriptor using the
variantDescriptor = generateVariants(scenarioDescriptor,variation);
Get a variant scenario, as a drivingScenario object, from the scenario variant descriptor object by using the
getScenario function. In the variant scenario, the ego vehicle travels at a speed of 5.5 m/s and hits a pedestrian at the center of the front of the vehicle.
scenarioVariant = getScenario(variantDescriptor,Simulator="DrivingScenario");
Review the generated variant scenario using the Driving Scenario Designer app.
Notice that the generated variant has an ego speed of 5.5 m/s, and the collision occurs at the center of the front of the ego vehicle. You can use this scenario with the
AEBTestBench model to perform closed-loop testing.
Simulink Test enables you to automate the generation and testing of scenario variants with the closed-loop model. Using Simulink Test, you can generate a large number of variations from a seed scenario and catch the failures.
Perform Iterative Testing with Scenario Variants
This example includes a Test Manager configuration for automating testing of the AEB application with the generated variants. Open the
AEBScenarioVariationTest.mldatx test file in the Test Manager.
sltestmgr testFile = sltest.testmanager.load("AEBScenarioVariationTest");
The Simulink scripted iteration framework enables you to automate scenario variant generation and closed-loop testing using the test bench model. The iterative testing script in the
AEBScenarioVariationTest.mldatx file describes how to combine scenario variant generation and closed-loop testing of the AEB application.
setVariable function sets these parameters for each iteration:
v_set— Set Velocity of the ego vehicle.
scenario— Driving Scenario object.
egoVehDyn— Initialization parameters for
Vehicle Dynamicssubsystem, specified as a structure.
addIteration function adds test iterations. For more information on creating scripted iterations, see the Test Iterations (Simulink Test).
After simulating the test case, the Test Manager uses the
helperPlotAEBResults function from the CLEANUP callback to generate the post-simulation plots. For more information on these plots, see Autonomous Emergency Braking with Sensor Fusion example.
Run and Explore Results for CPNC Scenario Variants
To test the system-level model with the generated variants from Simulink Test, use this code:
resultSet = run(testFile); testFileResults = getTestFileResults(resultSet); testSuiteResults = getTestSuiteResults(testFileResults); testCaseResults = getTestCaseResults(testSuiteResults); testIterationResults = getIterationResults(testCaseResults);
Visualize Results and Review Generated Report
testIterationResults output contains the results for all the iterations. To plot the summary of results in the form of a color grid, use the
helperPlotAEBVariantResults function. This helper function plots the grid results with specific colors for each cell that indicate the pass or fail status of closed-loop simulation of the
AEBTestBench model with the generated scenario variants.
In the result grid, the columns represent the collision point values of
0.7. The rows represent nine ego speed values ranging from 10 km/h to 50 km/h in steps of 5 km/h. For each value of ego speed, there are four possible collision points, which results in a total of 36 variations. The right table in the figure shows the color lookup table for the VUT Test Speed vs VUT Impact Speed Range. VUT test speed is the initial speed of the ego vehicle and VUT impact speed is the speed of the ego vehicle at the time of the collision. The difference between the test speed and the impact speed is known as speed reduction. This lookup table shows the speed reduction to infer AEB system performance at the time of the collision. The green column represents significant speed reduction and the red column represents lower speed reduction of the ego vehicle.
Each cell of the grid shows the final ego speed using a color that corresponds to the amount of speed reduction. Notice that all cells of the grid are green, which indicates that the algorithm in the AEB test bench passed all 36 variations of the CPNC scenario.
Use this code to generate the report, for further analysis of specific iterations:
sltest.testmanager.report(testIterationResults,"Report.pdf", ... Title="Automate Testing for CPNC Scenario Variants of AEB System", ... IncludeMATLABFigures=true, ... IncludeErrorMessages=true, ... IncludeTestResults=0, ... LaunchReport=true);
Review Generated Report
Report.pdf. Observe that the
Test environment section shows the platform on which the test is run and the MATLAB version used for testing. The
Summary section shows the outcome of the test and duration of the simulation, in seconds. The
Results section shows pass or fail results based on the assessment criteria. This section also shows the plots logged from the
Enable MPC update messages.
- Vehicle Body 3DOF (Vehicle Dynamics Blockset) | Driving Radar Data Generator | Vision Detection Generator
- Generate Scenario Variants for Testing AEB Pedestrian Systems
- Automate Testing for Autonomous Emergency Braking
- Autonomous Emergency Braking with Sensor Fusion
- Autonomous Emergency Braking with High-Fidelity Vehicle Dynamics
- Autonomous Emergency Braking with Vehicle Variants
- Autonomous Emergency Braking with RoadRunner Scenario