Simulate RoadRunner Scenarios with Actors Modeled in MATLAB and View in Unreal Engine Environment

R2026b

This example uses:

You can use the sim3d.scenario.ActorBehavior class to define a custom RoadRunner actor behavior that enables 3D visualization in Simulation 3D Viewer. The 3D Viewer window opens when you start simulation and runs along with RoadRunner Scenario.

The sim3d.scenario.ActorBehavior class also reduces the amount of code required to set up a cosimulation with MATLAB^®.

These images show the same example scenario in RoadRunner and the Simulation 3D Viewer.

The RoadRunner Scenario viewer shows a 4-way intersection with traffic lights from above. A green SUV is entering the West side of the intersection, traveling East. A yellow Sedan is exiting the North side of the intersection.

The Simulation 3D Viewer shows a view of a 4-way intersection from just above the road on the West side of the intersection looking East. A green SUV is just in front of the camera, entering the intersection from the West side. A yellow Sedan is in front and to the left of the camera, exiting the intersection's North side.

These examples assume that you have:

A RoadRunner license, and the product is installed. For more information, see Install and Activate RoadRunner (RoadRunner).
A RoadRunner Scenario license, and the product is installed.
A Simulink^® 3D Animation™ license, and the product is installed.
A RoadRunner project folder named MyRoadRunnerProject. For more information, see RoadRunner Project and Scene System (RoadRunner).

Build Custom Actor Behavior for 3D Visualization

Straight Line Actor Behavior

In this example, you create a custom behavior, Sim3dEgoVehicle1.m, as a MATLAB class file. In this behavior, the code reads the initial pose and velocity of an actor and updates them to make the actor follow a straight line at constant velocity.

MATLAB Code for Custom Straight Line Behavior with 3D Visualization

classdef Sim3dEgoVehicle1 < sim3d.scenario.ActorBehavior
    % Sim3dEgoVehicle1 implements an ego vehicle behavior using sim3d
    % Copyright 2023 The MathWorks, Inc.
    
    methods(Access = protected)

        function onSimulationStart(self, event)
            % Place startup code here
        end

        function onSimulationStop(self, event)
            % Place shutdown code here
        end

        function onSimulationStep(self, ~)
            % Compute the step position for the sim3d actor 
            self.Actor.Translation = self.Actor.Translation + ...
                self.Velocity * self.getSampleTimeImpl.SampleTime;

            % Update pose for the RoadRunner actor
            self.Pose = self.transform2matrix(self.Actor.Translation, ...
                self.Actor.Rotation, self.Actor.Scale);
        end

    end
end

In the custom behavior:

This code creates a subclass, Sim3dEgoVehicle1, of the sim3d.scenario.ActorBehavior class.
```
classdef Sim3dEgoVehicle1 < sim3d.scenario.ActorBehavior
```
This code specifies the algorithm to run at each simulation time step.
```
function onSimulationStep(self, event)
```
This code updates the translation of the actor using the velocity of the actor and the step size to move the actor at constant velocity.
```
self.Actor.Translation = self.Actor.Translation + ...
    self.Velocity * self.getSampleTimeImpl.SampleTime;me;
```
This code updates the pose of the RoadRunner actor. The transform2matrix function converts the translation, rotation, and scale vectors into the four-by-four pose matrix format used by RoadRunner.
```
self.Pose = self.transform2matrix(self.Actor.Translation, ...
                self.Actor.Rotation, self.Actor.Scale);
```

Custom Actor with Camera

In this example, you create a sim3d.Actor object and attach a camera to this actor to navigate the actor. This code creates a box-shaped body and custom behavior to follow a straight line at constant velocity . An image from the camera displays in a MATLAB figure at each simulation time step.

MATLAB Code for Custom Camera Behavior

classdef Sim3dEgoVehicle2 < sim3d.scenario.ActorBehavior
    % Sim3dEgoVehicle2 implements an ego vehicle behavior using sim3d

    %   Copyright 2023 The MathWorks, Inc.
    properties
        Camera = []
    end

    methods(Access = protected)
        function onSimulationStart(self, event)
            % event is an instance of Simulink.ActorSimulation

            % Create a vehicle
            [translation, rotation, scale] = self.matrix2transform(self.Pose);
            actorName = self.generateActorName(event);
            self.Actor = sim3d.Actor(ActorName=actorName, ...
                Translation=translation, ...
                Rotation=rotation, ...
                Scale=scale);
            self.Actor.Mobility = 'Movable';
            self.World.add(self.Actor);
            
            % Create a body for the vehicle
            actorModel = event.get('ActorModel');
            box = actorModel.getAttribute("BoundingBox");
            
            body = sim3d.Actor(ActorName=actorName + "_Body", ...
                Translation=[0,0,(box.Max(3) - box.Min(3))/2]);
            body.createShape('box', ...
                [box.Max(2) - box.Min(2), ...
                box.Max(1) - box.Min(1), ...
                box.Max(3) - box.Min(3)]);
            color = actorModel.getAttribute("PaintColor");
            body.Color = [color.r,color.g,color.b]/255.0;
            self.World.add(body,self.Actor);
            % Tag
            tag = sim3d.Actor(Translation= ...
                [box.Max(2),(box.Max(1) + box.Min(1))/2,(box.Max(3) + box.Min(3))/2]);
            tag.createShape('sphere',[0.1 0.1 0.1]);
            self.World.add(tag,self.Actor);

            % Create a camera
            self.Camera = sim3d.sensors.IdealCamera(ActorName= ...
                self.Name + "_Camera1", ...
                ImageSize=[768,1024], HorizontalFieldOfView=60);
            self.Camera.Translation = [box.Max(2) + 5,(box.Max(1) + box.Min(1))/2,(box.Max(3) + box.Min(3))/2];
            self.Camera.Rotation = [0,-pi/12,0];
            self.World.add(self.Camera,self.Actor);
            
            % Set the view point
            view = self.World.createViewpoint();
            view.Name = 'Viewpoint1';
            view.Translation = translation + [0 0 2];
            self.World.setView(view);
           
        end

        function onSimulationStep(self,event)
            % event is an instance of Simulink.ActorSimulation
            
            % Read the camera and ...
            sceneImage = self.Camera.read();
            imshow(sceneImage);

            % ... compute and update the vehicle position
                self.Actor.Translation = self.Actor.Translation + ...
                    self.Velocity * self.getSampleTimeImpl.SampleTime;
            self.Pose = self.transform2matrix(self.Actor.Translation,self.Actor.Rotation,self.Actor.Scale);

        end
    end
end

In the custom behavior:

This code creates a subclass Sim3dEgoVehicle2 of sim3d.scenario.ActorBehavior with an additional property, Camera.

classdef Sim3dEgoVehicle2 < sim3d.scenario.ActorBehavior
% Sim3dEgoVehicle2 implements an ego vehicle behavior using sim3d

    %   Copyright 2023 The MathWorks, Inc.
    properties
        Camera = []
    end

This code defines the algorithm to run at the start of simulation.
```
methods(Access = protected)
    function onSimulationStart(self, actorSimulation)
```

This code creates a new actor in the 3D visualization and sets its position and orientation to match the RoadRunner actor.

% Create a vehicle
[translation,rotation,scale] = self.matrix2transform(self.Pose);
self.Actor = sim3d.Actor(ActorName=self.Name, ...
    Translation=translation, ...
    Rotation=rotation, ...
    Scale=scale);
            
self.Actor.Mobility = 'Movable';
self.World.add(self.Actor);

This code gets the bounding box of the RoadRunner actor and creates a box-shaped body for the 3D actor.

actorModel = event.get('ActorModel');
            box = actorModel.getAttribute("BoundingBox"); 
            body = sim3d.Actor(ActorName=actorName + "_Body", ...
                Translation=[0,0,(box.Max(3) - box.Min(3))/2]);
            body.createShape('box', ...
                [box.Max(2) - box.Min(2), ...
                box.Max(1) - box.Min(1), ...
                box.Max(3) - box.Min(3)]);

This code sets the color of the actor based on the RoadRunner actor color and adds the actor to the 3D world.

color = self.ActorModel.getAttribute("PaintColor");
body.Color = [color.r,color.g,color.b]/255.0;
            
self.World.add(body,self.Actor);

This code creates a custom camera object, configures its position and orientation, and adds it as a child of the actor. This process attaches the camera to the moving actor in the 3D world.

% Create a camera
self.Camera = sim3d.sensors.IdealCamera(...
    "=ActorName=self.Name + "_Camera1", ...
    ImageSize=[768, 1024], ...
    HorizontalFieldOfView=60);
            
self.Camera.Translation = [box.Max(2) + 5, (box.Max(1) + box.Min(1))/2, (box.Max(3) + box.Min(3))/2];
self.Camera.Rotation = [0,-pi/12,0];

self.World.add(self.Camera,self.Actor);

This code creates a custom viewpoint and set the viewpoint to view the 3D environment.

view = self.World.createViewpoint();
view.Name = 'Viewpoint1';
view.Translation = translation + [0 0 2];
self.World.setView(view);

This code specifies the algorithm to run at each simulation time step.
```
function onSimulationStep(self, actorSimulation)
```
This code reads from the camera and displays the view in a MATLAB figure to navigate the custom actor.
```
% Read the camera and ...
sceneImage = self.Camera.read();
imshow(sceneImage);
```

This code updates the position of the actor in the 3D world to follow a straight line at its initial velocity.

% ... compute and update the vehicle position
    self.Actor.Translation = self.Actor.Translation + ...
        self.Velocity * self.getSampleTimeImpl.SampleTime;

This code updates the position of the actor in the RoadRunner simulation using the transform2matrix utility function.
```
self.Pose = self.transform2matrix(self.Actor.Translation, ...
    self.Actor.Rotation,self.Actor.Scale);
```

Associate Actor Behavior in RoadRunner

This section describes how to associate any custom behavior to your actor and how to start co-simulation. For a more detailed explanation, see Connect MATLAB and RoadRunner to Control and Analyze Simulations and Simulate RoadRunner Scenarios with Actors Modeled in MATLAB.

In your RoadRunner scenario, select the Library Browser and then the Behaviors folder.
To create a new behavior, right-click an empty space in the list of behaviors, pause on New, then select Behavior. Enter a name for your new behavior, such as MyNewBehavior. This animation shows how to complete these steps.
On the Attributes pane, set Platform to MATLAB/Simulink. For the File Name, use the location of your sim3d.scenario.ActorBehavior file Sim3dEgoVehicle1.m, or Sim3dEgoVehicle2.m.
- If your actor behavior file is in your working folder, you can enter the name of your file together with its extension .m, for example, Sim3dEgoVehicle1.m.
- You can also use the full path to enter the location of your file, for example, MyLocation\Sim3dEgoVehicle1.m.
For example, add a new Suv actor to your scenario MyExampleScenario. For more information on adding a new vehicle, see Add Vehicles (RoadRunner Scenario).
To associate the sim3d.scenario.ActorBehavior behavior with a RoadRunner actor, select Suv. Then, in the Attributes section, in the Behavior box, add your new behavior to CompactCar by clicking and dragging the behavior icon to the box.
Save your scenario.
Close RoadRunner and reopen it from MATLAB using this command. Replace MyRoadRunnerProject with the path to your RoadRunner project.
```
rrApp = roadrunner('MyProjectLocation');
```

Open the scenario MyExampleScenario.

openScenario(rrApp, 'MyExampleScenario');

Get the simulation object to control simulation from MATLAB.
```
rrSim = createSimulation(rrApp);
```
Start the simulation from the command line.
```
set(rrSim, 'SimulationCommand', 'Start');
```
For more information about simulating your scenario in RoadRunner or controlling a scenario simulation using MATLAB, see Overview of Simulating RoadRunner Scenarios with MATLAB and Simulink.

Visualize Your Scenario Simulation in 3D with an Observer

You can view your existing scenarios in the Simulation 3D Viewer by attaching a sim3d.scenario.Viewer observer to your simulation. This action enables Unreal Engine^® 3D visualization of your simulation independent of how you define actor behaviors.

This code defines an observer template called Sim3dScenarioViewer. Attaching this object then starting simulation opens the Simulation 3D Viewer.

Note

If you define an actor behavior using sim3d.scenario.ActorBehavior, you do not need to attach an observer to visualize your scenario simulation in Simulation 3D Viewer.

MATLAB Code for Observer Template

classdef Sim3dScenarioViewer < sim3d.scenario.Viewer
    % Sim3dScenarioViewer creates a Simulation 3D Viewer for a RoadRunner scenario

    %   Copyright 2023 The MathWorks, Inc.
end

To add this object as an observer:

Follow steps seven through nine in Associate Actor Behavior in RoadRunner to create a simulation object.
Use this code to add Sim3dScenarioViewer to the simulation as an observer.
```
addObserver(rrSim, "Sim3dScenarioViewer", "Sim3dScenarioViewer");
```
Here, the first instance of "Sim3dScenarioViewer" specifies a name for the observer, and the second instance specifies the class name. For more information, see addObserver.
Follow step ten in Associate Actor Behavior in RoadRunner to start the simulation. The Simulation 3D Viewer window opens and shows the simulation.