Integrating RoadRunner with MATLAB-Simulink

Testing an autonomous vehicle in a virtual environment is a very widely used low-cost and effective method for developing and testing automated driving algorithms. In this video, we will learn how to integrate MATLAB and Simulink with Roadrunner for developing or training your automated driving algorithms in a detailed environment.

I will be demonstrating a workflow to make use of the photo-realistic see-nothing features of Roadrunner to create a scene within minutes, and use MATLAB and Simulink to design our training algorithms using Unreal Engine.

Integrating MATLAB and Simulink with Roadrunner has four major components. First, we start with building a custom scene in RoadRunner. Next, we export the 3D portion of the scene as a Filmbox file to Unreal Engine and export the road semantics as an OpenDRIVE file to MATLAB and Simulink. Here, the Unreal Engine will act as a code simulation platform to visualize the scene, as well as simulate the vehicles.

We will then import the OpenDRIVE file into MATLAB and Simulink using Automated Driving Toolbox's Driving Scenario Designer app and customize the scene by adding vehicles to it. And finally, we will be perform the code simulation with Unreal and Simulink with the scenarios and the vehicles.

So let's get into it. We start with building a scene in RoadRunner. You can use different assets, stencils, props, and different features within Roadrunner to customize the scene. A key point to remember during this step is to have as many road centers as possible. This ensures that the co fitting between the two roads, highlighted by the red dot, matches the road generated when importing the OpenDRIVE file in Driving Scenario Designer.

Once you're happy with the scene you have created, we move to step two, where we export the generated scene into a Filmbox file, to input the file into an Unreal Engine, and an OpenDRIVE file, to input the file into MATLAB and Simulink. On the MATLAB and Simulink site, install and configure the automated driving tool box interface for Unreal Engine 4.

To copy the project to a local folder and to plug into Unreal Editor, use the following one-time steps code to configure and install the files. Make sure to specify where the Unreal Engine installation folder is and the destination folder to install example files that are shipped with the support package. Upon successful installation of the plugin, you will receive the following prompt in MATLAB.

To import the scene in Unreal Editor, the Roadrunner plugin must be installed. Start by downloading the latest Roadrunner Unreal Engine plugin from exporting to Unreal page, and extract the plugin into a local directory. Next, creating an Unreal Engine C++ 4 project. Currently, Unreal Engine 4.23.1 is supported with R2020a and R2020b.

Now, close the project and copy the Roadrunner Importer folder and Roadrunner Materials folder into the Plugins folder under the Project Directory. If a Plugins folder does not exist in a Project Directory, create one. Now, right click on the project and select Gendered Visual Studio project files to generate binary files for the project.

After this step, open MATLAB and run the Sim 3D editor to open the project with both MATLAB and Roadrunner plugins. You will need to provide the path and the Unreal project title to the same 3D editor. Click yes to build the plugins. The script then opens the Unreal editor. In Unreal, navigate to the plugins folder in the Edit tab to make sure it's enabled. If it's not enabled by default, select the checkbox and hit the Restart Now option to enable the plugins.

Once the two plugins are installed, import the OpenDRIVE file we had earlier created into Driving Scenario Designer. Add a vehicle and specify waypoints to customize your open-loop scenario. Feel free to specify additional vehicles or waypoints. After customizing the scenario, explored the scenario into Simulink model. You will need to save the scenario before exporting.

Once the scenario block has been exported, to control the Ego Vehicle, add a Cuboid to 3d Simulation block and Simulation 3d vehicle with Ground following block. And to control actors, additionally specify a Vehicle to World coordinate block. You will also have to enable the output Ego Vehicle Pose option within Scenario Reader to get the Ego Vehicle coordinates.

Finally, use a Sim configuration block to specify the source of the scene. Set the scene source to Unreal Editor and specify the project part. The scene view shows us a chase cam perspective of the vehicle selected. Also, make sure to set the Solver to Variable Step Solver. The plugins and the platform are now set.

To import the scene in Unreal, drag and drop the Unreal Filmbox scene created in Roadrunner in Unreal Editor. In the following prompt, set Normal Input Method to Import Normals, and disable Remove Degenerates. Hit Import. This will begin the importing the scene. And depending on number of assets added, the importing step may take some time.

The Roadrunner scene is now imported into Unreal Editor. The props, assets, used in the scene can be viewed in the Unreal Editor. Before we begin the code simulation, repairing the blueprint is required. You can do so by navigating to Blueprints and setting the Repairment Blueprint to Sim 3D level script actor.

Now hit play on the Simulink side. The dynastic view will prompt the user to select Play in Unreal Editor to view the scene. Hit play in Unreal Editor. And this will begin the code simulation. The car then follows a set of waypoints we had defined earlier in the Driving Scenario Designer app within this open-loop simulation. You can specify multiple actors, vehicles, and the waypoints to further customize your scene.

Following similar steps, you can create a closed-loop automated driving algorithm that can use sensory information from detections made in Unreal Engine. In this demonstration, the vehicle travels within a marked lane and maintains set velocity, or safe distance to a preceding vehicle, the same lane using region processing, sensor fusion, decision logic, and control components. This example uses a combination of vehicle dynamics blocks and model predictive control and Automated Driving Toolbox.

To learn more about Roadrunner, Simulink, and Unreal, follow the link in the description. Thank you for watching.