Automated Driving in the Urban Environment with RoadRunner Scenario
Advait Valluri, Senior Application Engineer, MathWorks
This talk presents concepts that can be used to expedite the development of advanced driver assistance and automated driving systems by leveraging the power of simulation. It also provides solutions to augment your simulation environment with vehicle-to-everything (V2X) communication, which will become relevant in the future. Learn how to:
- Create a complex urban scene consisting of intersections with traffic lights in RoadRunner.
- Generate a V2X MAP from RoadRunner HD Map and a V2X SPAT (signal phase and timing) from the RoadRunner traffic signalization.
- Implement a mission planner to search the shortest path for a given start and destination position using an A-star planner.
- Design a behavioral planner to follow traffic lights in the urban intersections using V2X MAP and SPAT in RoadRunner Scenario.
Published: 4 Jun 2024
Let's get right into it. Welcome to this talk on automated driving. So what we will be looking at today, Reza has set a really good state, so I can skip a lot of content today. We will look at how we can set up a virtual environment. That is, scenes and scenarios you saw extensively how has done that with our tools. So maybe this is something I can cut short.
Then we will look at, Asta gave me my spoilers. He said V2X already in the introduction. So you know that we will be looking at some V2X functionalities today and how you can use MathWorks tools to simulate V2X. And finally, see how a closed loop automated driving simulation setup can look like.
The primary goal today, however, is to understand how simulation frameworks can help to develop automated driving systems and also instill confidence or trust in higher levels of automation. Speaking about levels of automation, this morning, we saw Professor Bertolazzi speak about the SAE levels of automation. So I'm not going to go into detail about this.
But what I would like to say is level two technology is already available in the market today. You can buy cars with level two functions. And the current state of the art, the automated driving industry is moving strongly or pushing strongly to bring in level three, level four, and level five automation functions into the market.
What we will be looking at today is the urban traffic light follower function with path planning. So this is something I would categorize as a level four function. And we will incrementally build up a simulation platform which will help us do closed loop simulation for such functions.
First, let us see what automated driving systems have to tackle. In urban environments, especially, there is complex road geometry. There could be maneuvering at intersections with traffic lights. There could be ad hoc construction sites. And an automated driving function must be able to tackle this. So we may be able to train the function on a lot of data. But like Reza already mentioned, it could be that unseen scenarios just pop up.
So sensors, many vehicles today come with onboard sensors. So the capability of the onboard sensors is affected massively in urban environments, primarily because of tall buildings, occlusion occurring due to other parked vehicles, or any other urban components. So we have limited sensor field of view or even sensor occlusion or communication and connectivity issues, primarily GPS signal loss or even poor reception.
So one of the ways we could tackle this is to use V2X, as Asta already mentioned. And with V2X, or vehicle to everything communication, there are certain things we need to keep in mind, primarily the infrastructure enhancement which is needed. So you might need new units which will send or transmit and receive signals. We need to adhere to any regulatory compliance standards or even develop the messaging or the transmission of the information according to a particular standard. As you can see here, there are some examples of SAE J2735.
But before we move ahead, maybe I can convince Asta to give away another speaker or something like that. If you can tell me what this is. This is indeed Midtown Madness. It's a game from the '90s. But what we did today with automated driving is actually "Midtown Calmness." So I'm going to show you a function which does exactly that. So as you can see here, the vehicle is now approaching an intersection, and it notices that the current signal, which is governing its motion as red.
So the vehicle now brakes and waits at the intersection until the signal turns green so that it can perform the maneuver through the intersection. This is what we will now incrementally build. So as you can see, the vehicle now starts to turn left at the green light, completes its maneuver, goes on to the other side of the intersection and performs or traverses this intersection in a safe fashion.
Now imagine if a real vehicle had to do this all by itself. Can you imagine the level of complexity that is needed? And that is exactly what we will do today. We will begin by creating an urban scene using RoadRunner. Then we will design a path planner which will help us to reach a specified destination on my map or in the urban scene. And finally, design a behavioral planner which adheres to certain traffic rules.
For this, certain components are needed, for example, a V2X map or vehicle to everything map; then the V2X SPaT message, which is the signal phase and timing message; and finally, a traffic light follower module, which will then safely use traffic light information to design a path and even adhere to this path.
Let's begin by looking at RoadRunner. Like I mentioned, we already saw RoadRunner. So I'm going to skip this part. But what is important to is the tool already ships with some complicated scenes, like the one you see here, which is a US city block containing about 15 intersections with traffic lights. And using RoadRunner's editing tools, every aspect of this scene can be modified. So you have full access to change the road semantics, add additional assets, or even change the position of the assets, etc.
If I zoom into one of these intersections, you will see that there is an intersection with traffic lights equipped. Next, we will design a path planner to determine the shortest path to a given destination on the map. The automated Driving Toolbox comes with certain preshipped examples. In this case, there's one which demonstrates a lane level path planner using RoadRunner Scenario.
So RoadRunner Scenario is something which builds on top of RoadRunner, where you can define actors that trajectories and logic, that is, the scenario part of my virtual world. This example illustrates how a shortest path can be found between a given start and goal position. So the Path Planner, the workflow looks as follows. Once you have your RoadRunner scene, you can convert it into a RoadRunner map, which is our internal map model from MathWorks, which is essentially a collection of links and notes.
So I'm representing my map in the scene as a collection of links and notes and then creating a nav graph out of it. The navigation graph, navigation toolbox, for example, helps you do this. We have preshipped algorithms which help you do that. From the nav graph object, I can instantiate a Path Planner. In this case, we use the A-star Path Planner. And using the Path Planner, I can now pan a plan a path between the start and end position as you can see here.
So this is the logic we'll also be using for our scene. So the scene we saw earlier, I've now converted it into a HD map, and then defined the goal position here. I'm beginning here. Let's see how that looks in the scene we just saw. So our path now traverses two intersections-- the first one here, the next one here. And we will then shortly build our Behavioral Planner.
For that, first we need to what should a Behavioral Planner do. So what should its capabilities be? So first thing is it needs to detect intersections. If it's approaching an intersection, as we saw in the video earlier, it needs to recognize that this is an intersection. Then it needs to understand what is the current state of the traffic light. What is the timing left on the traffic light?
Once this information is available, it can decide on an appropriate maneuver. Do I go straight? Do I turn left? Do I turn right? And finally take a decision, do I stop or do I continue? Which is typically the two choices you have at an intersection. For this, we will now look in detail about how we can generate a V2X map, how we can generate a V2X SPaT message, and finally, the traffic light follow module.
For the map, we already have a shipping example in the automated driving toolbox, which shows if you have a RoadRunner scene, how you can create a V2X map message out of it, adhering to certain standards. So this is exactly what we do. This is the scene we saw earlier. I'm converting it into my HD map. And the solution or the example provides us a solution to generate the V2X map message.
RoadRunner also comes with a very powerful Matlab API, which helps you to script a lot, so you don't necessarily have to open RoadRunner. So the API helps you to interact with RoadRunner programmatically and also even generate the HD map, as you can see. RoadRunnerHD map is a simple data model consisting of lanes, lane boundaries, and junctions. And I can in fact even plot the map as we saw in some of the screenshots.
So once I have the map, I can create the V2X map message. So this includes nodes, for example, which is the intersection links, which are the roads coming into the intersections, and finally some lanes, which is then defining how many lanes are there on every road. Once I have this information, I can pack it into a V2X map message, as defined by certain standards. So the one we used for this demonstration is from the China SAE, but it's very, very similar to the SAE, that is, the Society of Automotive Engineers, the J2735 standard.
So the V2X message contains primarily nodes, links and lanes in a hierarchical structure. And the lane entry contains additional information, which is very essential for the behavioral planner, like lane ID, lane width, speed limits, any geometric points, and finally, some connectivity information. Especially regarding connectivity, I want to give you a specific example. Let's say we have a vehicle at an intersection and I pinpoint it on the map.
I can then, from the V2X map, query, identify what is the node this vehicle is on, what is the link, and finally, which lane it is on. With this information, I already have a lot. But additionally, the V2X map query can also include information on connectivity. So this is exactly what I want to speak about. For example, the vehicle on this lane is now on a lane which is governed by two traffic signals. So as you can see here, signal one and two, signal two defines that a maneuver is allowed to go on to node number 22 using a left turn, or signal one defines I can go straight onto node number 23.
In addition to the V2X map, we would need the V2 x SPaT or the signal phase and timing message. So RoadRunner comes with the junction tool, which helps you to signalize junctions. A phase defines the current active signals and their timing. So as you can see here, each phase can be defined with certain maneuvers, and typically green, yellow, and red timing phases.
So RoadRunner comes with the signal tool, which is this, well, the signal symbol. And once you click on any junction which is defined in RoadRunner, you have the option to define signal phases. As you can see here, this is a four-way protected left. And for every phase, there is the option to adjust the duration of the individual lights, that is, the green, yellow, and red lights.
In this case, phase one is having right turns. And the next phase contains left turns. And then it repeats for the other leg of the road. With this information, what I can do is I can convert the lights into state flow. So please remember that RoadRunner signals are there primarily for export purposes. So no vehicle in RoadRunner would adhere to a traffic light because it's not a simulator per se, which is why we implemented a state flow chart kind of replicating or mimicking the traffic signals we will see at this intersection.
But once I have my signalization information, I can also pack it just like the map into a V2X SPaT message, again, defined by the standard. And the SPaT message contains, similar to the map, a hierarchical structure for intersections, the phases, and finally the phase state, which includes the current light of the signal and the timing information. With the map and the SPaT message, I can now finally design my traffic light follower.
What you are seeing here is the Simulink implementation of the traffic light follower. We're not going to dive into detail. For that, we have a group of experts at MathWorks. We also have a booth outside for RoadRunner. So please feel free to visit us. But what I want to speak about shortly is how Simulink and RoadRunner work in tandem or hand-in-hand to enable this closed loop simulation framework, which we will see at the end.
So these blocks are showing how you can interact with RoadRunner. So this Simulink model is passed as a behavioral model to my vehicle in RoadRunner. So what happens is in every timestamp, RoadRunner is sending the current position of the vehicle to Simulink. Simulink does some calculation based on the map message based on the SPaT message, and then finally calculates a new position for the vehicle, which is then sent back to RoadRunner Scenario. And this is how, with the lockstep simulation I have a full closed loop framework.
The Traffic Light Follower ensures and controls the vehicle speed so that the signals are adhered to. And finally, the Path Planner adjusts the vehicle's lateral position such that the planned path is followed through. In order to implement the Simulink model, we need additional information. So I mentioned earlier the V2X map query. So once I can pinpoint my vehicle on the map, the V2X map query gives me information about the current node, the link, and lane ID.
Apart from this, some important or critical information is the current distance and the time to the intersection, to the stop line. Because this is quite valuable, or in fact, necessary to perform the control action that I plan. Additionally, we also need some connectivity information. Like I mentioned, this lane, for example, if I want to adhere to my planned path and I'm on this lane, I have to perform a left maneuver. And for example, this signal ID number two is providing me exactly that information that I'm the one who governs your movement.
Once I have this, what I can do is I need to now decide if I have to stop or continue at the intersection. So what I can do is I can compare the remaining time on a particular signal, in this case, the signal ID number two, to the time to stop of my ego vehicle. And in this case, 5.9 seconds higher than my current time to stop, for example, then I can say that, OK, now I have to stop at the intersection. I can't continue because the light is still going to remain red.
This way, reaching every signal I can incrementally plan, adjust my current longitudinal motion as well as lateral motion based on the path and the signal information, and slowly develop the behavioral planner. Let me quickly show you a simulation result for the traffic light follower with co-simulation via RoadRunner Scenario. So as you can see here, the current time is larger than my time to stop. So the vehicle decides to brake at the intersection. It's going to wait at the intersection until the signal turns back to green.
And once that happens, the vehicle will continue moving. And if you observe closely, we did have a lane change here. So that is exactly what's going to happen. And you can also see that the vehicle's speed is changing. So this was where it was standing at the intersection. And now it's approaching the intersection with the constant speed as defined by the path planner. It reaches the next intersection. There, it again observes that, OK, my current signal time is much higher than my time to stop, so it makes a decision to stop again.
And then waits until the time runs out. And once the signal turns to green, it continues to follow the path, as you can see with the velocity map here. That way, it reaches the end of the junction and then completes the maneuver. So we just saw an example with two intersections. But this is how you can incrementally create a closed loop simulation framework.
That brings me actually to the end of the presentation. So let me quickly summarize what we did. So we created a complex scene in RoadRunner, and also the scenario with RoadRunner Scenario. We use the RoadRunner and RoadRunner asset library. We saw how junctions with intersections govern these traffic lights. Then we designed a Path Planner using an A-star planner and the navigation toolbox, and finally designed the Behavioral Planner, where we incorporated certain capabilities of the V2X standard using a map message, a SPaT message, and the traffic light follower module.
We implemented this using Simulink, Stateflow, and RoadRunner Scenario. All these tools come together with the Automated Driving Toolbox, which provides the overall simulation framework. That being said, I think I'll hand it over back to Asta. And thank you for your attention.
[APPLAUSE]