Deploy AI-Based Functions into Rapid Prototyping for Real-Time Applications
Yana Catalina Vanegas Maldonado, Powertrain Advanced Engineer, Hyundai Motor Europe
David Martinez Núñez, Senior Engineer, Hyundai Motor Europe
New functions based on machine learning models are being developed across the automotive industry for multiple applications with the goal of improving vehicle performance and customer satisfaction. To study the viability and perform the calibration of such functions, they can be deployed into rapid prototyping units to execute in-vehicle validations. However, the deployment of models coming from newer development frameworks into real-time applications can bring some challenges using the classical toolchain approach.
This presentation shows how Hyundai Motor Europe (in cooperation with MathWorks) uses Simulink® code generation and the target library for third parties to deploy different machine learning models into a defined toolchain for rapid control prototyping and ECU calibration. Together, we have established a workflow including different solutions to run models designed and trained within MATLAB® or PyTorch® frameworks. This gives the team great flexibility to extend its research and development capabilities across different environments.
Published: 3 Jun 2024
Today, we want to share our use case. It's our story on how we used AI and how we tested that in our vehicles in using rapid control prototyping. And we hope that we can give a short overview to you on our work and inspire you for using AI.
A little bit of the agenda of today-- first, a small introduction about our company. Then I will speak about the motivation and the benefits of applied AI, what we see. And then I will speak about the function development workflow and some issues we had during the process. Then later on, David will talk about the summary of those challenges as well as their results and conclusions.
Yeah. So about the company, we are Hyundai Motor Europe Technical Center. And this is the European R&D for Hyundai. We are located in Russelsheim and in Nurburgring. And we are working with the three main brands-- or the three brands of the group, which are Hyundai, Kia, and Genesis. And the main activities of the company are related to electrified propulsion development, which is where David and I are working on, also vehicle development, electronic system development, as well as commercial vehicles and design.
So about the motivation and the benefits that we see of using AI in automotive, specifically in power train applications, to ensure that we have the best-- or we give the best experience to the customers, we see, for example, the function personalization potential. For that, we have done some driver characterization, where we recognize what kind of driver is using the car.
Also, to improve the experience, we have speed prediction models. Similar to that, also, forecasting conditions, like torque prediction as a time series. And yeah, related to that, if we can predict what will be the condition of the car in the near future, we can also adapt the power train, for example, to have a higher energy efficiency.
Also, related to AI, we have SOC prediction, where we reduce the complexity. Similar to what Martin was showing us before, instead of having a very high-fidelity model, we can train a model that gives us this prediction.
Our final goal in the team is to integrate these functions fast in the vehicle, to test them, to drive them. So that's why we call this driving AI, to give the driver a feeling, a subjective evaluation of how to drive these functions, how these functions are working.
So to look into the workflow of the development of these functions, first, we start with the modeling of the function. Then of course we need to compile it for simulation and then run it into a virtual environment. This we know very well. And then we have to go into the target hardware, which is usually a rapid prototyping unit.
After we are running in the rapid prototyping unit, we need to connect that unit with the vehicle. So we have to integrate it with the can, usually, and validate the functions in real time, which means driving the car with the functions and test them. Then we can involve the calibration engineers and finally release the function.
Now I want to speak about how was the process when we started using AI. For that, I have two different examples of what we were working on. So the first one is a case where we used only MATLAB. So from the data preparation, data cleaning, to the model design and model training, we used MATLAB. So at the end, we had this MATLAB object.
In the case number two, we had already an existing model, coming from PyTorch. So another colleague of us was working on that. He collected all the data, the engineering features. And then, yeah, model design, model training was done in PyTorch. So we used the add-on of MATLAB to convert the PTH object into to the MAT object, basically. And yeah, the idea was using or having that MAT object, we could easily jump into Simulink for the next step, which was the virtual validation.
So the idea of this slide is to show you that we can work either with MATLAB or with PyTorch and finally arrive to the MATLAB object that allows us to use Simulink as a virtual validation tool. So let's look into that.
Here, again, we have two different approaches for the virtual validation. First, we used here the deep learning block of Simulink, where we have the MATLAB object. And this is simulated with a physics-based vehicle model. So we have a lot of blocks inside there. And this is simulating the car.
The second case, we used measurements from the car. So instead of using the vehicle model, we use recorded data. But here, it's important to mention that we could as well do it the other way around. So basically, as long as the model or this block is running in Simulink, we can use it. We can use both options.
We can even use a third-party tool, which is a compatible with Simulink. So that's the final step. And until here, until this point, everything has run pretty well. No big issues. But when we go into the rapid prototyping, is where we found many of the challenges.
So in our company, we're working with a defined toolchain. And this is most of the times constraining us to use older MATLAB versions. So we cannot use the best or the newest MATLAB version and the fixes and the new apps. We have to work-- find works around using older MATLAB versions.
So in the first case, which I explained, the MATLAB-based neural network, we had a similar-- some issue, which was first that, of course, we had to stick with MATLAB version-- older MATLAB versions. So we had to first code the deep learning block here into an S function. But the S function was giving us an output which was incorrect.
So after some troubleshooting, together with MathWorks, we found out the reason, which was a parameter-- a specific parameter inside the network, which was provoking some divisions by zero and then propagating not-number values. After fixing that, we tried again. But we had another issue, which was the instruction set extension.
By default, Simulink is using this extension. But our target hardware here does not or is not compatible with this. So we had to change this simple parameter to none. And then we could generate the code and then compile this model into our rapid prototyping unit.
For the case number two, the issues were different. Here, the original model had a 1D convolutional layer, which was not supported for code generation for that specific MATLAB version. So we decided, together with MathWorks Support here again, to code it manually using the dlnetwork function. But here again, it was not supported.
However, the dlnetwork was supported for-- in MATLAB 2023. So in our case, in this case, we went to MATLAB 23, generated an S function, and then imported the S function into MATLAB 2021, then generated the full code for that function. And then we could compile it into the target hardware.
Now I know that MATLAB 2024 is supporting this type of layer. So if we could use 2024 now, we would need to do that. But still, our toolchain is constraining us still to older MATLAB versions. So the workaround is still valid. Now I will give the word to David, which will explain you more about the next steps of the process.
OK, thank you very much, Yana. And now I would like to talk about the real-time validation phase as part of this function development workflow. So first of all, we were able to solve the issues that Yana just mentioned, with the support of MathWorks.
And thanks to this, we will move to the next phase, which was to run the Simulink models in the target hardware and start with the evaluations in the vehicle, performing the test drives that we need to do to perform the full validation. So you can see here the results about the speed and torque predictions that the AI algorithms calculated.
And I would like to emphasize that the high correlation between the predictions in real time and the predictions from the simulated environment. And finally, these results were quite aligned with the actual values in the vehicle.
So we started from the PyTorch and MATLAB AI developments. Then we wrapped these functions in Simulink to be able to deploy them into the rapid prototyping hardware that we had and finally bring this hardware into the vehicle and run the test drives.
Now I would like to talk about the challenges that we face. So we had to work in six different areas, starting with the training data generation, then developments coming from different frameworks, like MATLAB and PyTorch. Then we had the area of the function modeling, where we performed the wrapping of these functions for the deployment in the rapid prototyping toolchain.
And also, as part of an initial verification and validation, we created the simulation environments with different-- or using different tools also for simulation. And finally, we had the system running in the vehicle, where we could adjust and calibrate the functions.
So challenges were related to compatibility among different parties. As Yana mentioned, we had the limitations to use older MATLAB versions. Some of the characteristics of our AI models also brought some limitations in a straightforward MATLAB code generation. And this was also where we had some issues with code generation and compilation to have the correct behavior in real time.
Now, in terms of the results, I would like also to talk about the purpose of our research. So what we want to-- what we are targeting is to enhance the vehicle-to-driver collaboration, like, for example, adapting the performance of the power train between more eco drivers or more sporty drivers, and also have an idea of how good the artificial intelligence approach is for the nature of our problems.
So we were demonstrating that AI had a very good capabilities for subjective driver characterization of their own driving styles. And some complex predictions were also solved by artificial intelligence, like the two examples about the speed and the torque predictions.
So results is we were able to achieve driving style recognition, perform predictions about the speed and the torque and also about the high-voltage battery state of charge. And in terms of the conclusions, finally, we were able to successfully deploy AI models from Simulink into our existing toolchain. The AI capabilities using MATLAB for real time were demonstrated.
And yes, we had to incorporate a lot of manual process due to the nature of our software and hardware. But thanks to and very much appreciated the help from MathWorks, we were able to solve the problems for the two specific cases that we saw before. We know that, perhaps for the future, there are other approaches in terms of software and hardware combinations that might have easier integration steps.
And of course, we see a big advantage in using newer MATLAB versions with specific, more specific, target hardware, as I just mentioned. And yeah, this is all from our side. And thank you very much for your attention. If you have any questions, feel free to ask.
[APPLAUSE]