AI Certification for Avionics - MATLAB
Video Player is loading.
Current Time 0:00
Duration 54:10
Loaded: 0.31%
Stream Type LIVE
Remaining Time 54:10
 
1x
  • Chapters
  • descriptions off, selected
  • en (Main), selected
    Video length is 54:10

    AI Certification for Avionics

    Overview

    AI Certification has been a major challenge in the aerospace industry due to safety critical assets. To accommodate for the advancement with AI solutions, the conventional V-cycle has been adapted into a W-cycle for system validation & certification.

    In this focused webinar, we will uncover:

    1. The workflow details for certifying an AI algorithm along with a DAL-D compliant Certifiable Airborne Deep Learning System Example.
    2. How MATLAB & Simulink based automated applications adapt to the W-cycle to drive certification for safety critical applications in aerospace.

    About the Presenter

    Peeyush Pankaj| Sr. Application Engineer | MathWorks

    Peeyush is a senior application engineer at MathWorks, where he promotes MATLAB products for data science. He works closely with customers in the areas of predictive maintenance, digital twins, enterprise integration, and big data. Peeyush has over 11 years of industry experience with a strong background in aviation. Prior to joining MathWorks, he worked on aircraft engine design, testing, and certification. Peeyush holds a master’s degree in advanced mechanical engineering from the University of Sussex, UK. He has filed 25 patents on advanced jet engine technologies and prognostic health monitoring of aircraft engines.

    Recorded: 1 Feb 2024

    Good evening to all, and welcome to the MathWorks Modern Aircraft Design and Engineering Series. I'm Prashant, part of the marketing team here at MathWorks, and I'll be your host for today. The series aims to provide you insights into some of the most important elements of aircraft design and engineering and how modern applications have evolved to support advanced aerodynamics to propulsion systems to navigation, both in the commercial and defense domain. We really hope the series helps you in your respective project areas and how MathWorks can support.

    Also, before I proceed, I would like to give you a glimpse on the lineup of industry speakers we have for the series, and each are eminent in their own domain sharing their valuable insights and learnings. We really urge you to attend all our five sessions and get the opportunity to listen to them.

    We start our first session on the topic of AI Certification for Avionics, and it's MathWorks' honor to have Dr. Ramakrishnan Raman from Eaton present his talk on AI ML considerations for safety critical complex systems. A brief about him Dr. Ramakrishnan is currently senior chief engineer at Eaton. Prior to joining Eaton, he was fellow at Honeywell Aerospace. He is a certified six sigma black belt and Inkosi-certified expert systems engineering professional fellow at Indian Society of Systems for Science and Engineering and a IEEE distinguished lecturer.

    He has to his credit several publications in preferred international conferences and journals on complex systems architecture design and artificial intelligence machine learning and has also received best paper awards. Thank you, Mr. Ramakrishnan for joining us.

    From MathWorks, we have Mr. Peeyush Pankaj. Peeyush is a principal application engineer at MathWorks where he promotes Matlab products for data science. He works closely with customers in the areas of predictive maintenance, digital twins, enterprise integration, and big data. Peeyush has over 11 years of industry experience with a strong background in aviation. I would now like to hand it over to Dr. Ramakrishna Ramen for his presentation. Thank you.

    Hello, I am Ramakrishnan Raman, senior chief engineer at Eaton. This presentation will discuss AI-ML considerations for safety critical complex systems. First, I would like to thank MathWorks for this opportunity to speak on this topic in the Modern Aircraft Design and Engineering webinar series.

    To set the context for this topic, why does AI matter? AI and other advanced technologies are becoming increasingly popular among scientists and engineers to inculcate differentiating intelligence in modern systems. Modern systems are envisioned to emulate and simulate beyond human intelligence to achieve their goals and perform better than their traditional predecessors.

    From Siri and Alexa to self-driving cars and robots in warehouses, AI already surrounds our daily life. Through AI, we expect machines to be able to carry out tasks in a smart ways. Machine Learning or ML is a subfield of AI where machines learn from data without explicit programming instructions. ML is currently one of the most predominant approach for AI wherein we give machines access to data and let them learn by themselves.

    However, when incorporating AI in safety critical complex systems, there is a critical demand that users of the system can be certain of the behavior and performance of that system. There is increasing awareness and acceptance that prevailing approaches and tools need refinement for modeling, design, and development of AI-based complex systems.

    Moving on to set a brief introduction on complex systems. In a general sense, the adjective "complex" describes a system or component that, by design or function or both, is difficult to understand and verify. A complex system is characterized by emergence of global properties which are very difficult, if not impossible, to anticipate just from complete knowledge of the component behaviors. In this presentation, we will look at complexity from the point of view of predicting the future behavior of the system. So there is a system that is being observed. The capability of the observer, and the behavior that the observer is trying to predict.

    A key characteristics of complex systems is emergent behavior. In the case of complex systems, emergence refers to the ability of a system to produce a highly structured collective behavior over time from the interactions of the individual subsystems. Common examples include a flock of birds flying in a V formation, ants forming societies of different classes of individual ants, and so on.

    These patterns are not induced by a central authority. In the case of complex systems, we do not have adequate knowledge on the combination of events that would lead to a specific emergent behavior. This emergent behavior is very difficult to envision and predict and cannot be attributed to any of the constituent systems individually. Again, emergent behavior can be categorized into positive and negative behavior based on the impact on the measures of effectiveness of the system.

    Now, how is complexity handled? The figure on the left, illustrates the system hierarchy for a space transportation system. For any specific system of interest, the focus is on how the next level subsystems integrate and interact with each other to meet the desired functionality and behavior of the system and how the system interacts with its peer systems to meet the desired functionality and behavior of the containing system.

    Each system typically has about, say, five to seven subsystems to facilitate easy comprehension and understanding of the functionality and behavior and the interactions through. Complex systems typically involve multiple disciplines. In modern systems, there is an increasing footprint of software, electronics, and networking wherein systems are increasingly getting interconnected with each other.

    Multiple nets are typically used to uncover issues. The design development of complex systems that include reviews, coverage, analysis, verification, tests, validation of reuse, and so on. But many new failures are appearing beyond just those driven by component failures.

    There are two major failure-prone areas the interdisciplinary cracks, and the interactions that have not been envisioned. So often we encounter scenarios where no components would have failed, but then the failure is due to the complexity of the interactions and the emergent behavior, as driven by the system design.

    To illustrate, let's consider the case of the autonomous vehicle where it hit a pedestrian. The driving system detected an object about six seconds prior to the impact but was not sure what the object was and kept switching in its assessment. Then the object was detected, the brakes were selected, but then new actions were subject to interlock to prevent precipitous actions through, and the pedestrian was hit. So this is a case of complex interaction systems leading to really undesirable behavior.

    So while contemplating incorporate ML models in our systems, let's see some of the potential challenges that we might encounter towards ensuring that our system exhibits the desired functionality, the desired behavior, and the performance. Let's take an example of a problem pertaining to the force applied on a football.

    As we are aware from physics, the force is a product of the mass and acceleration. So if you're using a conventional software program, you would compute the force as a product of mass and acceleration. For those of us who are familiar with the conventional software program, we feed in the input data to the program, and the program computes the force and returns the value of force as output.

    In this case, a knowledge of the physics domain that the force is a product of mass and acceleration was used. And when we start looking at this program, we could actually define test cases, come up with a verification strategy pretty easily, and then decide on when the function is fully tested through.

    Now, let's say that we are not aware of the governing physics law that force is a product of mass and acceleration. We can then try to gather data of force, mass, and acceleration, and then feed the data to a ML model that will then learn the relationship of mass and acceleration with force. Once trained, the ML model can be used to predict the force given a mass and acceleration.

    So this slide illustrates the training of the ML model on the force, mass, acceleration data. So Matlab Machine Learning Toolbox was used for this. Typically, the data set is split into training, validation, and test sets, and the training is done. So in this case, the trained model had an r value of 0.99, indicating a good training.

    Now, let's go beneath the surface to look at the details. In this case, it was a neural network that was trained. As you can see, compared to the earlier conventional program that clearly depicted the computation of force as a product of mass and acceleration, what we see here is just a set of numbers pertaining to matrices that are kind of subjected to computations and transformations. Is this understandable? How can we arrive at the criteria for testing this and indicating that the testing is complete?

    If you take a step back and look at in the conventional program shown a few slides back, we had the input data fed to the program, and the output was produced. But in this case, it more seems to be of the data, and the output being provided to the machine. And the machine learns a program that correlates the input data with the output.

    Now, how are the ML models being trained and deployed? So this slide illustrates a schematic workflow of inculcating intelligence in a form of machine learning models in systems. Here is depicted as a pipeline, which starts with the raw data, for example, a collection of images of various roads with associated labels.

    Again, the data may pertain to one or more of types such as text, or time series, or image data. This raw data is then processed to extract numeric features and obtain feature vectors. Learning algorithms are then device that can learn from the data. For instance, if it's a neural network, the algorithms automatically tune and learn the weights and biases for the neural network so that the output produced by the network closely matches the desired output by the training data.

    The trained network now has a desired intelligence, which is then deployed in the system making any system an intelligent system. In the current maturity, there is a human in the loop for most of these steps through. It seems straightforward, but there is more complexity to it.

    And from the systems perspective, we need to analyze what is the intelligence that is required for the system? What are the desired behaviors? What are the desired performance levels? What are the acceptable tolerances for errors, false positives, false negatives, and so on? And there are a whole lot of operational scenarios that need to be considered.

    ML models need data to be trained. So we need to analyze what is the source of the data on which the intelligence can be built, what are the data types, what are the restrictions on usage of data, what should be the training validation data sets, what could be the potential features, does the data span the entire operational scenarios of the system as envisioned to encounter? There are so many additional considerations that significantly matter.

    It is hence evident that the intelligence of the ML model is as good as the data that was used to train it. So with this implication in mind, it becomes essential that a lot more focus is kept on the operational environment of the system and the various scenarios the system would encounter. A challenge is that the operational environment may not be benign, and the implications of undesirable behavior may be serious. For instance, the ML model's behavior may need to be checked in the presence of adversarial input data.

    So when we contemplate ML models for safety critical complex systems, many additional considerations emerge. The entire system design and development process may need to be tailored to incorporate the nuances of developing ML models, for instance, the data requirements. Also, enhanced guidelines may be required to decide when verification validation can be said to be complete and when the system can be considered to be ready for deployment in the operational environment.

    Again, from the data perspective, it may need to be ensured that the data set covers all the identified use cases and should not be biased. And more importantly, a safe and effective human interaction should be facilitated, including explainable intelligence.

    This slide attempts to enumerate some of the original considerations categorized under new failure modes, unpredictability, and trust. For instance, unsafe exploration in the context of reinforcement learning pertains to ensuring that the system does not exploit areas that have bad repercussions.

    Then there are aspects pertaining to unpredictability of the system behavior. For instance, say once a system is deployed, what if the system encounters data that was unseen as part of its training data? What should be the behavior? How should the system respond? Again, can the Intel system explain, in a human comprehensible manner, why it is taking a specific position and thus provide the required trust?

    Earlier in this presentation, we discussed on typical nets such as reviews, verifications, coverage analysis, and so on that are used in the system design and development lifecycle. But for safety critical complex systems, additional nets may be required towards gaining confidence on the intelligent system's behavior and performance.

    Some of these are the DOE fractional factorials focusing on scenarios driven by multiple factor levels, multidimensional state analysis through principal components, ML models to observe the system state and monitor, and formal verification to assert negative behavior. You may not have time to go through all of them in detail, but in this session, we look at how ML models can be used to observe the state of a complex system and monitor the behavior.

    So we'll go through the case of a ML classifier pertaining to the system of a pitch control of an aircraft. So this work was presented in INCOSE International Symposium. We will begin with an outline of the proposed approach for evaluating emergent behavior of a complex system that are governed by machine learning models.

    The proposed approach involves building a ML classifier that observes the various MOEs at the system level and the MOPs at the subsystem level and learns their emergent behavior. As illustrated in this figure, the complex system has constituent subsystems A, B, and C. But the ML classifier observes the system MOEs and subsystem MOPs and learns on the positive and negative emergent behavior.

    So we need some data on the MOEs and MOPs, and then we need to label the data to indicate whether the behavior exhibited was good or not. The ML classifier then learns this and in the future, if it observes a similar pattern of behavior, it can predict the probability of the behavior being good or bad.

    So this slide illustrates an overview of the proposed approach. So as we know, there is a complex system. And we need to analyze the known and exhibited behaviors through. So we classify the behaviors as positive or negative emergence, and we get the data on the MOEs and MOPs. And we use that to train the ML classifier and establish the behavior model.

    Optionally, this ML classifier can be used in a formal verification engine, and based on the behavior model, it can check for and formally assert negative behaviors. So with this behavior model and the trained ML classifier, we can actually plug it into a complex system when it is in operation, and then observe the behavior of a complex system. And it can be used to predict the occurrence of a negative emergent behavior.

    So obviously, there may be an experience of new emergent behaviors. And so we would then go through this process of again, classifying the behavior as positive negative, and then returning the ML classifier through.

    This slide illustrates the case of a aircraft pitch controller. Generically, a control system aims at getting the desired outputs as per the inputs. The specific complex control system under consideration is responsible for controlling the pitch angle. So MOEs for this system pertaining to the comfort level of the flight.

    In a worst-case scenario, the pilot of the aircraft will be unable to maintain the set data value, resulting in an unstable divergent behavior. So instability, at a lower scale, will feel like a roller coaster ride, not divergent, but an oscillatory unsettled behavior. In case of marginal issues, the flight will be felt like a roller coaster with lower amplitude finally settling down to a stable flight path. The equations of the pitch angle of the aircraft are, again, shown in the slide. So here, we have incorporated variations of mass, speed, and density to try to arrive with the variations in the plant model.

    So in this slide, the figure on the left, depicts the Matlab Simulink model of the pitch control system. An orthogonal array of experiments was set up to analyze the behavior of the control system for different values of mass, speed, and density. The figure on the right, shows the response of the closed-loop system for the PID controller. As you see, the variation in the response to a data set point of five degrees is shown in the figure.

    The overshoots vary due to the change in the plan parameters. But the closed-loop system I've seen is stable, and it shows adequate performance for the 25% change in the plan parameters. So this stable behavior is classified as positive emergent behavior.

    So the scenario of the control system with the ML model is illustrated in this slide. So instead of the conventional PID controller, the ML model-based controller commands the plan to achieve the desired output. The training data set for the ML model to learn the controller, is built from the simulation runs of the PID controller.

    The inputs to the ML model are the desired set points, the state of the plant as represented by the angle of attack, the pitch rate, and the pitch angle, and the error. The data set is split into training set, validation, and test sets. And typically, the training set is used to fit the model while the validation set is used to estimate the prediction error for the model selection. The test set is then used for assessment of the generalization error of the final chosen model.

    So this slide illustrates the ML model training performance. The learning is stopped at an epoch of 1,000. The MSE, in this case, is about 0.988 with R value of about 0.99 after the completion of the epochs. Point to note, is that learning could be further tuned by tuning various hyperparameters, including the number of neurons, increasing from the 10 that is used in this case or incorporating additional hidden layers and having the training continue for further epochs.

    However, what is the purpose of dealing with a system that can exhibit negative emergent behavior at times? Further improvements in the learning of the model is stopped.

    The system behavior with the ML-based controller is illustrated in the slide. As is evident in the figure, the behavior is not as robust. The response time and the offset to the target is not as robust as the traditional PID controller. This is primarily because we stopped the training of the ML model so as to get the case of a system that has a potential to exhibit negative behavior for the purpose of illustrating our proposed approach through.

    So now we come to the ML classifier. So the ML classifier is built by tapping in the same inputs that were used for the ML controller and fitting a set of six values corresponding to the current values corresponding to the current time T and the previous five instances of T minus 1. T minus 2, all the way to T minus 5 as illustrated in the slide. The learning algorithm that was used in this case is based on the scaled-conjugate algorithm.

    The training performance of the ML classifier is illustrated in this slide. The data set comprised about five million records. It was split between training, about 75%, testing about 15%, and validation of 15% of data sets. The performance of the learning is assessed in terms of cross entropy function, wherein minimizing the cross entropy leads to better classifier. The confusion matrix that you can see in this figure illustrates the accurate and inaccurate classifications through.

    The performance of the ML classifier when plugged on to the ML controller is illustrated in this slide. The figure indicates snapshots of the tool scope monitor wherein values closer to one indicates a positive emergent behavior being exhibited, while values closer to zero indicates a negative emergent behavior being exhibited.

    The ML classifier is specifically applicable for ML-based complex systems since it can be based on tapping the same inputs as that is used for the ML model that implements the functionality. The functioning of the classifier is independent of the control algorithm or the process. So this ML classifier can hence be used as an additional net to understand the behavior and monitor the behavior of the complex system.

    So to conclude this presentation, so there is an increasing demand to leverage AI to make systems more intelligent, but additional considerations are needed when leveraging AI for safety critical complex systems, and additional nets may be needed to gain increased confidence and trust on the behavior and performance of complex systems.

    To know more on AI from systems learning perspective, please do refer to the article in the system measuring body of knowledge, SEBok that was authored by the INCOSE AI systems working group, which I'm co-chairing. I would like to once again thank MathWorks and the organizers for having me as part of the Modern Aircraft Design and Engineering webinar series. Please do not hesitate to reach out to me for any further queries or comments. Thank you.

    Thank you, Dr. Ramakrishnan, for your insightful presentation. Attendees, if you have any questions, Mr. Ramakrishnan is available towards the end of the webinar, and then we can take the questions. I would now like to hand over the presentation to Mr. Peeyush Pankaj, who will talk on the topic of AI certification for avionics. Peeyush, you can share your slides.

    Can you see my slides, Prashant?

    Not yet, not yet. Yes, we can see it now.

    All right, I'm just maximizing the screen before starting. I hope you can see now.

    Yes, we can see.

    All right, so very good evening, everyone. My name is Peeyush, and first of all, I would like to thank Dr. Ramke for very insightful presentation on machine learning modeling for the complex systems. So in the second phase of this particular webinar, we will try to break it into three phases.

    Firstly, we will try to understand how the certification workflows should look like for safety critical in applications for airborne systems. Then we will deep dive into a particular use case, and with that, we will try to map it with respect to the certification workflow. And towards the end, in the third phase, we will try to understand if we can tackle more aggressive safety critical requirements that is DAL C level of requirements under DO178 C system requirements.

    So to start out with the introduction, as we are seeing the number of use cases to be growing across the industry for the AI systems, that there is more need for explaining the behavior of these AI models. And that is where in safety critical one applications, we have to add more requirements which are pertaining to data management, model explainability, and neuron coverage, data completeness, et cetera.

    And a lot of these requirements is something which is not part of our classic software certification because these are new systems, data-driven systems, and that is where there is a need to incorporate these kind of requirements.

    So I would like to also bring up a success story that MathWorks has helped Gotion, an automotive engineering company, in order to build a state-of-health estimation application for batteries. And while building this application goes into the approach of going through the certification workflow, completely mapping it with respect to the re-cycle that has traditionally being seen in the automotive industry. And that is where our Toolchain was basically used to build an end-to-end system requirement definitions to verification and validation workflow of this ML models.

    And they had also used the opportunity to speak at our MathWorks Automotive Conference. Once you have the recording and the resource material of this particular webinar series, you can go through the hyperlinks to go through their talk in much more detail.

    So with that, first, let's understand, in the transportation industry in general or in aviation in more specific, so the picture that you see on the screen is basically back in 1980s, there used to be four pilots or four engineers within the cockpit of the aircraft, which has significantly been reduced to just two co-pilots these days because of the advancement in software and electronics.

    And now with the data-driven solutions coming into the picture, the industry is targeting a single pilot operation further down the line where we can have more autonomy defined within the aircraft. And that is where if any of the systems are AI ML based, that is where we need to come up with a very stringent certification requirement for the aviation industry.

    So that takes me to the current set of certification guidelines that has to be adhered as per DO 178C And you will find that out of the 71 objectives which are there in DO 178C, 15 of them are not compatible for data-driven algorithms. Now, if it was a traditional software, that is where we know the rules of the system, the physics, the equations, and all and defining the coverage traceability and explainability is actually-- I mean, it is an easier job over there.

    But when it comes to data-driven solutions, that is where the actual equations of the system could be a black box, as well. And that is where having more weightage given towards data coverage, model explainability, is something that we should definitely can't ignore. And that is where EASA and FAA, the two certification bodies, have been working towards getting a certification guideline out in the industry.

    By 2025, it is expected that we should have first level of the AI ML certification guidelines defined by EASA and FAA. So EASA has been working on this approach from 2019 onwards. And FAA also published a paper back in 2022.

    Having said that, MathWorks is also playing a critical role while working with SAE and EUROCA in order to ensure that we have compatible products to address the certification guidelines in the airborne applications.

    So coming back to the DO 179C workflow, you will observe that, based on the failure categories or different design assurance levels of the software, 26 objectives, which is out there DO 178C can be used for complying a minor safety critical system. For DAL C, 62 out of 71 objectives have to be met.

    So that is where our first approach would be building use cases and solutions towards DAL D equivalent systems. And in our future releases, we are also working on having a DAL C compliant solutions for AI ML systems. Because in DAL D approach, your machine learning model could be a black box model. You do not have to go very deep into explaining the architectural level detail of the model. You can have a high level system level requirement, and furthermore, stringent testing framework testing, and verification framework in order to comply with the DO 178C DAL D level of certification.

    So this is our W diagram that has been recently published by EASA in their paper, which shows right from system requirement definition all the way to verification and validation, with a lot of weightage on data management, learning process, management, and verification, along with integration into the real-world systems has to be there.

    So traditionally, you would have seen, for classic software, a V-level kind of certification has been followed, which is usually the case when you do not have a non-- I mean when you do not have a machine learning component out there in the software. But once you are having a machine learning component out there in the software, that is where the data-related aspects have to be answered, as well.

    So before actually tackling the use case and showing that, with respect to the W diagram, how do we map that particular use case, I will take a very quick poll and try to understand where are you in your AI journey. And you can spend a couple of your minutes in answering the poll that is out there in the poll section in Webex.

    So we would like to understand where are you in your AI journey. Are you working on a real-world application? By working, it means that whether you are in the process of developing a model or you have already deployed some model into production. It could be a standalone system. It doesn't have to go on board.

    Or do you have plans to start in the next three to six months? Or you are here to upskill yourself, gain more knowledge around this particular framework, and then you would be interested to actually take this kind of applications?

    So I see a good number of folks have started answering. Still 50% of you have not answered. But Yeah, I see, yeah, the number of answers are increasing now. That's good. Yeah, so a lot of you are here who have the plan to start out on AI application in the next three to six months. Few of you, a handful of you, actually, are in the room who have been working on these applications or maybe have successfully deployed these applications.

    But a large proportion of you are here to upskill yourself. That is good insight that I am getting from this particular poll. Last few seconds, last 50 seconds, I will read out the question again that, where are you in your AI journey? Are you working on a real-world application? Do you have plans to start in the next three to six months, or you are here to upskill yourself?

    Right, so I think last 20 seconds or so is remaining. But we have got the vote from almost 70%-plus people. So yeah, the results are weight equivalent to what I've mentioned a few minutes back that a good volume of you are having plans to start out on the next three to six months. All right, so that is good to know. I think the poll is ending now.

    So I'll move on with the second phase of the presentation. That is, we will take a use case around classifying the sign on the runway of the airport, so the scenario is there will be a camera module, which will be streaming a set of images continuously, or it could be a video. And the software that is sitting out there, maybe, in the aircraft system, will have the job to classify whether these labels that you see near the runway are informational or it is mandatory to take some action by the pilot.

    So we have this couple of examples in our shipping example in 23B release. And that is where if you try to map this particular example with respect to the W diagram, you will see that there are different modules defined in this particular problem.

    And in 23B release, you will have the complete source code that is accessible for you and different toolboxes, which is mentioned on the screen, is basically required in order to run this particular problem. But at a high level, you will see that to start out with the W diagram on the system requirement definition, you will note that there is this requirement toolbox that we are having in Matlab, which has this very nice app called Requirement Editor.

    Now, the context here is that you can have the whole system-level, component-level, and data-level requirement defined right within this Requirement Editor application. And apart from just defining the requirement in a text, you can link the scripts which are going to test this requirement.

    And at the end of the day, if these requirements are implemented and verified, you can generate an HTML or PDF report out of it. So that single Requirement Editor can save you a lot of time while going through the complete certification workflow over here.

    So if I go a level down, you can see system level, machine learning component level, and data level requirement defined over here. Just taking an example that with respect to the data coverage, you will see that we have requirements defined around capturing different types of weather conditions as an example. So you see fair weather, rainy weather, snow weather, and foggy weather kind of scenarios which has to be part of the data coverage.

    Along with that, I would stress upon that Matlab is pretty capable of handling out-of-memory data. We have a unique data construct called Datastore. And if you are someone who has also managed the data pretty effectively in the directory, it could be a cloud directory or a local directory, for that matter. Datastore can also read the data from multiple different directories, and it can represent that what is the volume of the data that you are having over here so that you get the complete report that what is the number of observation with respect to different scenarios that you are trying to have out there.

    Now, one of harsh realities of deep learning is that your images or videos, upon which you are actually doing the classification algorithm, has to be labeled. And that is where labeling becomes a very tedious task. So we have this data labeler application called Image Labeler, where you can interactively create the different bounding boxes around the objects of interest.

    And not only that, if you are having some image processing function defined, then basically, that function can be used to automate the complete labeling process over thousands of images. So that is where labeling can be done at a pretty fast rate compared to some of the traditional approaches.

    Now, coming to the data traceability and coverage, you can see that in this particular use case, there are so many number of observations around different weather conditions. The data has been captured from different airports also at the different time of the day. The data has been captured or the images has been captured from a distance, from certain angle, from side. And knowingly, we have left out any kind of data around rotating the signs, in this particular case, because that is not something which is practical for this kind of example.

    All right, so automatically using Matlab Report Generator, you can also generate this kind of HTML equivalent or PDF equivalent kind of report which explains your coverage and traceability. So further, if let's say, by mistake, the Labeler application because of automating any of this stuff has inappropriately labeled any of the object, then basically, you can manually correct those things, as well.

    And that is where your testing script will further validate that how much of the data percentage is correct and how much is not, so that could be one of the metric that you can actually set.

    And since we are talking about data coverage, in this particular use case, we have also taken the liberty to play with the different color space, so color data augmentation using HSV color space that is Hue, Saturation, and Value has been used over here. And random scaling of the images up and down by 50% is something that has also been applied so that the different types of scenarios that can happen in reality can be simulated.

    Now, I'm moving to the lower portion of the W diagram that is around learning process management model training and process verification. So that is where in Matlab, there are these two automated-- I mean, not automated. These are GY applications that are out there.

    Now in Deep Network Designer, you can interactively create the neural network right from scratch. And if you are somebody who is interested in leveraging the transfer learning kind of approach, maybe, you want to use some of the publicly trained networks like ResNet50 VGG, GoogleNet, AlexNet, et cetera, those are also available in this particular app. This Deep Network Designer app also lets you validate the mapping between the different layers of the neural network so that you don't run into any kind of training errors while training the model.

    And the second app is Experiment Manager, so obviously, it is a well-known truth that when you are conducting deep learning model training, a lot of hyperparameters have to be tuned. And then maybe you need to also play with different types of models and also Experiment Manager lets you set up the complete experiment where hyperparameter tuning of the respective parameters can be defined.

    And then it can use the parallel processors that you are having. You might be having access to GPUs or HPC cluster. So you can make use of that, basically, in order to run a very effort consuming experiment, which could be automated completely using Experiment Manager application.

    So that is from the tools perspective, but coming to the workflow, you will see that in this particular use case, we have trained three different YOLO networks V2, V3, and V4 and also one matrix that we had in the requirements definition was around choosing the number of anchor boxes, not just randomly but having some metric defined over mean intersection over union metrics over here. So that should be above 90%.

    So that was utilized for meeting one of the system level requirement. But these are the images from YOLO V2, V3, and V4 training as you will see that the loss function in each of these trainings are actually decaying over the number of epochs. But now since all of these models are having a decaying loss function, and maybe giving you a good accuracy on training and validation data set, as well. now the metric that we have defined in the system level requirement is that you should have a good precision score for all levels of recall. And that is where you will see that YOLO V3 model is outperforming the YOLO V2 and V4 in this particular case.

    So moving forward, that took care of our learning process verification. Now we are moving towards the implementation part of the W diagram. So that is where generating the code for different target specific application comes in. We have Matlab Coder, Embedded Coder, GPU Coder, and HTL Coder as different products for different target-level code generation.

    And if you are interested in target agnostic code, Matlab Coder also generates C, C++ target agnostic code. It is also compatible of generating .exe or the shared libraries, basically, that you might be interested in. The thing that is worth noting is that we have this one-line command called analyzeNetworkForCodeGen.

    Before even training your model, if you put in your command along with your network, It will give you that which your layers are compatible for code generation and which are layers are not. And it is advised that you pay attention to code generation if you are building this robust pipeline of certification.

    So we have techniques as part of our Matlab Toolchain around model quantization, compression, as well. So this is our Deep Network Quantizer app, and as you will see in this particular use case, using this quantization approach, we were able to reduce the total number of learnable parameters in the network from 25 million to something like 6 million, which is 75% reduction in terms of the memory footprint.

    So basically, now comes the part when we are looking into the code generation. So in this particular slides, I'm showing you the code generation using our GPU Coder Toolbox. And that is where CUDA code generation is what we are targeting over here. But at the same time, in this particular problem itself, we have also covered the C C++ code generation.

    So while a lot of the script is something that you are seeing, but we have automated application around that, as well. But this is model inference video that is plugged in with our Simulink Toolchain. And you will see that some camera module is relaying the video over a period of time and at such good inference rate, the machine learning model is able to predict the different classes based on the images that are being flashed out. So this is to ensure that your machine learning model can work well in context of the larger system. It can integrate pretty well with respect to the larger system.

    Now having mentioned that, we are also going to look back at the Requirements Editor app again. And if I test different requirements that we had mentioned earlier, you will see that most of the requirements are already satisfied. And these are the different tests which are validating that you have implemented these system level requirements, data level requirements, and component level requirements.

    And as I had mentioned, intentionally, we had removed the sign rotation in the data coverage. And that is what is being flagged because this particular requirement was not validated in this use case. But apart from that, most of the requirements are validated.

    Now, that is what we have as a shipping example for DAL D compliance set of objectives. Now, the third phase of this presentation is around DAL C compliance because as I mentioned earlier, it is having more number of objectives that is covered under DO 178C.

    So the bigger part of the DAL C certification requirement is that your model explainability has to be there, as well as at the same time, different model performances have to be also compared. So the complete source code of the model is something that will be looked at, as well. In this, there is no black box model that you can use over here.

    And that is where we also have this Deep Learning Toolbox Verification Library in Matlab. What it does is it adds random noise onto your data set and sees that your model is still able to give you different types of results using formal verification approach or not.

    And then it also compares the next set of images that are being received as in distribution to the training set that you had defined while training these models or whether it is out distribution. And the histogram looks something like this, where if you look at the image at the bottom, this is condition, which we did not have included in our training phase. And when this conditions are appearing, that is where you will see that your intersection over union kind of metric, or your data distribution is looking completely different from your training set where the maximum peak was somewhere over here in the training set.

    So to elaborate this further, this is video that I'm playing where this particular methodology was embedded in Simulink to see how it is performing with respect to the real runtime monitoring. So you will see that as long as the images are pretty clear in nature, that is where the trustworthiness score of these models are pretty high because the set of images are in distribution.

    But if I flick this particular switch on and start relaying the set of images where the conditions are smoggy, that is where you will see that the trustworthiness score is pretty low. And that is where the model will keep on doing the job of predicting some class, some output class because it will never return Nan value. It will always give some classification to you.

    And that is where whether this classification has to be trusted or not trusted is very important metric over here. And that is where you should gain an insight that whether do you need to go back and train the model for these kind of new upcoming data that you are receiving or whether do you have to be having the same kind of model that you are having.

    And so the final conclusion is that DAL C requirements are pretty stringent. And that is where there is also a requirement that you can have different model architectures that can be trained on different training data set. Although, the output classes would be similar for both of them so that this is devised from a scenario where, let's say, one of the subsystems actually failed. One of these models actually are failing because of some issue.

    Then the other models would be able to give you the prediction. At the time when both the models are performing, then there has to be a good intersection over union between both these models. And then the set of prediction has to be relied upon. Not only one model prediction will be taken into this consideration.

    So MathWorks also had made a presentation at the DASC conference last year, which also won the Best Session Award at that particular conference. And this is within our future releases, you will see this particular model or this particular project, which will be covering our DAL C certification requirements as part of DO 178C.

    All right so the conclusion of this particular session would be that MathWorks has the capabilities of addressing each area of the W diagram, meaning that right from requirement definition all the way up to verification and validation of this machine learning components of the model is something that we take care of in our Toolchain. And as I showed you earlier, every requirement has to be mapped in the Requirements Editor. And there will be scripts which will do the compute validation.

    Then secondly, we have a robust testing and verification library called Deep Learning Toolbox Verification Library. As I showed you in DAL C requirement, some of in-distribution or out-of-distribution scores are also being checked as part of this particular library.

    And then MathWorks is also working with the governing bodies like EUROCA and SAE International in order to come up with some ready solutions or use cases around how do we certify machine learning models with respect to the new and upcoming certification guidelines.

    So with that, I would like to actually conduct another poll for you guys, just to understand that how you would like to implement some of these different applications in your near-term plans. If you are looking at anything around fleet analytics, visual inspection. And by the way, these are supposed to be multiple choice. You can take more than one option over here.

    So fleet analytics, vision-based applications, anomaly detection, fault monitoring, remaining useful life estimation, surrogate modeling, design optimization these are different types of industrial applications that we are seeing, especially in the aerospace industry.

    And then the second question in the same poll is, if you would like to get in touch with MathWorks team, how would you like us to follow up with you? Would you be interested in taking any kind of instructor-led training session around a particular area of interest or technology of interest? Would you be interested in doing a proof of concept on your sample data where our engineers can help you go with this proof of concept on your project using our Toolchain? Or if you would be interested in a master class or hands on session for your team

    Or MathWorks also has consulting service if you are working in a time-bound manner. And if your team is not skilled enough, let's say, on some of the workflows. That is where, whether it is testing and verification workflow, or whether it is model development workflow, we can take up those consulting requirements under discussion, as well.

    So I think this poll is slightly longer than the previous one. I would encourage that each one of you should actually fill the survey, which will help you understand that if you are looking at some of these practical applications to be developed for your industrial use case, how we can also partner with you or maybe guide you in establishing some proof of concept or can educate you through masterclass or hands-on sessions on different topics of interest.

    Yeah, so I think approximately 3.5 minutes are still left in the poll. I think 40% of you have casted some vote. But I would still increase that more number of you actually should cast your vote over here. Right, so I'll let the poll continue because there are three minutes left in the poll. And maybe I will stop sharing my screen. You can refer to the poll in the poll section.

    And I will start taking your questions that you might have posted in the Q&A segment. If you have not posted your question in the Q&A segment, please go ahead and do that.

    Yeah, thank you, thank you, Peeyush, for your presentation.

    Related Products

    View more related videos