Navigating SS1/23

Thank you for joining us. We will just wait one more minute for people to arrive, and then the event will start.

Good afternoon, everyone, and welcome to our event on navigating the challenges of SS 1/23. My name is Stuart Theakston. I'm the industry marketing manager for financial services here at MathWorks in Europe. Quick logistics item-- please feel free to post any questions that you have during the event into the Q&A panel or the chat, and we'll try to cover as many as we can.

I'm very pleased to say we're joined by Adam and Paul today. Adam is model risk management lead for KPMG in the UK. Paul is principal technical consultant specializing in financial services industry here at MathWorks. And with that, I'm going to hand over to Adam to get us started. Perhaps you'd like to say a bit about your role at KPMG first, Adam?

Thank you very much, Stewart. Hello, all. I'm Adam Tan. I'm a lead model risk management at KPMG in the UK. I've had the pleasure of supporting many banks on all things model risk over the years, and I'm really delighted to be with you today. And joining me this afternoon is Paul Peeling who, in a few moments, will step through MathWorks perspectives on the SS 1/23 journey to compliance and beyond.

I think it's fair to say there's been a mixture of excitement and trepidation in the world of model risk. And on the bright side, a window of opportunity is emerging to shape the MRN functions of the future. And the spotlight on model risk in the UK has never been greater.

And as many of you know, we are about 9 to 10 weeks away before SS 1/23 goes live on the 17th of May 2024. And many firms are, as you would expect, in the midst of performing, or in some cases, finalizing their gap analysis against the requirements of SS 1/23.

And on the back of the gaps, identified firms are thinking about and drafting their remediation plans, redesigning MRM frameworks, calling out dependencies, and shaping implementation plans, and the list goes on. And I think it's fair to say the industry recognizes there is still much to be done. And in many cases, the devil is indeed in the details.

And it would be fair to say that full compliance with SS 1/23 will not happen in one fell swoop even for firms who've had the joys of complying with the feds SS 11/7 and having internal models in their respective inventories. So whilst many of us would agree that it's an evolution rather than a revolution, I think it's fair to say that model risk management has indeed reached an inflection point. And firms will need to take a step back and really ask themselves, do we actually have the governance processes, controls, and mechanisms in place to effectively deal with the proliferation of new model types and new quantitative techniques and with the flavor of the day AI gathering pace?

How do frameworks contend with artificial intelligence in the context of model explainability, bias, ethics, hallucinations, data drift, and the list goes on. But there are many other questions that abound on, say, the topic around DQMs, or Deterministic Quantitative Methods. How do we define, discover, identify, and document these DQMs? Can we really leverage other governance and controls such as existing governance and frameworks of EUCs as an example? Or do they not serve the same purpose and require some reconfiguration?

Model inventory-- do we really truly understand our existing model inventory or model estate? Will the model inventory benefit from further enhancement and enrichment of model attributes? Could one potentially use it as a model validation planning tool in the future?

Policies is an interesting one as well. I mean, what about policies and standards? Are they good enough? Do they sufficiently capture the requirements at the activity operational level, or do they require an uplift, and are they well understood by stakeholders?

How well are they embedded? The SMF question has also come up a number of times, and it's not simply identifying who the SMF is. But if you take a step back, it's about understanding does the MRM framework or the ecosystem sufficiently-- is it sufficiently robust and transparent and effective to support the SMF to discharge their responsibilities?

And lastly, I think MRM frameworks-- I think would MRM frameworks and operating models benefit from some re-augmentation, re-engineering of processes, and potentially the rationalization of model controls as well? So these are some of the many examples that the industry is currently grappling with. And against this backdrop, Paul will bring some of these challenges to life, especially in relation to model governance and the model life cycle. Thank you, and over to you, Paul.

Thank you very much, Adam. So this is great introduction to this area, which is-- there are many different angles we can take on this, and quite a lot of challenges that are still faced by the industry even with this upcoming date. So my name is Paul Peeling. I'm a consultant at MathWorks, and as well as working in the financial services industry, supporting the development of a model risk management platform at MathWorks known as Modelscape.

So throughout this session, we'll show you some of the approaches that we're using with the Modelscape platform also to address these So the topic we wanted to cover was journey to compliance and beyond. So I'm assuming the majority of the audience will have read SS 1/23 and will have observed that there is a principles-based approach taken to MRM.

It's not simply prescriptive, it's not simply a checkbox framework. It expects and instills certain competencies within an MRM function and also within the expectations that that MRM function is expected to provide. So we're not going to cover everything today, but we're going to cover some of the key challenges that we've heard that relate to what Adam was describing, and we're going to cover some topics.

So they'll include data lineage-- so exactly how is your data, both in your inventory and the data that you monitor, acquired and managed? Clearly, model lifecycle is a very key component of that, but how does that model life cycle fit in with workflows automation and planning. And then documentation still forms a key part of the MRM process, whether that be the actual documentation of the models, or the processes, and procedures, and the standards that they are working towards.

So over the course of the remainder of the time, we'll be diving into these areas. So just a brief summary of some of the questions then when faced with SS 1/23 that we hear about. So the first one is the establishment of the clear roles and responsibilities. Like Adam was mentioning, the SMF.

This is could be a new role with the institution, but then clearly the roles and responsibilities of this new MRM function needed to be found. And how can they map onto the existing banks procedures, landscape, and so on like that? And from there, then there needs to be appropriate levels of communication and a reporting structure.

A lot of model risk management requires four eyes principles, requires approvals, and a correct oversight. How can this be embedded into the roles and responsibilities, and how can this be mapped to the reporting structure in the bank? Clearly, model risk management is of little value if the data that is managed by the inventory is of poor quality.

The same goes for models, same goes for the model inventory. So a poor quality model will not give good outputs. A poorly maintained model inventory and other accompanying systems will not provide accurate and timely reporting data.

They will give at best an incorrect, at worst a misleading, representation of how model risk is being handled. And of course, the models themselves also need to provide reliable and accurate results, as part of a model risk management framework is to ensure that models are used appropriately. And reliable and agile with respect to changes in the data, changes in the operating environment, changes in regulatory expectations.

The final challenge that I say is then, given all that, having an infrastructure in place to ensure that the models are functioning correctly. So there's an increasing sense of centralizing the way that resources are used to both work with models-- monitor models record them as well. So this concept of a centralized inventory to have all this information together and a platform-- scalable and flexible platform on which to run on.

So I'm not going to go through every single principle in SS 1/23 nor go into the details of those, but I'm just picking out various aspects that we've seen that can then relate to some of the topics I was covering first. So principle one is about model identification and model risk classification. So here, we need to be thinking clearly about what is the definition of a model, the scope for defining that, and being able to modify that.

There are challenges around the use of non-deterministic calculations about end user computing site that Adam was mentioning. And then once those have been categorized, how then are they organized and managed through an appropriate mechanism So in terms of deciding whether or not to onboard a model in the model inventory, a set of decisions needs to be made and monitored.

And also then once the models are in the inventory, then one popular approach is to then assign risk ratings, risk hearings to those models. They form a bunch of data that's coming into them there. But they also are helpful for planning, as in certain higher risk models generally have more oversight applied to them, and usually are reviewed and validated more frequently.

So let's just pick that from Modelscape. Exactly what are what are we talking about here when we're looking at that here? So just going into Modelscape in the inventory from our perspective, then we can see the model identification principles brought out right here.

Then, we can go in and start to answer particular questions. This is a dynamic questionnaire built in on deciding whether or not something or not is a model. Create it, register it in there, and then we will have a large quantity of information to then populate as part of that.

So once we've got our model, then we may end up with something looking a bit like this. This is clearly large amounts of information that needs to be presented, managed, and handled as an aspect of that. So this is an exhaustive view of many of the metadata that's expected to be involved in a model.

And as you can see, this particular model doesn't have a risk rating associated with it yet. It has various activities that are required. We can show and hide the information. So these are the MRM activities.

This is a model in flux. So we can see that various workflows and approvals are taking place, and then the model risk rating there. So just an illustration of what's going on here.

Now to make changes to this model, we can then, for example, edit this directly, and then start to look at some of the aspects of here. So this is the view where we'll see some of the workflows that get applied. And you can see right at the beginning here, we have these concepts of risk tiering here.

That's one of the things that will apply to models based on the data that we know about the model-- what risk ratings should we apply. And also certain aspects of the model may not be editable for various users. So there needs to be oversight applied to that here.

So even with a fairly simple straightforward onboarding of a model populating of data, we start to see some of the complexities that need to be addressed through an appropriate governance mechanism. So for the first part, we're going to talk about the risk tiering here.

What is risk tiering? So risk tiering is effectively a calculation or a determination. So on the bottom of this diagram from Keenan 2015, there are a bunch of inputs that form part of the determination. So how good was the data used to build the model, how tightly is it specified, any issues and the implementation of use?

Any flags in this here will then result in an increase in the intrinsic model score. So this is risk inherent to the model based on its own sense before we start to consider other aspects of how this model is used. Alongside this is then the exposure. So how much is the bank exposed to the results of this model? Any risk mitigation that's being applied?

And then finally, this will result in often one level or two levels of scores, but ultimately a model risk score which then may be numeric. It may be a categorical high, medium, low-- that sort of aspect there. So this is, all in all, fairly straightforward. I mean, you have this sort of data at the bottom within your model inventory-- all of this here.

And then adding up and exposing these should end up with something that you can calculate from the inventory. But then you relate to some challenges that you have merged. So for example, as Adam was mentioning, models are being used more and more with broader scope.

There are new types of models, such as AI models, use of unstructured data, that increase their implementation risk. And of course, regulations like SS 1/23 and others around the world mean there's increased oversight of model risk. So what we would expect to see given these inputs is we would start to expect to see a shift towards higher risk tiering across the portfolio of models simply because some of those inputs are there. And also the increased use of risk mitigants where we have to apply additional processes outside of the models to manage those risks there.

And what this result is is a response that means that models have to be validated more frequently. So there's an increased pressure on doing that there. And in addition, more forward planning needs to happen. Does the-- are there sufficient validation resources there.

So the-- as Adam was mentioning, can the inventory be used to drive planning and validation? Well, yes, here's a particular example, where the input-- the risk tiering input results in a more frequent set of validations. So let's have a look a little bit about an example of how risk tiering may be achieved.

So the first example, we've got here is a fairly straightforward questionnaire that's built directly into the model inventory. So we'd open it up. There are various papers of particular risk tiering mechanisms that can be used, usually either forming a decision tree like this one or a set of scorecard questions here.

So as you can see, I'm busily populating the various fields there. I can also apply overlays on that information to actually both provide the inputs that I was describing on the slides before, and then start to insert overrides. That's the example of that right hand column, which are overriding the existing score. So in this case here, although I think the model is a medium risk tier, I've actually then gone and assigned a low risk tier for whatever overlay reason there might be.

So what that actually means is that the model risk tiering was applied to a specific model in question, and we were asked a series of questions about that model. What was the exposure, what was the score, and so on like that. And many will tell you this is quite a limited way limited approach to risk tiering, because you're only considering the model in its immediate context in the data that is actually stored in the inventory, and this is simply not what the case is the model.

So the model here in the schematic is just represented as a black box here. But what about the data that's used to drive the model? You could see in the Keenan paper that data quality was a very important concept going into model risk, yet that simple questionnaire didn't mention data quality at all.

The code is a very important part of the implementation. These days, there are a lot of modern frameworks for developing models, both traditional and using AutoML. And so code and the quality of the implementation can very much feed into that results there.

There needs to be adequate oversight of the documentation that's associated with the model. So being able to review, look at the documentation, extract out assumptions, limitations also needs to form part of that. And then quite importantly, it's-- a model is fine-- will have no risk if it's not used.

But when you need to assess a model risk rating, it needs to be considered in the context of how that model is being used, both its current and proposed uses. And then finally, models rarely exist in isolation. The outputs of some models will feed into other models.

These are often called feeder models, and those outputs may be used there. So actually, what we need to start looking at is a more interconnected view across multiple systems. And therefore, a modern day inventory that can help with this doesn't-- has to combine and link all of these things together. So anything spreadsheet-based-- spreadsheet or a list of models will quickly hit limitations when you're trying to assess something of this complexity.

So some of the things you'll see-- that we're going to see how to address this problem is that starting to think of everything having a life cycle-- a set of stages through which it moves to be. Also a better treatment of data. That's data that's used to train the model, but also the data that is the model itself.

And we'll show how the risk tiering uses that information from its interconnectedness. So within that, let's actually dive into the first topic, which is the application of data lineage and data governance techniques to model governance and model risk management. I've separated these because data lineage and data governance, although often combined to the same thing, actually have subtly different uses in this system.

So let's start out. Data lineage is the first one. Lines between data-- that's what lineage means. So you should be thinking that a data lineage will be able to answer you, where did that model come from?

What was the data used, and what process-- what stage was the data used? This is why we need a model lifecycle and the data lifecycle. Was the data used to validate the model, or was the data used to train the model?

And then when the model is being used, what data does this model consume and produce? These are almost always three or more different data sources, some in different systems, all with different levels of regulation and governance applied to them. In general, though, the data lineage questions can help you address principles three about the model development and implementation and principles four about model validation.

And so these are very important when you're addressing those principles to have a good angle and understanding of where data is used by model specifically. But that's not the only way-- place you can use data governance. The other side is on model data governance. I've just showed you some videos and screenshots of a model inventory developed in Modelscape.

You can see that that actually has a lot of fields and interconnectedness and links there. These are all-- that's all data itself. And a data governance apply to model metadata can tell you more about what we call the meta model. How do you define and how do you talk about models? What do these things mean, what does ownership mean, and so on like that.

And then in terms of an adequate oversight and process, who changed the model? I mean, who changed that field in the metadata? And then what is the impact of making those changes? And so from the data governance perspective, this very much supports principles one-- the example given, risk tiering-- but also the governance questions in principle two.

So bear those two aspects of data lineage and data governance in mind. And let's just show an example of what can be achieved today with that here. So here is a look at the same set of models that we're talking about for the inventory, but much more from a data perspective. So this is a data governance view.

And in particular, I'm looking for any models that work with a type of credit card data. So as you can see here, the interface is actually listing out a bunch of different models. Now, hive here is a type of database. So here we have some data stored in a database very simply.

And as you can see, it's capturing some of that information, but will also help us answer questions, like where was that data used in models? So here, we have a lineage diagram, and you can see here that the data set is used by two models.

And here's one of them-- LGT credit card. And it shows you how that model is derived from that data set using a process. And that process will be registered with your model development tool to achieve that lineage there. So ideally, when you're building models, you capture this automatically.

The model itself is here. And you can see that it's got things like properties. You can attach data. So I'm going to say, well, this was training-- the data type we used was for training, and so on like that. And save and combine those with that-- those user-defined properties.

Another aspect you can apply is data classifications. So PII-- Personal Identifiable Information-- is something that can be applied to a data set. And then you can see we have the options to then propagate that classification through to the underlying models that you're using it.

So here, I can propagate that from the data and also apply validity periods. So how long do I want this classification to apply? This, in practice, is a very useful communication tool that allows you to then, for example, alerts owners if there's an issue with data.

So you would say there's an issue with this data. I will apply this classification. All downstream model owners or model developers will be alerted of the fact that data is a problem. I'm sorry about the screen freezing here. I'm just going to try and skip through the video a bit more.

Now, let's come back to the beginning I'll just pause here. So although the video freezes, you can actually see now here that this model was based on that data set. And it has this propagated classification from it here.

Another aspect-- I'm just showing here that the video skipped forwards-- is then the audit trail. So all of these changes are then monitored and applied automatically. So every single change that may be made through the user interface, as you could see-- or automated changes like the propagation of a classification-- are also tracked entirely through the-- they're tracked entirely through the audit trail. So there's no possibility of missing steps in the process, missing information, or understanding who or what did what, because it's all tracked through this centralized data governance system.

So just in that brief video, you saw the data lineage and the data governance provided. So just as a reminder, we had this data lineage diagram here. Now, this is very simplistic, but because it's a graph, it can scale to the large and intercomplex relationships between different data sources and different models there.

And of course, if you've ever been thinking about model interconnectedness, I think you probably realize that this concept between data model can then be extended to cover not just the relationship between a data and a model, but then the relationship between a model and another model. And in many cases, when you practically do this, those models are actually related by intermediate data sets anyway. So capturing data lineage along with model lineage is a very natural thing to do.

We also showed classification propagation. So this helps answer the problems of any changes upstream to data or even models-- can be propagated through the network to downstream changes. So this is one way that any institution managing multiple models can start to develop some agility by being able to flag, add classifications and validity periods to upstream data sets and have those propagated to anything downstream automatically. And then these can tie into the notifications framework that you'll see later on.

The corollary-- we also looked a bit about model data governance. Now, the video skipped this, unfortunately, but as you can see, this particular model has a bunch of relationships between it here. So for example, the life cycle, the different versions of that model, the process used to generate it.

And then a set of what we call references that refer to the portfolio, the type of model, something like that. And also this can be then represented as a graph. So you can actually see graphically the relationship between the models and its references.

Now, these are static. So there's a slight difference between lineage and relationships-- that these relationships don't imply a directionality, whereas in the lineage concept, that's the very case. Something is produced by something else. So we can move on to principle two now.

This is governance. So we did already cover some aspects of model governance from a data perspective here. But the principle also talks about oversight, such as the board, the culture, the risk appetite.

And then appointing the SMF to ensure that the MRM framework is followed with effective practices. Now, this goes beyond just the simple auditing of models to know who did what.

But we know every system here will have a very precise audit trail of what's been going on. But then we need to start to think about the life cycle and the changes to models that are driven through this life cycle. So a model that's in draft phase will be very different from understanding what the model is in validation or approved phase.

And so the model evolves through this life cycle. Data gets added, documents get added, and so on like that. And also various individuals need to be involved in that to approve a model and move it to the next phase.

So we're going to dive into some of the practical implementations that can be achieved today with that. So starting here again-- so back to our favorite model. And we're going to now look at some of the fields which aren't editable. So you can see some of these fields here, like status and so on are editable by the model owner here or the model developer, but some of them are locked.

And in this case, the procedure is to actually have a second pair of eyes to approve it. So you bring up the ability to lock it, and then you can click and make the changes to the model there, and then request that a change goes on. So the reason for doing this rather than doing it outside of the inventory process in the email is then to have full tracking governance of that there. So not only is the model owner making this request, but the model validator needs to approve it and have that automatically applied into the inventory.

So just what we looked at here is what we call a permissions-based approach to model data. In here, that the user interface locks certain fields. And then we have the ability to make a request to change those fields and get those approved. And then from the validator side, what you end up with is something's looking like an approval.

So this particular example, we're driving through an approval mechanism. So you can see here you have the ability, when you're seeing this approval, to see what the changes are, apply comments, and choose your response. So you can approve it, you could reject the change, which means no changes made to the model, escalate it.

You can follow up, reassign it. So this is a very simple task-oriented based way that works with various products. So this is the Microsoft Teams way of approaching this so you can apply this directly and have this hooked into your inventory process just by making changes to fields and having those approved like that.

You can take this further than what I've shown today. You can take this for onboarding the entire model into a model inventory, and the developer or the owner proposing this can fill in those forms-- fill in the form that you've provided in the inventory. And then all of that information then goes into the approval mechanism done through here. And this is nice because then it allows you-- for example, if you're out of office, then the approvals can be delegated and sent on to someone else who can handle it within the team.

And then just really, how does that actually work? Is that behind the scenes-- going to give you a little bit of a hint here-- the inventory actually gathers all the information in the request and then sends it off to a trigger. So you can see some of this information is about the particular model that's being sent. That approval service and there's various approval products you can use, or you can use the ones within models get themselves.

Then give you the ability to directly review that information, approve it, reject it, and so on like that. So this is a very simple way of building up a workflow around changes to the model inventory that can be used to support model governance and model oversight by correctly understanding your organizational structure. This is, however, quite simplistic.

And we'll move on to now a much-- a somewhat more realistic and workflow-driven approach, where you can see actually various stages of the life cycle of this model presented at the top as a series of banners. And then each stage has several steps, several states-- several places in which approval review feedback can apply here.

So this looks like here, from this perspective-- again, if I'm a model owner, I will begin by starting to kick off the workflow which, in this particular inventory, is called a submission. So you can see there's a couple of submissions already in progress and the draft. I'm going to write, technology is failing me here.

So it's stepping through anyway. I'm going to have to step through this manually because the technology is not-- it's missing out various screenshots of the approach. So I'll just go back. Yeah, there's a lot of lag, I'm afraid.

Sorry. So here, this is showing before there's four stages here. There's a submission, validation. Then two levels of model governance approval-- one at a working group, and one at committee.

So larger organizations will have specific working groups devoted to particular areas of risk-- market risk, credit risk, and so on like that. Whereas the model governance committee will often include the SMF that's overall reported here. So from the submission level here, I'm able then to-- I'm not able to-- the video is dead.

I'm going to change plans slightly. And actually, it's live demo time. Here we go. I think this would be easier to see for the audience.

So we were looking at a particular model. Let's take this one. As you can see, there's several submissions. So this is actually in draft state and validation. I can click through this.

You can see the four different approaches here. There's no documentation associated with this yet, so we could link that later on. But anyway, you can then start to change this based on the different validation types, and start to work with this, and build this through, and then move this into the validation phase. So we'll move it into the validation--

Paul?

Yeah?

Paul, just a quick question from me. So from a validation perspective, obviously the model risks tiering in terms of complexity materiality, et cetera, would drive the extent of the intensity and the frequency of validations. Now, what about from a model performance measurement and monitoring perspective?

Model performance measurement is mentioned in principle three and performance monitoring is mentioned in principle four. Now, to what extent could any sort of breaches, et cetera, trigger a need for validations to be done a lot earlier than initially planned? Is there a mechanism for that to happen as well?

Yes, separately. So here would be where-- whenever you develop a new model with the risk rating, it will assign a revalidation due date. So typically, according to a process that you say-- that a high risk rating model will have a due date, as in one month's time. But if, for example, I set it to a low due date-- low risk, then it's set to a year's time.

So that's one particular way of starting to match up the risk rating with the tiering and the revalidation. However, then if you were to see the model breaches coming here, in a separate system, those breaches would then come in, and then they could automatically readjust the due date to then either trigger it within a week of the breach or so on like that. So that would be one approach there.

Another approach would also to be involved to start to address things like findings and issues in order to then-- these can be raised not only just during the validation phase, but also during the operation of the model, or a model monitoring exercise that can then be used to re trigger this process at the same time. And this is partly why using an inventory in these processes to derive this means that you can then start to look at these due dates. And the amount of work associated with them across a model, and then be able to build a more holistic view of what the validation team needs to do at various points. So there's just a few--

Yeah.

So there's just a few aspects of what you're seeing here then is that-- I've just shown you, just by talking to Adam, the interplay between actually doing a model risk-- or redoing a model risk rating assessment based on the validation step. And also then the planning aspect here, which modified the validation. The next time that this needs to be reviewed based on that information at the same point.

I will also quickly just explain that when you're dealing with documentation-- you'll see documentation later. If I just actually quickly link documentation to a model-- so I have a model there. If I go back here-- so it's already there, so that's why I couldn't see it.

So if I-- just that I can link any existing models that are attached to that library, and locate them, and add them at various stages in the life cycle. So as the life cycle progresses, these can be progressed and added there. Can also then start to address things like model interconnectedness and use this to search it from there.

So as you're starting to see, based on the particular risk rating that we had, we're now able to construct a much more-- a much better picture of the model, its interconnectedness, its data, and so on in order to perform this final model risk rating. And this then can be captured and is snapshotted with every single document, so every-- with every single submission.

So each time a change is made, you have this traceability alongside it there. There's a lot of additional information that's usually required. I'm not going to fill them in here, because one of things later is just show you how that this information about the intended, and actual model use, and stuff like that it's not usually input directly into the inventory.

It could be, but usually, it's not. Usually, it's sourced from model documentation. So instead of using it this way, we'll source it directly from the model documentation.

So just a couple ideas. Just seeing what you're seeing, but I had to do it live instead of the videos. So as part of that, you also started to see the process that was mapped out as a lifecycle. So you had those four stages at the top.

And these actually correspond to these four vertical swim lanes that you can see here in the life cycle diagram. The first, second, the third approval, and the fourth stage of approval, or the rejected and so on like that here. So these are the-- this is a life cycle attached to a model change rather than a model.

So you-- it's an appreciation of the flexibility you need to not only have models follow a life cycle, but data, and documents, and so on can follow the same life cycle. And the way that then this is actioned is through then looking at the transitions here. So for example, we looked at this full submission to validation transition that happened there.

We click through in the form. So we submitted the evidence, and then the risk rating appeared here. But what you can do is then you can associate that transition with a workflow.

So that workflow might, for example, be automated-- operating in something like Power Automate, which is a tool that's used in Microsoft to then integrate with teams, and planner, and some things like that. So that's one particular approach. And you can see here-- this is when I showed you before the approval. This is how it was implemented.

Now, what's nice is that all of the information from the inventory is passed through into the Power Automate template. So I can display all of the information that I have there in the approval mechanism. And then that information then can then get propagated directly back into inventory later on.

Another approach, which is quite popular, is an approach called business process-- process notation modeling here. So this involves drawing diagrams, like we can draw here, which has these boxes as actions. It has these decision points, which relate to various ways in this workflow can then change based on effectively user input.

So when I opened the form, I was able to request or cancel it. If I cancel, I go to this termination point. If I request, I then called out to Power Automate to bring up the approval form. And then the-- depending on what that action was there, then I can escalate it, which repeats the process to the person's manager or delegation.

The approval then updates the model, and the rejection then again terminates the activity. So I'm able to map this out and then attach each individual elements to the things that you saw before. So the first one is that hook-- is invoked by calling the service with all of this data from the inventory there. And then the approval mechanism was implemented here.

So it's a powerful mechanism to combine these process modeling diagrams here with the ability to then attach these to executable frameworks that enable you to then drive a very scalable efficient model risk management process. And the example that Adam was referring to here is that this workflow doesn't have to just be driven by a user clicking in the inventory. It could be driven by a model monitoring process that started a breach.

And then it would follow through exactly what we're saying here. An example of updating the metadata would involve updating the validation due date and so on like that. So having this ability to then treat the inventory as a business-- a set of business processes is a very powerful.

The last topic I'm going to talk about relates actually more specifically to model validation. We've talked a lot about the inventory, the data it manages, and the governance around that data and the processes. But a lot of this data around models is actually documentation-driven.

If you're a validator, your work often involves testing the models both using numerical techniques and existing data, like you see in this notebook format here. And documenting those tests plus also documenting more qualitative information, such as things like the assumptions, the robustness, the soundness of the model, and incorporating that. So an example then of actually creating this documentation you can see is here. So often in a notebook format here, we have a validator exploring, clicking through, visualizing, and working with the model.

So they're getting a lot of insight into this model. They can also look or hide or the code to understand how it works from that perspective. And then they can export this, for example, to various documentation formats. So this export is really a capture of the work that they've done over time.

A more powerful mechanism is then to actually deal with an existing model document like a template, like this, which is analysis of this particular model. And then synchronize the information that was in the notebook with the model document here. So you can now see that we've created these areas of content that can be replaced by visualizations in the chart and also by actual free text and stuff like that.

So this information here is great to be captured in a Word document-- in an authoring format here, but isn't necessarily that easy to then link directly into the inventory if you're trying to then locate some of this analysis at the same time. So what we then support is then actually for validators and developers to continue doing their work in the documentation environment-- so that's usually the notebook environment for the numerical modeling and the coding-- together with a Word document or another collaborative document format to then describe how that goes there.

And what that can also do to feed into the planning process is then you can create a dashboard on the progress through the model, as you can see in here. So here, we have a set of validation routines that are validator is expected to perform. Most of these are numerical.

And then the validator can go through and describe where they have been implemented, and they can be checked, and so on like that. And you can start to build a progress report over time. Each individual item can be assigned to an owner in the process or a sub-owner. And milestones attached so all this information is then captured while you're doing the documentation.

This looks, in practice, a bit like this, where you're starting to link to upstream documentation. So in this case, the particular regulatory metrics you have embedded within the document the ability there. You can also start to create links between the different documents and track process between phase and the testing.

You can also import existing documentation with its rich text into the database so you don't need to completely start everything here. And you can start to then here extract out the information and add in your own ability. So for example, every part has a description and a rationale. This can help then with the collaboration between the three teams-- the first line of defense, second line of defense, and then also have these managed.

So this is a view-- a documentation-focused view of the inventory. And you're seeing quite a lot of the details that we're able to provide by being able to deal with large structured documents, track progress against them, and then link that in with the inventory. So documentation is not a separate authoring process and then simply copy and paste or upload as a link into the inventory. It's a much more collaborative mechanism.

Some of these can then actually be driven also by testing and automation on that side as well. So anything that relates to making calculations, running tests, and so on like that doesn't necessarily need to be done on the validator's desktop machine. It can be done through a DevOps platform and automated that way at the same time.

And just an example of the ability to then link these two there, we have for example linking these tests here not to manual verifications, but to Python code that implements that, and then exporting the results from the CI system directly back into the documentation. And so this can then both combine and track qualitative work, where people are writing the documentation together with numeric work, where people are testing, writing code, and implementing the code at the same time.