Co-Authors:
Carolina Biagini - Senior Quantitative Analyst - Kevin D. Oden & Associates
Amul Bhatia - Partner - Kevin D. Oden & Associates
As Credit Unions (CUs) shift into automation, data-driven processes, and Artificial Intelligence (AI) to transform to digital banking, the use and complexity of models utilized will grow rapidly. While these technologies provide immense opportunity, they carry risk as well. Financial institutions should anticipate an expansion of in-scope AI and Machine Learning (ML) applications given regulatory predisposition toward inclusion. The OCC has already claimed that many AI/ML tools should be considered models, and their model risk should be properly managed[1].
Developing a sound, cost-efficient Model Risk Management (MRM) framework appropriately scaled for the institution is a key element of making the gains from increased model use sustainable. The framework discussed below aims to help in that endeavour.
Model Governance
“The real mechanism for corporate governance is the active involvement of the owners.”
- Louis V. Gerstner, Jr.
As SR-11-7 states,
“A strong governance framework provides explicit support and structure to risk management functions through policies defining relevant risk management activities, procedures that implement those policies, allocation of resources, and mechanisms for evaluating whether policies and procedures are being carried out as specified.”
When tasked with building out a MRM program, many CUs find it difficult to know where to start. That is where the notion of a model governance framework comes into play. Once the framework is established, the process of assigning roles and responsibilities (R&R), followed by the development of policies and procedures (P&P), becomes easier. Developing this framework and reviewing it on a regular basis lead to strong, sustainable model governance.
Developing a framework
Is it commonly claimed that the extent and sophistication of the governance framework should align with the extent and sophistication of model usage at the institution. But how does a CU go about putting together such a framework? The first step is determining objectives. Next, the CU needs to determine what it takes to achieve those objectives. Finally, the CU needs to assess what it takes to make those achievements sustainable. A concrete example of an MRM framework looks like this:
- Objectives:
- Identify all models used at the CU.
- Assess the risks associated with those models.
- Assess the institution’s appetite for the current model risk profile.
- Manage risk within the institution’s appetite.
- Plan to achieve objectives:
- Develop a model identification process.
- Develop a risk assessment process for individual models and in aggregate.
- Determine how much model risk the firm is comfortable with.
- Develop a plan to reduce or retain risk within the firm’s risk appetite.
- Plan to make objectives sustainable:
- Develop a revision process to ensure objectives are sustainable.
- Develop policies regarding R&R in #1, #2 and #3a.
- Develop a governance framework to review the execution of the plan, to assess the performance of individuals and groups in relation to P&P, and to make recommendations for refinements.
The CU’s model risk objectives should be placed in a risk appetite statement where the firm’s appetite (and capacity) for model risk is prescribed. Once these objectives are initially set, a governance framework should be developed to achieve those objectives and make them sustainable. As described above, governance is the tool used to ensure the CU achieves and sustains its objectives. This is very similar to how the U.S. Constitution serves to achieve and sustain the objectives of forming “a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defense,” etc.
AI/ML risk management should be integrated into the firm’s broader risk management framework. The oversight of AI/ML models should be consistent with the standards and processes in place for traditional models. Nevertheless, traditional MRM frameworks must be enhanced to incorporate the dynamics and integrated risks associated with AI/ML models. AI/ML models directly obtained from open sources or developed by third parties should be subject to the same MRM standards that apply to internally developed AI/ML models.
Who Is Responsible for Model Risk Management Governance?
MRM and governance can become challenging for many institutions, particularly for smaller CUs with budgetary constraints. Ultimately, it is the responsibility of everyone at the institution – starting with the board of directors, extending to senior management, including business heads that own and use the models, and naturally risk management and audit functions. Each of these stakeholders will have R&R in a properly functioning governance framework. However, the Chief Risk Officer should play a central role in designating someone as the CU’s “Head of Model Risk Management”. That individual (or group) may have additional titles and responsibilities depending on the size of the model inventory and the complexity and risk of the models. In the end, the Head of MRM should ensure that P&P are developed and implemented and that those P&P are effective, efficient, and commensurate with the risk footprint and risk appetite of the institution. We emphasize that the Head of MRM works for the CU as the bottom line is that strong model risk management is not only about P&P but also encompasses revenue retention and, in some cases, revenue enhancement.
A strong governance framework should provide reporting structure, approval authority, dispute resolution, and escalation procedures and set the enterprise model risk profile and tolerances within the stated risk appetite of the board. It is important that the board of directors be the driving force behind a strong MRM governance structure, as the board is ultimately responsible for the success or failure of the firm. As SR-11-7 states:
“As part of their overall responsibilities, a bank’s board and senior management should establish a strong model risk management framework that fits into the broader risk management of the organization. That framework should be grounded in an understanding of model risk – not just for individual models but also in the aggregate. The framework should include standards for model development, implementation, use, and validation.”
Risk Appetite Statement
“The best is the enemy of good.”
- Voltaire
CUs, large and small, should have a risk appetite statement approved by the board, that lays out the objectives of the MRM program and will, in fact, guide its development and sustainability. It does not have to be perfect at first. The statement will evolve as the CU evolves. Something as simple as the following is a good starting point:
Identify all models utilized or to be used at the CU, assess their risk to the institution and manage these risks within the CU’s appetite. This entails at minimum ensuring that all models are validated prior to first use with limited and time-constrained exceptions; ensuring models are revalidated on a regular basis commensurate with the risk of the models; resolving issues identified in the model validation process in a timely fashion; monitoring the performance of all models and taking prompt corrective action when model performance deteriorates below acceptable thresholds; ensuring models are properly documented and users know the appropriate use of any model they utilize; ensuring model changes are properly managed and reviewed; and ensuring model dependency is understood and properly managed.
This simple risk appetite statement lays out the broad objectives of the MRM program to be developed. The first objective in the example above is the identification of all models utilized at the firm. To do this, the CU’s governance framework must provide a model definition. We will discuss the model definition and identification process in next week’s article, as well as the development and maintenance of the model inventory.
The level of AI/ML risk that is acceptable to institutions is highly contextual as well as application and use-case specific. Risk tolerance can be influenced by policies and norms established by AI/ML system owners, organizations, industries, communities, or policy makers. Additionally, risk appetite is likely to change over time as AI/ML systems, policies, and norms evolve. Hence, the MRM framework should be flexible enough to accommodate risk practices to the applicable law, regulations and use case settings.
Model Risk Reporting
“If you don't know where you are going, you'll end up someplace else.”
- Yogi Berra
Reporting requirements will continue to grow as the board, senior management, clients, investors, and regulators seek to manage their interests related to the CU. Reporting requirements around model risk management are not immune to this trend as the use and complexity of models continues to grow at CUs. When outlining reporting requirements, it is important to focus on the objectives laid out in the MRM framework and how those objectives impact the board. This will naturally lead to individual business/functional line reports as these leaders will need to understand their impact on board-level reporting.
When outlining reporting requirements, we recommend start by asking (and in turn answering) the following question: What does the board need to know? Though the answer depends on the CU, some core requirements are listed below:
- What are the most important models for the firm?
- Importance should be stratified by risk type.
- The model risk ranking/rating can be utilized to determine importance.
- Under what conditions are the most important models expected to work well and not work well?
- In what circumstances are they likely to break down?
- Collectively, are model outputs credible?
- What “moves the dial” in terms of key assumptions or judgements?
- Are those assumptions and judgments reasonable?
- What are the key dependencies, assumptions and linkages in risk?
- Are all key models dependent on a limited number of factors, or
- Are models essentially independent of inputs and assumptions?
- Is the governance framework working properly?
- Are we adequately monitoring and describing/reporting on the state of model risk?
- Are we quickly identifying gaps in model governance and risk assessment?
- Are we working with business, risk, and audit to provide adequate oversight?
- Are controls maintaining model risk within acceptable bounds (i.e., risk appetite)?
Each of these elements are critical in achieving the CU’s model risk framework objectives and achieving sustainability. The challenge is efficiently organizing the risk into (at least) the “buckets” identified above to reduce complexity and to increase clarity, so board members understand the risk. Starting from this perspective also better aligns business and risk reporting with the information the board receives.
To efficiently digest and utilize this information most institutions now have a risk committee of the board. This is an oversight committee that guides the risk management practice and objectives of the firm. However, at many smaller institutions model risk management is new to the board and has not been fully integrated into the board risk committee functions. We next discuss best practice in this regard.
Risk Committees
The risk committee is a management level committee that has a crucial role in any CU, providing critical oversight and guidance to management to execute business within the risk appetite of the firm. Most CUs, large and small, have developed risk committees. However, including model risk management as a responsibility of the risk committee will be new for many smaller institutions. To accomplish this, the risk committee needs to have a member(s) familiar with model risk and its various components as well as the best and current practices in managing that risk. This is a challenge for most small CUs that can be minimized, but not alleviated, with ongoing training and the hiring of a model risk expert (presumably the model risk manager) and her inclusion on the committee. To be effective, the committee members should be representative of both the risk management side and the business side of the CU. There should also be P&P for escalation to the risk committee, which should have ultimate decision-making authority. Examples of model risk issues that could be escalated to the committee are:
- Is the CU using overlays too often for a particular model?
- Is it time to recalibrate or rebuild a model?
- Does the CU have the appropriate number of resources dedicated to model risk oversight?
Conclusion
It is important that CUs start building an effective model risk management program as it takes time to develop a strong risk culture with everyone at the institution understanding and fulfilling their roles appropriately. The framework discussed above aims to help in that endeavour. See you in Week 2!
If you have any questions or would like to get in touch, feel free to reach out – we're here to help!
Comments