Ongoing monitoring

Monitoring of record (model) performance in risk management involves regularly assessing a record’s accuracy, stability, and robustness to ensure it remains reliable after deployment.

Monitoring is a critical component of risk management, as emphasized in regulations such as SR 26-2, SS1/23, and E-24,¹ and includes:

¹ Regulatory requirements

• Tracking key performance indicators, detecting model drift, recalibrating as needed, and validating assumptions.
• Backtesting to verify predictive power, maintaining transparent reporting and governance, and ensuring ongoing compliance with regulatory standards.
• Continuous monitoring to sustain the record’s integrity and mitigating risks associated with its use as they arise.

Monitoring scenarios

Scenarios where ongoing monitoring is warranted:

• Pre-approval monitoring of new records — New records such as models should undergo a trial phase of monitoring before full approval and subsequent deployment, particularly for high-risk or regulatory records, to ensure reliability before deployment.

  Trial phases where a record is subject to ongoing monitoring are typically fairly short, ranging from a few days to several weeks.

• Monitoring during significant updates — When a record undergoes a significant update, ongoing monitoring should compare the updated record’s performance to the original. This process, called parallel runs, involves running both versions of records such as AI models simultaneously for a set period.

  Record outputs should be closely monitored to assess whether the update improves performance or introduces new risks. The results help determine if the updated record should replace the original or if further adjustments are needed. Parallel runs are especially important for regulatory or critical records, ensuring changes don’t harm performance.

• Post-production monitoring — After deployment into production, records should be regularly tested against performance benchmarks to identify deviations, enabling timely recalibrations or adjustments.

  The record’s output should be assessed regularly against predefined performance benchmarks to ensure it meets the required standards. Any deviations from the expected performance should be quickly identified, allowing for timely intervention.

Histogram of prediction probabilities to indicate model drift

Comparing predicted probabilities between reference and monitoring datasets

Ongoing monitoring plan

A robust ongoing monitoring plan is crucial for maintaining record accuracy and reliability. Developers should start regular monitoring from the outset, ideally during the pre-approval phase of development, refining the plan as new insights are gained. This plan should be included in the initial documentation, with clear instructions for execution and use of results.²

² Working with documentation

As monitoring continues, it’s important to report record status to key stakeholders, such as your risk management committee. Regular updates with summary metrics will keep stakeholder informed of artifacts and emerging risks, highlighting significant trends or issues that may need action.

Your ongoing monitoring plan should define:

• Metrics and records (models) — Specify which records and performance metrics will be monitored.
• Monitoring frequency — Set how often each metric will be monitored, based on the record’s risk and importance.
• Risk thresholds — Establish thresholds that trigger alerts or actions when metrics deviate from expected ranges.
• Notification system — Implement a system to notify stakeholders when significant issues or deviations occur.
• Regular reports— Present monitoring results using visual tools for clear and accessible decision-making.

Key concepts

backtesting

Comparing a record’s predictions against actual outcomes to verify its predictive power and reliability.

compliance and regulatory adherence

Ensuring that the record (model) continues to meet evolving regulatory requirements and standards.

model drift, drift

Changes in data patterns, input distributions, or record (such as a model) behavior that may indicate a degradation in performance over time.

model performance, performance

The measure of a record’s accuracy, stability, and robustness in achieving its intended outcomes, which is regularly evaluated through monitoring after deployment to ensure ongoing reliability.

ongoing monitoring, ongoing monitoring report, ongoing monitoring plan, monitoring plan

A comprehensive and structured periodic report assessing the record’s performance and compliance over time, ensuring it remains valid under changing conditions. Monitoring includes key elements such as data sources, inputs, performance metrics, and periodic evaluations, ensuring transparency and visibility of the record’s performance in the production environment.

recalibrating models, recalibrating

The process of adjusting a record (such as a model) to account for detected drift or changes in the underlying data or environment.

reporting and governance

The documentation of monitoring artifacts and communication to stakeholders to support decision-making and maintain transparency.

Design and implementation

The design of your ongoing monitoring plan overall should be a collaborative effort between the first line of defense, typically business units or record (model) owners, and the second line of defense, namely your risk management team. Together, they should select the performance metrics, determine monitoring frequency, and tailor the ongoing monitoring plan to the record’s specific use case and risk profile.

This entails that the ongoing monitoring plan is primarily designed and implemented by the developers involved in the record’s development and deployment. Their work is then reviewed during validation to ensure the robustness of the ongoing monitoring plan and alignment with risk management goals.

The implementation of the ongoing monitoring plan typically also falls on developers. This effort includes executing the monitoring activities, collecting performance data, and generating reports. The developers also ensure that the monitoring is carried out according to the established schedule and that any anomalies or deviations are promptly identified and addressed.

Testing

Monitoring should incorporate a variety of tests to ensure ongoing accuracy and reliability.

Key test areas to pay attention to:

• Benchmarking — Compare record estimates with alternative estimates to validate their accuracy.
• Sensitivity testing — Reaffirm the record’s robustness and stability under different conditions.
• Analysis of overrides — Evaluate any adjustments made to the record to understand their impact.
• Parallel outcomes analysis — Assess whether new data should be included in the record’s calibration to enhance its performance.

To try out monitoring, check out the code sample for ongoing monitoring of models.³

³ Quickstart for ongoing monitoring of models with ​ValidMind

Manage ongoing monitoring

Code samples

Available tests

What’s next

• Reporting and ongoing monitoring FAQ
• Working with analytics