ValidMind for validation 3 — Developing a potential challenger

Learn how to use ValidMind for your end-to-end validation process with our series of four introductory notebooks. In this third notebook, develop a potential challenger and then pass your challenger and its predictions to ValidMind.

A challenger is an alternate record (model) that attempts to outperform the champion, ensuring that the best performing fit-for-purpose record is always considered for deployment. Challengers also help avoid over-reliance on a single record, and allow testing of new features, algorithms, or data sources without disrupting the production lifecycle.

Learn by doing

Our course tailor-made for validators new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Validator Fundamentals

Prerequisites

In order to develop potential challengers with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform and granted yourself access to the model as a validator
• Installed the ValidMind Library in your local environment, allowing you to access all its features
• Learned how to import and initialize datasets for use with ValidMind
• Understood the basics of how to run and log tests with ValidMind
• Run data quality tests on the datasets used to train the champion, and logged the results of those tests to ValidMind
• Inserted your logged test results into your validation report

Need help with the above steps?

Refer to the first two notebooks in this series:

• 1 — Set up the ValidMind Library for validation
• 2 — Start the validation process

Setting up

This section should be quite familiar to you — as we performed the same actions in the previous notebook, 2 — Start the validation process.

Initialize the ValidMind Library

As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Validation from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="validation-report",
)

Note: you may need to restart the kernel to use updated packages.

2026-07-31 16:47:37,103 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model validation (ID: cmalguc9y02ok199q2db381ib)
📁 Document Type: validation_report

Import the sample dataset

Next, we'll load in the sample Bank Customer Churn Prediction dataset used to develop the champion that we will independently preprocess:

# Load the sample dataset
from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

Preprocess the dataset

We’ll apply a simple rebalancing technique to the dataset before continuing:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

Let’s also quickly remove highly correlated features from the dataset using the output from a ValidMind test.

As you know, before we can run tests you’ll need to initialize a ValidMind dataset object with the init_dataset function:

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

With our balanced dataset initialized, we can then run our test and utilize the output to help us identify the features we want to remove:

# Run HighPearsonCorrelation test with our balanced dataset as input and return a result object
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among dataset features to identify potentially redundant or highly collinear variable pairs. The result table reports the top correlations ranked by coefficient magnitude together with a Pass/Fail assessment based on the configured absolute correlation threshold of 0.3. In this run, the listed coefficients range from -0.1960 to 0.3449, and only one feature pair exceeds the threshold. The remaining reported relationships are all below the threshold and are marked as passing.

Key insights:

• One pair exceeds threshold: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3449, which is the only reported value above the 0.3 threshold and is therefore marked as Fail.
• All other reported pairs pass: The remaining nine reported feature pairs have absolute correlation values below 0.2, including (IsActiveMember, Exited) at -0.1960 and (Balance, NumOfProducts) at -0.1957, and all are marked as Pass.
• Correlation magnitudes are generally low: Aside from the (Age, Exited) pair, the reported relationships are weak in magnitude, with the rest of the coefficients clustered between approximately -0.196 and 0.0366.
• Both positive and negative relationships appear: The reported top correlations include both positive and negative coefficients, with the strongest positive relationship at 0.3449 for (Age, Exited) and the strongest negative relationship at -0.1960 for (IsActiveMember, Exited).

The test output shows a limited concentration of stronger linear relationships among the reported feature pairs. Only (Age, Exited) breaches the configured threshold, while all other listed correlations remain below 0.3 and are classified as passing. Overall, the reported correlation structure is characterized by one moderate positive relationship and otherwise low-magnitude pairwise linear associations.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3449	Fail
(IsActiveMember, Exited)	-0.1960	Pass
(Balance, NumOfProducts)	-0.1957	Pass
(Balance, Exited)	0.1490	Pass
(NumOfProducts, Exited)	-0.0641	Pass
(NumOfProducts, IsActiveMember)	0.0482	Pass
(Age, Balance)	0.0409	Pass
(Tenure, IsActiveMember)	-0.0405	Pass
(CreditScore, Exited)	-0.0396	Pass
(Tenure, EstimatedSalary)	0.0366	Pass

# From result object, extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3449	Fail
1	(IsActiveMember, Exited)	-0.1960	Pass
2	(Balance, NumOfProducts)	-0.1957	Pass
3	(Balance, Exited)	0.1490	Pass
4	(NumOfProducts, Exited)	-0.0641	Pass
5	(NumOfProducts, IsActiveMember)	0.0482	Pass
6	(Age, Balance)	0.0409	Pass
7	(Tenure, IsActiveMember)	-0.0405	Pass
8	(CreditScore, Exited)	-0.0396	Pass
9	(Tenure, EstimatedSalary)	0.0366	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

# Extract feature names from the list of strings
high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

We can then re-initialize the dataset with a different input_id and the highly correlated features removed and re-run the test for confirmation:

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

# Re-run the test with the reduced feature set
corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or multicollinearity. The results table reports the top feature pairs by absolute Pearson correlation coefficient, along with Pass/Fail status against the configured threshold of 0.3. In this run, the reported coefficients range from -0.1960 to 0.1490, and all listed pairs are marked Pass. The strongest observed relationships are shown for (IsActiveMember, Exited) and (Balance, NumOfProducts), both with absolute correlation values below the threshold.

Key insights:

• No threshold breaches observed: All 10 reported feature pairs pass the test under the configured maximum threshold of 0.3, with no absolute correlation coefficient exceeding the limit.
• Largest correlations remain modest: The highest absolute coefficients are -0.1960 for (IsActiveMember, Exited) and -0.1957 for (Balance, NumOfProducts), which remain materially below the threshold.
• Reported relationships are weak: All other listed coefficients are smaller in magnitude, including 0.1490 for (Balance, Exited) and -0.0641 for (NumOfProducts, Exited), indicating limited linear association among the top reported pairs.
• Both positive and negative associations appear: The table includes negative correlations such as (Tenure, IsActiveMember) at -0.0405 and positive correlations such as (Tenure, EstimatedSalary) at 0.0366, with all relationships remaining small in magnitude.

The reported correlation structure shows no high linear dependence among the top feature pairs under the 0.3 threshold. The largest observed associations are modest and all reported pairs pass the test, indicating that the table does not identify strong pairwise linear redundancy within the listed results. Overall, the observed Pearson correlations are limited in magnitude across both positive and negative relationships.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1960	Pass
(Balance, NumOfProducts)	-0.1957	Pass
(Balance, Exited)	0.1490	Pass
(NumOfProducts, Exited)	-0.0641	Pass
(NumOfProducts, IsActiveMember)	0.0482	Pass
(Tenure, IsActiveMember)	-0.0405	Pass
(CreditScore, Exited)	-0.0396	Pass
(Tenure, EstimatedSalary)	0.0366	Pass
(Balance, HasCrCard)	-0.0303	Pass
(CreditScore, IsActiveMember)	0.0267	Pass

Split the preprocessed dataset

With our raw dataset rebalanced with highly correlated features removed, let's now spilt our dataset into train and test in preparation for model evaluation testing:

# Encode categorical features in the dataset
balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
1361	642	8	117494.27	1	0	0	61977.82	0	False	False	True
1597	665	8	0.00	2	1	0	132152.32	0	False	True	True
7914	646	6	121681.91	2	0	1	61793.47	0	True	False	True
3554	850	1	96947.58	3	1	0	62282.99	1	True	False	False
1899	850	4	147972.19	1	1	0	60708.72	1	True	False	True

from sklearn.model_selection import train_test_split

# Split the dataset into train and test
train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

# Initialize the split datasets
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Import the champion model

With our raw dataset assessed and preprocessed, let's go ahead and import the champion submitted by the development team in the format of a .pkl file: lr_model_champion.pkl

# Import the champion model
import pickle as pkl

with open("lr_model_champion.pkl", "rb") as f:
    log_reg = pkl.load(f)

/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/base.py:525: InconsistentVersionWarning: Trying to unpickle estimator LogisticRegression from version 1.3.2 when using version 1.9.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
  warnings.warn(

Training a potential challenger model

We're curious how an alternate model compares to our champion, so let's train a challenger as a basis for our testing.

Our champion logistic regression model is a simpler, parametric model that assumes a linear relationship between the independent variables and the log-odds of the outcome. While logistic regression may not capture complex patterns as effectively, it offers a high degree of interpretability and is easier to explain to stakeholders. However, risk is not calculated in isolation from a single factor, but rather in consideration with trade-offs in predictive performance, ease of interpretability, and overall alignment with business objectives.

Random forest classification model

A random forest classification model is an ensemble machine learning algorithm that uses multiple decision trees to classify data. In ensemble learning, multiple models are combined to improve prediction accuracy and robustness.

Random forest classification models generally have higher accuracy because they capture complex, non-linear relationships, but as a result they lack transparency in their predictions.

# Import the Random Forest Classification model
from sklearn.ensemble import RandomForestClassifier

# Create the model instance with 50 decision trees
rf_model = RandomForestClassifier(
    n_estimators=50,
    random_state=42,
)

# Train the model
rf_model.fit(X_train, y_train)

RandomForestClassifier(n_estimators=50, random_state=42)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

Initialize the ValidMind models

In addition to the initialized datasets, you'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our two models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model objects with vm.init_model():

# Initialize the champion logistic regression model
vm_log_model = vm.init_model(
    log_reg,
    input_id="log_model_champion",
)

# Initialize the challenger random forest classification model
vm_rf_model = vm.init_model(
    rf_model,
    input_id="rf_model",
)

Assign predictions

With our models registered, we'll move on to assigning both the predictive probabilities coming directly from each model's predictions, and the binary prediction after applying the cutoff threshold described in the Compute binary predictions step above.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

# Champion — Logistic regression model
vm_train_ds.assign_predictions(model=vm_log_model)
vm_test_ds.assign_predictions(model=vm_log_model)

# Challenger — Random forest classification model
vm_train_ds.assign_predictions(model=vm_rf_model)
vm_test_ds.assign_predictions(model=vm_rf_model)

2026-07-31 16:47:49,659 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-31 16:47:49,661 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-31 16:47:49,662 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-31 16:47:49,664 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-07-31 16:47:49,666 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-31 16:47:49,668 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-31 16:47:49,668 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-31 16:47:49,669 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-07-31 16:47:49,671 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-31 16:47:49,695 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-31 16:47:49,697 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-31 16:47:49,719 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-07-31 16:47:49,723 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-07-31 16:47:49,737 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-07-31 16:47:49,738 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-07-31 16:47:49,752 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Running model evaluation tests

With our setup complete, let's run the rest of our validation tests. Since we have already verified the data quality of the dataset used to train our champion, we will now focus on comprehensive performance evaluations of both the champion and challenger models.

Run model performance tests

Let's run some performance tests, beginning with independent testing of our champion logistic regression model, then moving on to our potential challenger model.

Use vm.tests.list_tests() to identify all the model performance tests for classification:

vm.tests.list_tests(tags=["model_performance"], task="classification")

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.model_validation.sklearn.CalibrationCurve	Calibration Curve	Evaluates the calibration of probability estimates by comparing predicted probabilities against observed...	True	False	['model', 'dataset']	{'n_bins': {'type': 'int', 'default': 10}}	['sklearn', 'model_performance', 'classification']	['classification']
validmind.model_validation.sklearn.ClassifierPerformance	Classifier Performance	Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy,...	False	True	['dataset', 'model']	{'average': {'type': 'str', 'default': 'macro'}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.ConfusionMatrix	Confusion Matrix	Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix...	True	False	['dataset', 'model']	{'threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.HyperParametersTuning	Hyper Parameters Tuning	Performs exhaustive grid search over specified parameter ranges to find optimal model configurations...	False	True	['model', 'dataset']	{'param_grid': {'type': 'dict', 'default': None}, 'scoring': {'type': 'Union', 'default': None}, 'thresholds': {'type': 'Union', 'default': None}, 'fit_params': {'type': 'dict', 'default': None}}	['sklearn', 'model_performance']	['clustering', 'classification']
validmind.model_validation.sklearn.MinimumAccuracy	Minimum Accuracy	Checks if the model's prediction accuracy meets or surpasses a specified threshold....	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.7}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.MinimumF1Score	Minimum F1 Score	Assesses if the model's F1 score on the validation set meets a predefined minimum threshold, ensuring balanced...	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.MinimumROCAUCScore	Minimum ROCAUC Score	Validates model by checking if the ROC AUC score meets or surpasses a specified threshold....	False	True	['dataset', 'model']	{'min_threshold': {'type': 'float', 'default': 0.5}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.ModelsPerformanceComparison	Models Performance Comparison	Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,...	False	True	['dataset', 'models']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'model_comparison']	['classification', 'text_classification']
validmind.model_validation.sklearn.PopulationStabilityIndex	Population Stability Index	Assesses the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across...	True	True	['datasets', 'model']	{'num_bins': {'type': 'int', 'default': 10}, 'mode': {'type': 'str', 'default': 'fixed'}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.model_validation.sklearn.PrecisionRecallCurve	Precision Recall Curve	Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve....	True	False	['model', 'dataset']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.ROCCurve	ROC Curve	Evaluates classification model performance by generating and plotting the Receiver Operating Characteristic...	True	False	['model', 'dataset']	{}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.sklearn.RegressionErrors	Regression Errors	Assesses the performance and error distribution of a regression model using various error metrics....	False	True	['model', 'dataset']	{}	['sklearn', 'model_performance']	['regression', 'classification']
validmind.model_validation.sklearn.TrainingTestDegradation	Training Test Degradation	Tests if model performance degradation between training and test datasets exceeds a predefined threshold....	False	True	['datasets', 'model']	{'max_threshold': {'type': 'float', 'default': 0.1}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.model_validation.statsmodels.GINITable	GINI Table	Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets....	False	True	['dataset', 'model']	{}	['model_performance']	['classification']
validmind.ongoing_monitoring.CalibrationCurveDrift	Calibration Curve Drift	Evaluates changes in probability calibration between reference and monitoring datasets....	True	True	['datasets', 'model']	{'n_bins': {'type': 'int', 'default': 10}, 'drift_pct_threshold': {'type': 'float', 'default': 20}}	['sklearn', 'binary_classification', 'model_performance', 'visualization']	['classification', 'text_classification']
validmind.ongoing_monitoring.ClassDiscriminationDrift	Class Discrimination Drift	Compares classification discrimination metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ClassificationAccuracyDrift	Classification Accuracy Drift	Compares classification accuracy metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ConfusionMatrixDrift	Confusion Matrix Drift	Compares confusion matrix metrics between reference and monitoring datasets....	False	True	['datasets', 'model']	{'drift_pct_threshold': {'type': '_empty', 'default': 20}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_performance']	['classification', 'text_classification']
validmind.ongoing_monitoring.ROCCurveDrift	ROC Curve Drift	Compares ROC curves between reference and monitoring datasets....	True	False	['datasets', 'model']	{}	['sklearn', 'binary_classification', 'model_performance', 'visualization']	['classification', 'text_classification']

We'll isolate the specific tests we want to run in mpt:

• model_validation.sklearn.ClassifierPerformance
• model_validation.sklearn.ConfusionMatrix
• model_validation.sklearn.MinimumAccuracy
• model_validation.sklearn.MinimumF1Score
• model_validation.sklearn.ROCCurve

As we learned in the previous notebook 2 — Start the model validation process, you can use a custom result_id to tag the individual result with a unique identifier by appending this result_id to the test_id with a : separator. We'll append an identifier for our champion model here:

mpt = [
    "validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion",
    "validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion",
    "validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion",
    "validmind.model_validation.sklearn.MinimumF1Score:logreg_champion",
    "validmind.model_validation.sklearn.ROCCurve:logreg_champion"
]

Evaluate performance of the champion model

Now, let's run and log our batch of model performance tests using our testing dataset (vm_test_ds) for our champion model:

• The test set serves as a proxy for real-world data, providing an unbiased estimate of model performance since it was not used during training or tuning.
• The test set also acts as protection against selection bias and model tweaking, giving a final, more unbiased checkpoint.

for test in mpt:
    vm.tests.run_test(
        test,
        inputs={
            "dataset": vm_test_ds, "model" : vm_log_model,
        },
    ).log()

Classifier Performance Logreg Champion

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The results are presented by class for labels 0 and 1, along with weighted and macro averages across classes. Class-level precision ranges from 0.5744 to 0.6495, recall ranges from 0.5855 to 0.6391, and F1 ranges from 0.6050 to 0.6159. Overall accuracy is 0.6105 and ROC AUC is 0.6689.

Key insights:

• Class performance is closely balanced: F1-scores are similar across the two classes, at 0.6050 for class 0 and 0.6159 for class 1. Macro-average F1 of 0.6104 and weighted-average F1 of 0.6108 are also nearly identical.
• Precision-recall tradeoff differs by class: Class 0 shows higher recall than precision (0.6391 vs. 0.5744), while class 1 shows higher precision than recall (0.6495 vs. 0.5855). This indicates different error balance across the two class labels.
• Aggregate metrics are internally consistent: Weighted-average precision, recall, and F1 are 0.6145, 0.6105, and 0.6108 respectively, while macro averages are 0.6120, 0.6123, and 0.6104. The closeness of weighted and macro averages indicates limited divergence between class-level results in the reported metrics.
• ROC AUC exceeds accuracy: The reported ROC AUC is 0.6689 compared with accuracy of 0.6105. This shows stronger ranking performance than the single-threshold classification result reflected in accuracy.

The reported metrics show moderate and relatively even performance across both classes, with only small differences in F1-score between class 0 and class 1. The main variation appears in the precision-recall balance, where class 0 favors recall and class 1 favors precision. Aggregate averages remain closely aligned with one another, and the ROC AUC of 0.6689 is higher than the observed accuracy of 0.6105.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.5744	0.6391	0.6050
1	0.6495	0.5855	0.6159
Weighted Average	0.6145	0.6105	0.6108
Macro Average	0.6120	0.6123	0.6104

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6105
ROC AUC	0.6689

2026-07-31 16:47:59,875 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ClassifierPerformance:logreg_champion does not exist in model's document

Confusion Matrix Logreg Champion

The Confusion Matrix test evaluates the classification model’s predictive performance by comparing predicted class labels with observed class labels and summarizing outcomes as true positives, true negatives, false positives, and false negatives. In this result, the matrix shows 202 true positives, 193 true negatives, 109 false positives, and 143 false negatives. The diagonal cells represent correct classifications, while the off-diagonal cells represent classification errors across the two classes.

Key insights:

• Correct classifications exceed errors: The model records 202 true positives and 193 true negatives, for a total of 395 correct classifications, compared with 252 misclassifications from 109 false positives and 143 false negatives.
• False negatives are more frequent: False negatives total 143, exceeding the 109 false positives by 34 cases. This indicates more missed positive cases than incorrect positive assignments.
• Positive class detection is stronger than negative: Among actual positive cases, the model identifies 202 as true positives versus 143 as false negatives. Among actual negative cases, 193 are correctly classified as true negatives versus 109 classified as false positives.
• Error distribution is not symmetric: The two error types are both material, but they are not evenly distributed. The larger false negative count shows that classification errors are more concentrated in missed positive outcomes than in false alarms.

The confusion matrix shows that the model produces more correct than incorrect classifications, with both true positive and true negative counts exceeding their corresponding error counts. At the same time, misclassification remains meaningful in both directions, with false negatives occurring more often than false positives. Overall, the result reflects stronger capture of positive cases than missed positives, while also showing a non-trivial volume of classification error across both classes.

Figures

2026-07-31 16:48:09,530 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ConfusionMatrix:logreg_champion does not exist in model's document

❌ Minimum Accuracy Logreg Champion

The Minimum Accuracy test evaluates whether the model’s prediction accuracy on the assessed dataset meets or exceeds a predefined threshold. The result table reports an accuracy score of 0.6105 against a threshold of 0.7000, along with the corresponding pass/fail outcome. These results present the observed score, the comparison benchmark used in the test, and the resulting status.

Key insights:

• Accuracy below threshold: The observed accuracy score is 0.6105, which is lower than the configured minimum threshold of 0.7000.
• Test outcome is fail: Because the reported score does not meet the threshold, the test result is recorded as Fail.
• Gap to minimum standard: The difference between the observed score and the threshold is 0.0895, indicating the extent to which the measured accuracy falls short of the test benchmark.

The test result shows that the model did not satisfy the minimum accuracy criterion under this evaluation. The measured accuracy of 0.6105 remained below the 0.7000 threshold, and the test therefore returned a failing outcome.

Tables

Score	Threshold	Pass/Fail
0.6105	0.7	Fail

2026-07-31 16:48:15,859 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumAccuracy:logreg_champion does not exist in model's document

✅ Minimum F1 Score Logreg Champion

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6159 alongside a threshold of 0.5 and a pass/fail outcome. These values provide the basis for assessing whether the model’s observed balance of precision and recall satisfies the configured minimum standard.

Key insights:

• F1 score exceeds threshold: The validation F1 score is 0.6159 compared with a minimum threshold of 0.5, placing the observed result 0.1159 above the configured cutoff.
• Test outcome is pass: The reported pass/fail status is "Pass," consistent with the observed F1 score being greater than the threshold.
• Configured margin is positive: The difference between the measured score and threshold is positive, indicating that the validation result cleared the minimum criterion rather than matching it exactly.

The test result shows that the model achieved an F1 score of 0.6159 on the validation set against a minimum threshold of 0.5, and the recorded outcome is a pass. Collectively, the result and score-threshold comparison indicate that the model met the predefined minimum standard for this F1-based validation check.

Tables

Score	Threshold	Pass/Fail
0.6159	0.5	Pass

2026-07-31 16:48:19,795 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumF1Score:logreg_champion does not exist in model's document

ROC Curve Logreg Champion

The ROCCurve test evaluates classification performance by plotting the receiver operating characteristic curve and calculating the area under the curve (AUC). For logreg_champion on test_dataset_final, the figure shows a single ROC curve for a binary classification setting alongside the random-classifier reference line. The plotted curve rises above the diagonal benchmark across most of the false positive rate range, and the reported AUC is 0.67.

Key insights:

• AUC exceeds random baseline: The model’s AUC is 0.67, compared with the random reference value of 0.50 shown in the plot. This indicates observable separation between the two classes in the test dataset.
• ROC curve remains above diagonal: The ROC curve is positioned above the random-classifier line for most of the threshold range. This reflects higher true positive rates than the random benchmark at comparable false positive rates.
• Discrimination is moderate: The reported AUC of 0.67 indicates performance above chance, while remaining materially below a near-perfect discrimination pattern that would approach the upper-left corner more closely.

The ROC result shows that logreg_champion demonstrates positive discriminative ability on test_dataset_final, with an AUC of 0.67 and a curve that stays above the random baseline through most thresholds. The observed shape indicates that class ranking is better than chance across the evaluated threshold range. At the same time, the distance from the upper-left boundary indicates that discrimination is moderate rather than strong.

Figures

2026-07-31 16:48:28,783 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ROCCurve:logreg_champion does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for some test IDs.

That's expected, as when we run validations tests the results logged need to be manually added to your report as part of your compliance assessment process within the ValidMind Platform.

Log an artifact

As we can observe from the output above, our champion doesn't pass the MinimumAccuracy based on the default thresholds of the out-of-the-box test, so let's log an artifact (finding) in the ValidMind Platform (Learn more: Add and manage artifacts):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Validation under Documents.

3. Click on 2.2.2. Model Performance to expand that section.

4. Under the Model Performance Metrics guideline, click to expand the Artifacts panel.

5. Click Link Artifact and select Validation Issue as the type of artifact.

6. Click + Add Validation Issue and enter in the details for your validation issue, for example:

   • Title — Champion Logistic Regression Model Fails Minimum Accuracy Threshold
   • Risk Area — Model Performance
   • Documentation Section — 3.2. Model Evaluation
   • Description — The logistic regression champion model was subjected to a Minimum Accuracy test to determine whether its predictive accuracy meets the predefined performance threshold of 0.7. The model achieved an accuracy score of 0.6136, which falls below the required minimum. As a result, the test produced a Fail outcome.

7. Click Add Validation Issue to submit the validation issue.

8. Select the validation issue you just added to link to your validation report.

9. Click Update Linked Artifacts to insert your validation issue.

10. Confirm that the validation issue you inserted has been correctly inserted into section 2.2.2. Model Performance of the report.

11. Click on the validation issue to expand the issue, where you can adjust details such as severity, owner, due date, status, etc. as well as include proposed remediation plans or supporting documentation as attachments.

Evaluate performance of challenger model

We've now conducted similar tests as the development team for our champion, with the aim of verifying their test results.

Next, let's see how our challengers compare. We'll use the same batch of tests here as we did in mpt, but append a different result_id to indicate that these results should be associated with our challenger:

mpt_chall = [
    "validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger",
    "validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger",
    "validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger",
    "validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger",
    "validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger"
]

We'll run each test once for each model with the same vm_test_ds dataset to compare them:

for test in mpt_chall:
    vm.tests.run_test(
        test,
        input_grid={
            "dataset": [vm_test_ds], "model" : [vm_log_model,vm_rf_model]
        }
    ).log()

Classifier Performance Champion Vs Challenger

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The results compare log_model_champion and rf_model across class-level metrics for classes 0 and 1, together with macro average, weighted average, accuracy, and ROC AUC. Reported values show the relative performance of each model across both classes and at the aggregate level, allowing direct comparison of classification quality and discrimination strength.

Key insights:

• Random forest outperforms champion model: rf_model exceeds log_model_champion on every reported aggregate metric. Accuracy is 0.6909 versus 0.6105, weighted-average F1 is 0.6911 versus 0.6108, and ROC AUC is 0.7786 versus 0.6689.

• Class-level gains are consistent across both classes: For class 0, rf_model records precision/recall/F1 of 0.6518/0.7252/0.6865 compared with 0.5744/0.6391/0.6050 for log_model_champion. For class 1, rf_model records 0.7331/0.6609/0.6951 versus 0.6495/0.5855/0.6159, indicating higher performance for both target classes rather than improvement concentrated in one class.

• Performance is relatively balanced by class within each model: In log_model_champion, F1 scores are 0.6050 for class 0 and 0.6159 for class 1, while macro-average and weighted-average F1 are also close at 0.6104 and 0.6108. In rf_model, class F1 scores are 0.6865 and 0.6951, with macro-average and weighted-average F1 of 0.6908 and 0.6911, showing limited divergence between class-specific and aggregate results.

• Recall remains stronger for class 0 than class 1: Both models show higher recall for class 0 than for class 1. log_model_champion has recall of 0.6391 for class 0 versus 0.5855 for class 1, and rf_model has recall of 0.7252 versus 0.6609, indicating the same directional class pattern across the two models.

The results show a clear separation between the two models, with rf_model delivering higher precision, recall, F1-score, accuracy, and ROC AUC than log_model_champion across all reported summaries. The improvement is broad-based across both classes rather than isolated to a single label, while each model also maintains relatively similar class-level and aggregate F1 values. A consistent class pattern is present in both models, with recall higher for class 0 than for class 1.

Tables

model	Class	Precision	Recall	F1
log_model_champion	0	0.5744	0.6391	0.6050
log_model_champion	1	0.6495	0.5855	0.6159
log_model_champion	Weighted Average	0.6145	0.6105	0.6108
log_model_champion	Macro Average	0.6120	0.6123	0.6104
rf_model	0	0.6518	0.7252	0.6865
rf_model	1	0.7331	0.6609	0.6951
rf_model	Weighted Average	0.6952	0.6909	0.6911
rf_model	Macro Average	0.6925	0.6930	0.6908

model	Metric	Value
log_model_champion	Accuracy	0.6105
log_model_champion	ROC AUC	0.6689
rf_model	Accuracy	0.6909
rf_model	ROC AUC	0.7786

2026-07-31 16:48:37,632 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ClassifierPerformance:champion_vs_challenger does not exist in model's document

Confusion Matrix Champion Vs Challenger

The ConfusionMatrix:champion_vs_challenger test evaluates classification performance by comparing predicted and observed class labels through confusion matrix counts. The results present separate confusion matrices for the champion model (log_model_champion) and challenger model (rf_model), showing counts of true positives, true negatives, false positives, and false negatives. For the champion model, the matrix contains 202 true positives, 193 true negatives, 109 false positives, and 143 false negatives. For the challenger model, the matrix contains 228 true positives, 219 true negatives, 83 false positives, and 117 false negatives.

Key insights:

• Challenger shows more correct classifications: The challenger model records higher true positive and true negative counts than the champion model, with 228 vs. 202 true positives and 219 vs. 193 true negatives.
• Challenger reduces both error types: False negatives decrease from 143 in the champion model to 117 in the challenger model, while false positives decrease from 109 to 83.
• Positive-class capture is stronger: Within the actual positive class, the challenger identifies more positives correctly and misses fewer cases, as reflected by 228 true positives and 117 false negatives versus 202 and 143 for the champion.
• Negative-class separation is stronger: Within the actual negative class, the challenger produces more true negatives and fewer false positives, with counts of 219 true negatives and 83 false positives compared with 193 and 109 for the champion.

Across all four confusion matrix cells, the challenger model exhibits more favorable classification counts than the champion model. The observed differences are consistent in both the positive and negative classes, with higher correct classifications and lower misclassifications in each case. This result indicates a uniformly stronger confusion-matrix profile for the challenger relative to the champion on the evaluated sample.

Figures

2026-07-31 16:48:49,901 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ConfusionMatrix:champion_vs_challenger does not exist in model's document

❌ Minimum Accuracy Champion Vs Challenger

The Minimum Accuracy test evaluates whether each model’s prediction accuracy meets or exceeds a predefined threshold. The results table reports accuracy scores, the common threshold of 0.7, and the corresponding pass/fail outcome for each evaluated model. Two models are shown: log_model_champion with an accuracy score of 0.6105 and rf_model with an accuracy score of 0.6909. Both results are assessed against the same threshold, and both are marked as failing.

Key insights:

• Both models fall below threshold: log_model_champion scored 0.6105 and rf_model scored 0.6909, while the threshold is 0.7 for both evaluations. Each model is therefore recorded as a fail in this test.
• Random forest is closer to passing: rf_model has the higher accuracy score at 0.6909 compared with 0.6105 for log_model_champion. Its score is 0.0091 below the threshold, whereas log_model_champion is 0.0895 below.
• Performance gap between models is material: The difference in accuracy between rf_model and log_model_champion is 0.0804. This indicates stronger observed classification accuracy for rf_model within this test, despite both models remaining below the required cutoff.

The test results show that neither evaluated model achieved the minimum accuracy threshold of 0.7 on the dataset used for this assessment. Among the two, rf_model produced the stronger result and was substantially closer to the threshold than log_model_champion. The overall outcome is a failed minimum-accuracy assessment for both models under the same evaluation criterion.

Tables

model	Score	Threshold	Pass/Fail
log_model_champion	0.6105	0.7	Fail
rf_model	0.6909	0.7	Fail

2026-07-31 16:48:59,411 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumAccuracy:champion_vs_challenger does not exist in model's document

✅ Minimum F1 Score Champion Vs Challenger

The MinimumF1Score test evaluates whether each model’s F1 score on the validation dataset meets the predefined minimum threshold. The results table reports the validation F1 score, the threshold value, and the pass/fail outcome for two models: log_model_champion and rf_model. Both models are assessed against the same threshold of 0.5, with observed F1 scores of 0.6159 and 0.6951, respectively.

Key insights:

• Both models pass threshold: log_model_champion records an F1 score of 0.6159 and rf_model records 0.6951, with both exceeding the minimum threshold of 0.5 and receiving a pass result.
• Random forest has higher F1: rf_model achieves the higher validation F1 score at 0.6951 versus 0.6159 for log_model_champion, a difference of 0.0792.
• Positive margin over minimum requirement: The observed margins above the threshold are 0.1159 for log_model_champion and 0.1951 for rf_model, indicating both results are above the defined minimum by measurable amounts.

The validation results show that both evaluated models satisfy the minimum F1 score criterion under the common threshold of 0.5. Among the two, rf_model demonstrates the stronger F1 result and the larger margin above the threshold. The test outcome indicates that neither model falls below the specified minimum performance level on this metric.

Tables

model	Score	Threshold	Pass/Fail
log_model_champion	0.6159	0.5	Pass
rf_model	0.6951	0.5	Pass

2026-07-31 16:49:04,920 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.MinimumF1Score:champion_vs_challenger does not exist in model's document

ROC Curve Champion Vs Challenger

The ROCCurve:champion_vs_challenger test evaluates classification performance by plotting the ROC curve and calculating the AUC for each model on the test dataset. The results show ROC curves for log_model_champion and rf_model, each compared against the random-classification reference line. The reported AUC values are 0.67 for log_model_champion and 0.78 for rf_model, with both curves remaining above the diagonal baseline across most of the false positive rate range.

Key insights:

• rf_model has higher AUC: rf_model records an AUC of 0.78 versus 0.67 for log_model_champion, indicating stronger class separation on the evaluated test dataset.
• Both models exceed random baseline: Both ROC curves lie above the random reference line and both AUC values are greater than 0.5, showing discrimination above chance level for each model.
• rf_model shows stronger early lift: At relatively low false positive rates, the rf_model curve rises more sharply, reaching higher true positive rates earlier than log_model_champion.
• Champion curve is closer to baseline: The log_model_champion ROC curve tracks closer to the diagonal reference line than rf_model, consistent with its lower AUC value.

The ROC results indicate that both evaluated models demonstrate positive discriminatory ability on the test dataset, with rf_model showing stronger performance throughout the curve and a higher aggregate AUC. The separation between the two AUC values, 0.78 versus 0.67, reflects a materially stronger ranking performance for rf_model relative to log_model_champion.

Figures

2026-07-31 16:49:18,263 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.ROCCurve:champion_vs_challenger does not exist in model's document

Based on the performance metrics, our challenger random forest classification model passes the MinimumAccuracy where our champion did not.

In your validation report, support your recommendation in your validation issue's Proposed Remediation Plan to investigate the usage of our challenger by inserting the performance tests we logged with this notebook into the appropriate section.

Run diagnostic tests

Next, we want to inspect the robustness and stability testing comparison between our champion and challenger.

Use list_tests() to list all available diagnosis tests applicable to classification tasks:

vm.tests.list_tests(tags=["model_diagnosis"], task="classification")

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.model_validation.sklearn.OverfitDiagnosis	Overfit Diagnosis	Assesses potential overfitting in a model's predictions, identifying regions where performance between training and...	True	True	['model', 'datasets']	{'metric': {'type': 'str', 'default': None}, 'cut_off_threshold': {'type': 'float', 'default': 0.04}}	['sklearn', 'binary_classification', 'multiclass_classification', 'linear_regression', 'model_diagnosis']	['classification', 'regression']
validmind.model_validation.sklearn.RobustnessDiagnosis	Robustness Diagnosis	Assesses the robustness of a machine learning model by evaluating performance decay under noisy conditions....	True	True	['datasets', 'model']	{'metric': {'type': 'str', 'default': None}, 'scaling_factor_std_dev_list': {'type': 'List', 'default': [0.1, 0.2, 0.3, 0.4, 0.5]}, 'performance_decay_threshold': {'type': 'float', 'default': 0.05}}	['sklearn', 'model_diagnosis', 'visualization']	['classification', 'regression']
validmind.model_validation.sklearn.WeakspotsDiagnosis	Weakspots Diagnosis	Identifies and visualizes weak spots in a machine learning model's performance across various sections of the...	True	True	['datasets', 'model']	{'features_columns': {'type': 'Optional', 'default': None}, 'metrics': {'type': 'Optional', 'default': None}, 'thresholds': {'type': 'Optional', 'default': None}}	['sklearn', 'binary_classification', 'multiclass_classification', 'model_diagnosis', 'visualization']	['classification', 'text_classification']

Let’s now assess the models for potential signs of overfitting and identify any sub-segments where performance may inconsistent with the model_validation.sklearn.OverfitDiagnosis test.

Overfitting occurs when a model learns the training data too well, capturing not only the true pattern but noise and random fluctuations resulting in excellent performance on the training dataset but poor generalization to new, unseen data:

• Since the training dataset (vm_train_ds) was used to fit the model, we use this set to establish a baseline performance for how well the model performs on data it has already seen.
• The testing dataset (vm_test_ds) was never seen during training, and here simulates real-world generalization, or how well the model performs on new, unseen data.

vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger",
    input_grid={
        "datasets": [[vm_train_ds,vm_test_ds]],
        "model" : [vm_log_model,vm_rf_model]
    }
).log()

Overfit Diagnosis Champion Vs Challenger

The Overfit Diagnosis test evaluates differences between training and test performance by feature segment to identify regions where the AUC gap exceeds the 0.04 threshold. Results are reported for both log_model_champion and rf_model across multiple features, with slice-level training AUC, test AUC, and gap values shown in the table and corresponding plots. For log_model_champion, exceedances are concentrated in selected slices of CreditScore, Tenure, Balance, NumOfProducts, and EstimatedSalary, while the binary indicator features shown in the plots remain below the threshold. For rf_model, training AUC is 1.0 in every reported slice, and all reported feature groups contain positive gaps above the threshold.

Key insights:

• Random forest shows universal gap exceedance: For rf_model, every reported slice has training AUC = 1.0 and test AUC below 1.0, producing gaps above the 0.04 threshold across CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, Geography_Germany, Geography_Spain, and Gender_Male.

• Largest random forest gaps are substantial: The maximum reported gaps for rf_model are 0.6667 in CreditScore (400.0, 450.0] and 0.6667 in Balance (190710.048, 214548.804]. Other large gaps include 0.3857 for Tenure (9.0, 10.0], 0.3564 for Balance (71516.268, 95355.024], 0.3387 for NumOfProducts (0.997, 1.3], and 0.3300 for Geography_Germany (0.9, 1.0].

• Champion model gaps are more localized: For log_model_champion, over-threshold gaps are present in a subset of slices rather than throughout all reported features. The largest gaps are 0.2623 for Tenure (9.0, 10.0], 0.2547 for CreditScore (400.0, 450.0], 0.1730 for CreditScore (750.0, 800.0], 0.1699 for CreditScore (800.0, 850.0], and 0.1438 for EstimatedSalary (-188.318, 20001.354].

• CreditScore is a prominent source of gap: CreditScore contains multiple over-threshold slices for both models. In log_model_champion, gaps above threshold occur at (400.0, 450.0] (0.2547), (450.0, 500.0] (0.0443), (600.0, 650.0] (0.0453), (750.0, 800.0] (0.1730), and (800.0, 850.0] (0.1699); in rf_model, all nine reported CreditScore slices exceed the threshold, with gaps ranging from 0.1616 to 0.6667.

• Tenure deterioration is strongest at the highest slice: The slice (9.0, 10.0] has the largest Tenure gap for both models: 0.2623 for log_model_champion and 0.3857 for rf_model. For rf_model, all Tenure slices exceed the threshold, while log_model_champion exceeds the threshold in four reported Tenure slices.

• Binary indicators differ sharply by model: In the plots for log_model_champion, HasCrCard, IsActiveMember, Geography_Germany, and Gender_Male remain below the 0.04 threshold, while Geography_Spain shows one negative slice below the threshold magnitude and one positive slice below threshold. In rf_model, all corresponding binary feature slices exceed the threshold, with gaps ranging from 0.2113 to 0.3300.

• Balance gaps are broad for the random forest: rf_model exceeds the threshold in every reported Balance slice, with gaps from 0.0944 to 0.6667. By contrast, log_model_champion shows only two reported Balance slices above threshold: (95355.024, 119193.78] at 0.0518 and (119193.78, 143032.536] at 0.0997.

• EstimatedSalary is mixed for the champion and consistently elevated for the random forest: For log_model_champion, five reported EstimatedSalary slices exceed the threshold, with gaps between 0.0665 and 0.1438. For rf_model, all ten reported EstimatedSalary slices exceed the threshold, with gaps ranging from 0.1659 to 0.2644.

The results show a clear contrast between the two models at the feature-slice level. rf_model exhibits positive training-to-test AUC gaps above the 0.04 threshold in every reported segment, with training AUC fixed at 1.0 throughout and several large slice-level gaps across CreditScore, Tenure, Balance, NumOfProducts, and binary indicators. log_model_champion also contains threshold exceedances, but they are concentrated in selected regions, most notably in low and high CreditScore bands, the highest Tenure band, selected Balance ranges, and several EstimatedSalary slices, while the plotted binary indicator features remain below the threshold.

Tables

model	Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
log_model_champion	CreditScore	(400.0, 450.0]	48	10	0.6991	0.4444	0.2547
log_model_champion	CreditScore	(450.0, 500.0]	111	31	0.6152	0.5708	0.0443
log_model_champion	CreditScore	(600.0, 650.0]	472	126	0.6852	0.6399	0.0453
log_model_champion	CreditScore	(750.0, 800.0]	245	48	0.7569	0.5839	0.1730
log_model_champion	CreditScore	(800.0, 850.0]	156	51	0.7376	0.5677	0.1699
log_model_champion	Tenure	(1.0, 2.0]	258	61	0.6368	0.5968	0.0400
log_model_champion	Tenure	(2.0, 3.0]	266	74	0.6790	0.6038	0.0752
log_model_champion	Tenure	(5.0, 6.0]	260	57	0.6964	0.6444	0.0520
log_model_champion	Tenure	(8.0, 9.0]	257	64	0.7124	0.6676	0.0448
log_model_champion	Tenure	(9.0, 10.0]	131	41	0.7576	0.4952	0.2623
log_model_champion	Balance	(95355.024, 119193.78]	514	117	0.7366	0.6848	0.0518
log_model_champion	Balance	(119193.78, 143032.536]	516	143	0.7059	0.6061	0.0997
log_model_champion	NumOfProducts	(0.997, 1.3]	1468	378	0.6671	0.6153	0.0518
log_model_champion	EstimatedSalary	(-188.318, 20001.354]	261	69	0.7528	0.6090	0.1438
log_model_champion	EstimatedSalary	(59980.902, 79970.676]	270	67	0.6797	0.6045	0.0752
log_model_champion	EstimatedSalary	(79970.676, 99960.45]	237	71	0.7092	0.6419	0.0672
log_model_champion	EstimatedSalary	(119950.224, 139939.998]	249	66	0.7065	0.6343	0.0722
log_model_champion	EstimatedSalary	(179919.546, 199909.32]	247	54	0.7606	0.6941	0.0665
rf_model	CreditScore	(400.0, 450.0]	48	10	1.0000	0.3333	0.6667
rf_model	CreditScore	(450.0, 500.0]	111	31	1.0000	0.7646	0.2354
rf_model	CreditScore	(500.0, 550.0]	278	63	1.0000	0.7959	0.2041
rf_model	CreditScore	(550.0, 600.0]	367	86	1.0000	0.8089	0.1911
rf_model	CreditScore	(600.0, 650.0]	472	126	1.0000	0.7382	0.2618
rf_model	CreditScore	(650.0, 700.0]	497	131	1.0000	0.8162	0.1838
rf_model	CreditScore	(700.0, 750.0]	398	100	1.0000	0.7809	0.2191
rf_model	CreditScore	(750.0, 800.0]	245	48	1.0000	0.8384	0.1616
rf_model	CreditScore	(800.0, 850.0]	156	51	1.0000	0.6908	0.3092
rf_model	Tenure	(-0.01, 1.0]	389	91	1.0000	0.7779	0.2221
rf_model	Tenure	(1.0, 2.0]	258	61	1.0000	0.7823	0.2177
rf_model	Tenure	(2.0, 3.0]	266	74	1.0000	0.7970	0.2030
rf_model	Tenure	(3.0, 4.0]	277	48	1.0000	0.8151	0.1849
rf_model	Tenure	(4.0, 5.0]	242	73	1.0000	0.8182	0.1818
rf_model	Tenure	(5.0, 6.0]	260	57	1.0000	0.7809	0.2191
rf_model	Tenure	(6.0, 7.0]	263	58	1.0000	0.7537	0.2463
rf_model	Tenure	(7.0, 8.0]	242	80	1.0000	0.7722	0.2278
rf_model	Tenure	(8.0, 9.0]	257	64	1.0000	0.8618	0.1382
rf_model	Tenure	(9.0, 10.0]	131	41	1.0000	0.6143	0.3857
rf_model	Balance	(-238.388, 23838.756]	839	210	1.0000	0.8627	0.1373
rf_model	Balance	(23838.756, 47677.512]	16	9	1.0000	0.7500	0.2500
rf_model	Balance	(47677.512, 71516.268]	72	19	1.0000	0.9056	0.0944
rf_model	Balance	(71516.268, 95355.024]	238	56	1.0000	0.6436	0.3564
rf_model	Balance	(95355.024, 119193.78]	514	117	1.0000	0.7448	0.2552
rf_model	Balance	(119193.78, 143032.536]	516	143	1.0000	0.6782	0.3218
rf_model	Balance	(143032.536, 166871.292]	269	64	1.0000	0.6828	0.3172
rf_model	Balance	(166871.292, 190710.048]	93	24	1.0000	0.6991	0.3009
rf_model	Balance	(190710.048, 214548.804]	25	4	1.0000	0.3333	0.6667
rf_model	NumOfProducts	(0.997, 1.3]	1468	378	1.0000	0.6613	0.3387
rf_model	NumOfProducts	(1.9, 2.2]	931	220	1.0000	0.7264	0.2736
rf_model	NumOfProducts	(2.8, 3.1]	150	41	1.0000	0.8526	0.1474
rf_model	HasCrCard	(-0.001, 0.1]	797	213	1.0000	0.7568	0.2432
rf_model	HasCrCard	(0.9, 1.0]	1788	434	1.0000	0.7887	0.2113
rf_model	IsActiveMember	(-0.001, 0.1]	1377	345	1.0000	0.7684	0.2316
rf_model	IsActiveMember	(0.9, 1.0]	1208	302	1.0000	0.7682	0.2318
rf_model	EstimatedSalary	(-188.318, 20001.354]	261	69	1.0000	0.7559	0.2441
rf_model	EstimatedSalary	(20001.354, 39991.128]	257	62	1.0000	0.7367	0.2633
rf_model	EstimatedSalary	(39991.128, 59980.902]	258	73	1.0000	0.8154	0.1846
rf_model	EstimatedSalary	(59980.902, 79970.676]	270	67	1.0000	0.8107	0.1893
rf_model	EstimatedSalary	(79970.676, 99960.45]	237	71	1.0000	0.7763	0.2237
rf_model	EstimatedSalary	(99960.45, 119950.224]	283	61	1.0000	0.8341	0.1659
rf_model	EstimatedSalary	(119950.224, 139939.998]	249	66	1.0000	0.7356	0.2644
rf_model	EstimatedSalary	(139939.998, 159929.772]	260	51	1.0000	0.8092	0.1908
rf_model	EstimatedSalary	(159929.772, 179919.546]	263	73	1.0000	0.7446	0.2554
rf_model	EstimatedSalary	(179919.546, 199909.32]	247	54	1.0000	0.8032	0.1968
rf_model	Geography_Germany	(-0.001, 0.1]	1791	462	1.0000	0.7878	0.2122
rf_model	Geography_Germany	(0.9, 1.0]	794	185	1.0000	0.6700	0.3300
rf_model	Geography_Spain	(-0.001, 0.1]	1993	487	1.0000	0.7662	0.2338
rf_model	Geography_Spain	(0.9, 1.0]	592	160	1.0000	0.8188	0.1813
rf_model	Gender_Male	(-0.001, 0.1]	1248	330	1.0000	0.7618	0.2382
rf_model	Gender_Male	(0.9, 1.0]	1337	317	1.0000	0.7838	0.2162

Figures

2026-07-31 16:49:41,771 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.OverfitDiagnosis:champion_vs_challenger does not exist in model's document

Let's also conduct robustness and stability testing of the two models with the model_validation.sklearn.RobustnessDiagnosis test.

Robustness refers to a model's ability to maintain consistent performance, and stability refers to a model's ability to produce consistent outputs over time across different data subsets.

Again, we'll use both the training and testing datasets to establish baseline performance and to simulate real-world generalization:

vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression",
    input_grid={
        "datasets": [[vm_train_ds,vm_test_ds]],
        "model" : [vm_log_model,vm_rf_model]
    },
).log()

❌ Robustness Diagnosis Champion Vs Log Regression

The Robustness Diagnosis:Champion_vs_LogRegression test evaluates model resilience by measuring AUC changes after adding Gaussian noise to numeric input features across increasing perturbation sizes. Results are reported for log_model_champion and rf_model on both train_dataset_final and test_dataset_final, from the baseline condition through perturbation sizes of 0.1 to 0.5. The tables and plots show baseline AUC levels, the corresponding performance decay at each noise level, and whether each run passed the defined threshold.

Key insights:

• Logistic model remains stable under noise: log_model_champion passes all train and test evaluations from baseline through perturbation size 0.5. Its train AUC declines from 0.6910 to 0.6754, while test AUC moves from 0.6689 to 0.6480, with reported performance decay remaining between -0.0027 and 0.0309.

• Random forest shows large train-side decay: rf_model train performance decreases steadily from AUC 1.0000 at baseline to 0.8074 at perturbation size 0.5. Reported train performance decay reaches 0.0535 at 0.2, 0.1001 at 0.3, 0.1535 at 0.4, and 0.1926 at 0.5, and these train runs are marked as failed from 0.2 onward.

• Random forest test performance is more stable: On test_dataset_final, rf_model starts at AUC 0.7786 and remains between 0.7294 and 0.7823 across all perturbation levels. Performance decay on test ranges from -0.0036 to 0.0492, and all test runs are marked as passed.

• Test AUC is consistently higher for random forest: Across all perturbation sizes, rf_model test AUC exceeds log_model_champion test AUC. At baseline the values are 0.7786 versus 0.6689, and at perturbation size 0.5 they are 0.7294 versus 0.6480.

• Noise response is not strictly monotonic on test data: Both models show small non-monotonic movements in test AUC under added noise. log_model_champion test AUC increases from 0.6662 at 0.1 to 0.6716 at 0.2 before declining, while rf_model test AUC is slightly above baseline at 0.1 and 0.2 before decreasing at larger perturbation sizes.

The robustness results show two distinct patterns across the compared models. log_model_champion maintains relatively small AUC changes on both training and test data and passes all evaluated perturbation levels, whereas rf_model exhibits substantial degradation on training data with failures beginning at perturbation size 0.2. At the same time, rf_model retains higher test AUC than log_model_champion across all noise levels, with test-side decay remaining limited enough to pass throughout the test range.

Tables

model	Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
log_model_champion	Baseline (0.0)	train_dataset_final	2585	0.6910	0.0000	True
log_model_champion	Baseline (0.0)	test_dataset_final	647	0.6689	0.0000	True
log_model_champion	0.1	train_dataset_final	2585	0.6896	0.0015	True
log_model_champion	0.1	test_dataset_final	647	0.6662	0.0027	True
log_model_champion	0.2	train_dataset_final	2585	0.6876	0.0035	True
log_model_champion	0.2	test_dataset_final	647	0.6716	-0.0027	True
log_model_champion	0.3	train_dataset_final	2585	0.6823	0.0088	True
log_model_champion	0.3	test_dataset_final	647	0.6591	0.0098	True
log_model_champion	0.4	train_dataset_final	2585	0.6819	0.0091	True
log_model_champion	0.4	test_dataset_final	647	0.6379	0.0309	True
log_model_champion	0.5	train_dataset_final	2585	0.6754	0.0156	True
log_model_champion	0.5	test_dataset_final	647	0.6480	0.0209	True
rf_model	Baseline (0.0)	train_dataset_final	2585	1.0000	0.0000	True
rf_model	Baseline (0.0)	test_dataset_final	647	0.7786	0.0000	True
rf_model	0.1	train_dataset_final	2585	0.9842	0.0158	True
rf_model	0.1	test_dataset_final	647	0.7823	-0.0036	True
rf_model	0.2	train_dataset_final	2585	0.9465	0.0535	False
rf_model	0.2	test_dataset_final	647	0.7811	-0.0025	True
rf_model	0.3	train_dataset_final	2585	0.8999	0.1001	False
rf_model	0.3	test_dataset_final	647	0.7626	0.0160	True
rf_model	0.4	train_dataset_final	2585	0.8465	0.1535	False
rf_model	0.4	test_dataset_final	647	0.7392	0.0394	True
rf_model	0.5	train_dataset_final	2585	0.8074	0.1926	False
rf_model	0.5	test_dataset_final	647	0.7294	0.0492	True

Figures

2026-07-31 16:50:01,797 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.RobustnessDiagnosis:Champion_vs_LogRegression does not exist in model's document

Run feature importance tests

We also want to verify the relative influence of different input features on our models' predictions, as well as inspect the differences between our champion and challenger to see if a certain model offers more understandable or logical importance scores for features.

Use list_tests() to identify all the feature importance tests for classification:

# Store the feature importance tests
FI = vm.tests.list_tests(tags=["feature_importance"], task="classification",pretty=False)
FI

['validmind.model_validation.FeaturesAUC',
 'validmind.model_validation.sklearn.PermutationFeatureImportance',
 'validmind.model_validation.sklearn.SHAPGlobalImportance']

We'll only use our testing dataset (vm_test_ds) here, to provide a realistic, unseen sample that mimic future or production data, as the training dataset has already influenced our model during learning:

# Run and log our feature importance tests for both models for the testing dataset
for test in FI:
    vm.tests.run_test(
        "".join((test,':champion_vs_challenger')),
        input_grid={
            "dataset": [vm_test_ds], "model" : [vm_log_model,vm_rf_model]
        },
    ).log()

Features Champion Vs Challenger

The FeaturesAUC test evaluates the discriminatory power of each individual feature in the binary classification task by computing a standalone AUC for each feature. The result is presented as a ranked horizontal bar chart for the test dataset, with feature-level AUC values spanning approximately 0.41 to 0.58. Geography_Germany and Balance appear at the top of the ranking, while IsActiveMember and NumOfProducts appear at the bottom. Most features shown cluster between roughly 0.45 and 0.53.

Key insights:

• Geography_Germany has the highest AUC: Geography_Germany shows the strongest univariate discrimination in the chart, with an AUC close to 0.58, making it the highest-ranked individual feature in this test.
• Balance is a close second: Balance has an AUC just below Geography_Germany, also near 0.58, indicating similarly strong standalone separation relative to the other listed features.
• Lower-ranked features remain near 0.41: IsActiveMember and NumOfProducts are the lowest-ranked features, each with AUC values around 0.41, placing them at the bottom of the observed univariate ranking.
• Most features lie in a narrow middle range: HasCrCard, EstimatedSalary, CreditScore, Geography_Spain, Tenure, and Gender_Male occupy the middle of the ranking, with AUC values concentrated around approximately 0.43 to 0.52.

The feature-level AUC profile shows a clear ranking of univariate discriminatory strength across the evaluated predictors. Two features, Geography_Germany and Balance, lead the distribution at roughly 0.58, while the weakest observed features are near 0.41. The remaining features are grouped within a relatively narrow intermediate band, indicating moderate dispersion in standalone feature performance across the test dataset.

Figures

2026-07-31 16:50:19,663 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.FeaturesAUC:champion_vs_challenger does not exist in model's document

Permutation Feature Importance Champion Vs Challenger

The Permutation Feature Importance test evaluates the relative contribution of each input feature by measuring the change in model performance after permuting feature values. The results are presented separately for the champion logistic model (log_model_champion) and the challenger random forest model (rf_model), with larger positive values indicating greater performance dependence on a feature. The two plots show the ranked importance of the same feature set across both models, including features with near-zero and negative importance values.

Key insights:

• Different primary drivers across models: The champion model is most influenced by IsActiveMember (approximately 0.05), followed by Geography_Germany (approximately 0.034) and Gender_Male (approximately 0.022). The challenger model is most influenced by NumOfProducts (approximately 0.13), followed by Balance (approximately 0.06).

• Challenger model shows stronger concentration: The largest permutation importance in the challenger model (NumOfProducts, approximately 0.13) is substantially higher than the largest value in the champion model (IsActiveMember, approximately 0.05), indicating a more concentrated dependence on its top-ranked feature.

• Champion model places limited weight on several features: In the champion model, Tenure and NumOfProducts have only small positive importance values, while HasCrCard is near zero and Geography_Spain, Balance, EstimatedSalary, and CreditScore are negative.

• Challenger model assigns broader positive importance above zero: In the challenger model, NumOfProducts, Balance, Geography_Germany, and IsActiveMember all show clearly positive importance, with smaller positive contributions from HasCrCard, Tenure, and Gender_Male.

• Negative importances appear in both models: Both models show negative permutation importance for CreditScore, EstimatedSalary, and Geography_Spain. The champion model also shows negative importance for Balance, while the challenger model shows Balance as the second most important feature.

The permutation importance results show materially different feature reliance patterns between the champion and challenger models. The champion logistic model is driven primarily by IsActiveMember and selected demographic indicators, whereas the challenger random forest model depends most heavily on NumOfProducts and Balance. Both models also contain features with near-zero or negative importance, indicating that several inputs contribute little or are associated with improved performance when permuted under this test.

Figures

2026-07-31 16:50:42,495 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.PermutationFeatureImportance:champion_vs_challenger does not exist in model's document

SHAP Global Importance Champion Vs Challenger

The SHAP Global Importance test evaluates global feature importance by measuring the average absolute SHAP contribution of each feature and visualizing both overall ranking and directional effect patterns. The results are shown for two models, log_model_champion and rf_model, using normalized feature importance bar charts and SHAP summary plots. For log_model_champion, the highest-ranked features are IsActiveMember, Geography_Germany, and Gender_Male, followed by Balance, while the remaining variables have materially lower normalized importance. For rf_model, NumOfProducts is the highest-ranked feature, followed by IsActiveMember, Geography_Germany, and Balance, with the summary plots showing how low and high feature values align with negative or positive SHAP contributions.

Key insights:

• Different leading drivers by model: log_model_champion is led by IsActiveMember at approximately 100 normalized importance, with Geography_Germany and Gender_Male next at roughly 80 and 74. In contrast, rf_model is led by NumOfProducts at approximately 100, while IsActiveMember and Geography_Germany are lower at roughly 51 and 47.

• Shared importance in core features: Both models assign relatively high importance to IsActiveMember, Geography_Germany, and Balance. Balance ranks fourth in both models, at roughly 42 for log_model_champion and roughly 43 for rf_model.

• Linear model shows concentrated importance: In log_model_champion, importance declines sharply after the top three features, with CreditScore, Tenure, NumOfProducts, HasCrCard, EstimatedSalary, and Geography_Spain all below roughly 20 normalized importance. This indicates a more concentrated attribution structure than in rf_model.

• Random forest gives stronger role to NumOfProducts: NumOfProducts has modest importance in log_model_champion at roughly 15, but is the dominant feature in rf_model at 100. The rf_model summary plot also shows a wide SHAP spread for this feature, including positive contributions above 0.4 and negative contributions below -0.2.

• Directional effects are visible in both models: In log_model_champion, higher values for Balance align primarily with positive SHAP values, while lower values align more with negative contributions. For CreditScore, higher feature values appear more concentrated on the negative side and lower values more on the positive side; for IsActiveMember, the two value states appear separated between negative and positive SHAP effects.

• Lower-ranked features remain near zero impact: In both models, HasCrCard and Geography_Spain have low normalized importance and SHAP values clustered close to zero relative to the leading features. EstimatedSalary and Tenure also show comparatively narrow contribution ranges.

The SHAP results show that the two models rely on overlapping but differently weighted feature sets. log_model_champion concentrates attribution in IsActiveMember, Geography_Germany, and Gender_Male, whereas rf_model places the largest emphasis on NumOfProducts and distributes importance more broadly across several features. Across both models, Balance, IsActiveMember, and Geography_Germany remain material contributors, while lower-ranked variables exhibit comparatively limited marginal impact in the SHAP summaries.

Figures

2026-07-31 16:50:59,206 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.model_validation.sklearn.SHAPGlobalImportance:champion_vs_challenger does not exist in model's document

In summary

In this third notebook, you learned how to:

• Initialize ValidMind model objects
• Assign predictions and probabilities to your ValidMind model objects
• Use tests from ValidMind to evaluate the potential of models, including comparative tests between champion and challengers
• Log an artifact in the ValidMind Platform

Next steps

Finalize validation and reporting

Now that you're familiar with the basics of using the ValidMind Library to run and log validation tests, let's learn how to implement some custom tests and wrap up our validation: 4 — Finalize validation and reporting

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial