ValidMind for development 2 — Start the development process

Learn how to use ValidMind for your end-to-end documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a record (model) is being built appropriately.

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the Document drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-09-24 21:45:53,943 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold....	False	True	['dataset']	{'min_percentage_threshold': {'type': 'float', 'default': 1.0}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests on datasets

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results are presented in separate summary tables for eight numerical variables and two categorical variables, reporting counts, central tendency, dispersion, percentile values, and category concentration. All listed variables have a count of 8,000 observations, and the numerical summaries include percentile information through the 95th percentile alongside observed minima and maxima. The categorical summaries report the number of unique values and the most frequent category with its frequency and share.

Key insights:

• Complete coverage across variables: All numerical and categorical variables show a count of 8,000, indicating that the reported summaries were calculated on the full set of observations for each listed field.

• Balance shows strong lower-tail concentration: Balance has a minimum of 0, a 25th percentile of 0, and a median of 97,264, indicating that at least one quarter of observations are at zero while the distribution extends to higher values up to 250,898. The mean of 76,434.10 is below the median, reflecting this concentration at the lower end.

• Credit score and age are broadly distributed: CreditScore spans from 350 to 850 with a mean of 650.16 and median of 652, while Age ranges from 18 to 92 with a mean of 38.95 and median of 37. Both variables show wide observed ranges with central values close to one another.

• Product holdings are concentrated at low values: NumOfProducts has a mean of 1.53, median of 1, 75th percentile of 2, and maximum of 4. This indicates that the distribution is concentrated in the lower product-count categories.

• Binary indicators differ in prevalence: HasCrCard has a mean of 0.7026, with the median and upper quartiles equal to 1, showing a larger share of positive values, whereas IsActiveMember has a mean of 0.5199, indicating a more even split between 0 and 1 outcomes.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the most frequent category at 4,010 observations (50.12%). Gender contains 2 unique values, with Male as the most frequent category at 4,396 observations (54.95%).

The descriptive statistics show complete observation counts across all reported variables and a mix of broad continuous ranges, low-cardinality discrete fields, and moderately concentrated categorical variables. The most distinct distributional feature is Balance, where zero values occupy at least the lower quartile while the remaining distribution extends to substantially higher amounts. Other variables such as CreditScore, Age, and EstimatedSalary exhibit wide ranges with means and medians that are relatively close, while the categorical fields show a modest concentration in the most frequent categories rather than extreme dominance.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a specified minimum percentage threshold. In this result, the target variable Exited is summarized across two classes, with reported class proportions and a pass/fail assessment based on the configured 30% threshold. The table and accompanying bar chart show that class Exited = 0 accounts for 79.80% of rows, while class Exited = 1 accounts for 20.20% of rows.

Key insights:

• Majority class dominates distribution: Exited = 0 represents 79.80% of the dataset, making it the substantially larger class in the observed target distribution.
• Minority class falls below threshold: Exited = 1 accounts for 20.20% of rows, which is below the configured minimum percentage threshold of 30%, resulting in a fail outcome for that class.
• Asymmetric pass/fail result: The test passes for Exited = 0 and fails for Exited = 1, indicating that the threshold criterion is not met uniformly across target classes.

The results show a two-class target distribution with a pronounced concentration in Exited = 0 and a materially smaller share in Exited = 1. Under the configured 30% minimum class threshold, only the majority class satisfies the test criterion, while the minority class does not. Collectively, the observed class proportions and pass/fail outcomes document an imbalanced target distribution under this test configuration.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the share of records in each class against a minimum percentage threshold. For the target variable Exited, the results show two classes, 0 and 1, each representing 50.00% of rows. The applied minimum percentage threshold is 30%, and the result table reports a pass/fail status for each class. The accompanying bar chart visually reflects the equal class proportions.

Key insights:

• Perfectly even class split: Both Exited = 0 and Exited = 1 account for 50.00% of the dataset, indicating no difference in observed class prevalence.
• All classes exceed threshold: Each class is above the 30% minimum percentage threshold used in this test, and both classes are marked as Pass.
• No underrepresented target class: The test output shows no class with a lower observed frequency than the configured threshold.

The observed target distribution is balanced across the two Exited classes, with identical 50.00% representation for each outcome. Under the configured 30% threshold, both classes pass the imbalance check. Collectively, the table and chart show a symmetric binary class distribution with no flagged minority class in this test result.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant variables or concentrated dependence within the dataset. The results table reports the top correlations ranked by absolute Pearson coefficient and classifies each pair against the configured threshold of 0.3. Among the ten reported pairs, coefficients range from -0.1923 to 0.3414, and only one pair exceeds the threshold. The highest reported correlation is between Age and Exited, while the remaining listed relationships are below the test limit and are marked as passing.

Key insights:

• Single threshold breach observed: Age and Exited records a Pearson correlation of 0.3414 and is the only reported pair classified as Fail under the 0.3 threshold.
• Most reported relationships are weak: The remaining nine reported coefficients fall between -0.1923 and 0.1515, indicating relatively small linear relationships among the other listed feature pairs.
• Largest non-failing correlation remains below threshold: IsActiveMember and Exited shows the next largest absolute correlation at -0.1923, which remains well below the configured threshold.
• Reported correlations include both directions: The table contains both positive and negative coefficients, with positive relationships such as Balance and Exited (0.1515) and negative relationships such as Balance and NumOfProducts (-0.1792).

The reported correlation structure is concentrated in a single pair above the configured threshold, with Age and Exited standing out from the remaining top-ranked relationships. All other listed pairs remain below 0.2 in absolute value, indicating limited linear association within the reported top correlations outside that exception. Overall, the test output reflects one flagged relationship and otherwise low-magnitude pairwise correlations among the reported variables.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3414	Fail
(IsActiveMember, Exited)	-0.1923	Pass
(Balance, NumOfProducts)	-0.1792	Pass
(Balance, Exited)	0.1515	Pass
(NumOfProducts, IsActiveMember)	0.0663	Pass
(NumOfProducts, Exited)	-0.0507	Pass
(CreditScore, Exited)	-0.0502	Pass
(Tenure, IsActiveMember)	-0.0447	Pass
(CreditScore, IsActiveMember)	0.0438	Pass
(Age, Tenure)	-0.0366	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3414	Fail
1	(IsActiveMember, Exited)	-0.1923	Pass
2	(Balance, NumOfProducts)	-0.1792	Pass
3	(Balance, Exited)	0.1515	Pass
4	(NumOfProducts, IsActiveMember)	0.0663	Pass
5	(NumOfProducts, Exited)	-0.0507	Pass
6	(CreditScore, Exited)	-0.0502	Pass
7	(Tenure, IsActiveMember)	-0.0447	Pass
8	(CreditScore, IsActiveMember)	0.0438	Pass
9	(Age, Tenure)	-0.0366	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify highly correlated variable pairs that may indicate redundancy or multicollinearity. The results table lists the top 10 strongest correlations observed in the dataset, along with each pair’s Pearson correlation coefficient and pass/fail status under the configured absolute threshold of 0.3. All reported coefficients are relatively small in magnitude, ranging from -0.1923 to 0.1515, and all pairs are marked as Pass.

Key insights:

• No pair exceeds threshold: All 10 reported feature pairs pass the test against the 0.3 threshold, with no absolute correlation coefficient above 0.1923.
• Strongest relationship is modest: The largest absolute correlation is between IsActiveMember and Exited at -0.1923, indicating a weak negative linear relationship relative to the configured threshold.
• Balance shows multiple top associations: Balance appears in four of the 10 listed pairs, with coefficients of -0.1792 against NumOfProducts, 0.1515 against Exited, and -0.0307 against IsActiveMember, indicating that several of the stronger observed relationships involve this feature while remaining below the threshold.
• Top correlations are concentrated near zero: Aside from the two largest Balance-related relationships and the IsActiveMember-Exited pair, the remaining coefficients fall between -0.0507 and 0.0663, indicating limited linear association among the listed feature pairs.

The observed correlation structure shows no highly correlated feature pairs under the configured 0.3 threshold. The strongest reported relationships are weak in magnitude, and all listed pairs pass the test. Collectively, the results indicate that the top observed pairwise linear associations in this dataset are limited and do not show concentrated high-correlation behavior within the reported feature pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1923	Pass
(Balance, NumOfProducts)	-0.1792	Pass
(Balance, Exited)	0.1515	Pass
(NumOfProducts, IsActiveMember)	0.0663	Pass
(NumOfProducts, Exited)	-0.0507	Pass
(CreditScore, Exited)	-0.0502	Pass
(Tenure, IsActiveMember)	-0.0447	Pass
(CreditScore, IsActiveMember)	0.0438	Pass
(Tenure, EstimatedSalary)	0.0329	Pass
(Balance, IsActiveMember)	-0.0307	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency among numerical variables by presenting pairwise Pearson correlation coefficients in a heat map. The matrix includes CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.19 to 0.15 outside the diagonal of 1.0 self-correlations. Most relationships are clustered close to zero, and the chart highlights the relative magnitude and direction of each pairwise association across the dataset.

Key insights:

• Correlations are uniformly weak: All off-diagonal correlation coefficients are small in magnitude, with the observed range spanning from -0.19 to 0.15. No pair approaches the 0.7 absolute-correlation threshold described in the test methodology.

• Strongest inverse relationship is limited: The largest negative correlation is between Exited and IsActiveMember at -0.19. A similarly sized negative relationship appears between Balance and NumOfProducts at -0.18, indicating only modest inverse linear association.

• Largest positive relationship remains modest: The highest positive off-diagonal correlation is between Balance and Exited at 0.15. Other positive relationships, such as IsActiveMember and NumOfProducts at 0.07, are materially smaller.

• Exited shows low linear association with other inputs: Correlations between Exited and the other numerical variables range from -0.19 to 0.15. This indicates that no single numerical variable in the matrix has a strong linear relationship with the target.

The correlation structure is sparse and low in magnitude across the numerical variables included in the test. The most pronounced relationships are the negative association between Exited and IsActiveMember and the negative association between Balance and NumOfProducts, but both remain weak in absolute terms. Overall, the result shows limited linear dependence among the variables represented in the matrix.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the documentation.

To demonstrate how to add test results to your documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

The Dataset Description test evaluates the structure, completeness, and column-level characteristics of the dataset used in model development. The results summarize 10 columns across numeric and categorical types, reporting non-missing counts, distinct-value counts, and distinct-value proportions for each field. All listed variables have 3,232 observed values with no missing entries, and the output highlights the relative cardinality of each feature, ranging from binary categorical variables to fully unique numeric fields.

Key insights:

• No missing values across columns: All 10 variables report a count of 3,232, missing count of 0, and missing percentage of 0.0%, indicating complete population coverage in the analyzed dataset.
• EstimatedSalary is fully unique: EstimatedSalary has 3,232 distinct values out of 3,232 observations, corresponding to a distinct proportion of 1.0, making it the highest-cardinality field in the dataset.
• Balance shows high numeric cardinality: Balance contains 2,203 distinct values, representing 68.16% of observations, which is substantially higher than the other numeric fields except EstimatedSalary.
• Several variables have very low cardinality: Geography has 3 distinct values, Gender, HasCrCard, IsActiveMember, and Exited each have 2, and NumOfProducts has 4, indicating multiple fields are discrete or binary in form.
• Numeric fields vary materially in granularity: CreditScore has 431 distinct values, Tenure has 11, NumOfProducts has 4, Balance has 2,203, and EstimatedSalary has 3,232, showing a wide spread in the level of numeric detail across inputs.

The dataset description indicates a fully complete input table with no missingness across the 10 reported features. The most notable structural characteristic is the contrast between low-cardinality categorical or discrete fields and high-cardinality numeric variables, particularly EstimatedSalary and Balance. Overall, the dataset combines binary and small-category features with more granular continuous-like numeric inputs, as reflected in the distinct-value counts reported for each column.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	431	0.1334
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2203	0.6816
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the percentage share of each class against a minimum threshold. For the Exited target, the results show two classes, 0 and 1, each representing 50.00% of rows. The table also reports a pass/fail outcome for each class using the configured minimum percentage threshold of 30%, and both classes are marked as Pass. The accompanying bar chart visually confirms the equal class proportions.

Key insights:

• Perfectly balanced target classes: Class 0 and class 1 each account for 50.00% of the dataset, indicating an even split in the target distribution.
• All classes exceed threshold: Both target classes pass the imbalance check against the configured 30% minimum percentage threshold.
• No minority class observed: Neither class falls below the other in representation, and no under-represented target class is present in the reported distribution.

The test results show that the Exited target is evenly distributed across its two classes, with each class contributing half of the observed records. Relative to the configured 30% threshold, both classes satisfy the test criterion and are recorded as passing. Taken together, the tabular and visual results indicate no observed class concentration within the target variable in this dataset.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

The Duplicates test evaluates whether the dataset contains repeated rows that could affect data quality and training behavior. The results table reports both the absolute count and percentage of duplicate rows identified in the dataset. In this run, the table shows 0 duplicate rows, corresponding to 0.0% of all rows evaluated.

Key insights:

• No duplicate rows detected: The test identified 0 duplicate rows in the dataset, indicating that no exact row-level repetitions were found in the evaluated data.
• Duplicate rate is zero: The reported percentage of duplicate rows is 0.0%, showing that duplicate entries do not contribute to the dataset composition in this test run.

The result indicates that the evaluated dataset contains no exact duplicate rows based on the scope of this test. Both the absolute duplicate count and the duplicate row percentage are zero, showing a clean outcome for this specific data quality check.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

The High Cardinality test evaluates the number of unique values in categorical columns to identify features with a large number of distinct categories. The result table reports, for each categorical column assessed, the number of distinct values, the percentage of distinct values, and the corresponding pass/fail outcome against the configured threshold. In this run, two categorical columns were evaluated: Geography with 3 distinct values and Gender with 2 distinct values, and both columns received a Pass result.

Key insights:

• No categorical columns failed: Both assessed categorical features, Geography and Gender, passed the test, indicating that neither exceeded the test threshold for distinct value count.
• Distinct counts are very low: Geography contains 3 distinct values and Gender contains 2 distinct values, reflecting a limited category set across the evaluated categorical fields.
• Distinct-value percentages are minimal: The percentage of distinct values is 0.0928% for Geography and 0.0619% for Gender, showing very small distinct-value proportions in both columns.

The results show that the evaluated categorical columns exhibit low observed cardinality under this test configuration. Both categorical features remained below the test threshold, with small distinct counts and very low distinct-value percentages. No high-cardinality categorical field was identified in the reported output.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

The Missing Values test evaluates dataset completeness by measuring the proportion of missing entries in each feature against the configured threshold. The results are presented at the column level, showing the number of missing values, the percentage of missing values, and the corresponding pass/fail outcome for each variable. In this test run, all listed features are reported with 0 missing values and 0.0% missingness, and each column receives a Pass result.

Key insights:

• No missing values detected: All 10 reported columns show 0 missing values and 0.0% missingness in the test output.
• Uniform pass outcome across features: Every feature in the results table, including both predictors and the target variable Exited, is marked as Pass.
• Completeness is consistent across variables: No column-level variation in missingness is present in the reported dataset, as all variables share the same observed missingness rate of 0.0%.

The test results indicate complete observed data coverage across all reported fields in this dataset snapshot. Missingness is absent for every listed variable, and no feature exceeds the test threshold based on the reported pass/fail outcomes. Overall, the dataset exhibits uniform completeness in the dimensions evaluated by this test.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

The Skewness test evaluates the asymmetry of numerical feature distributions by comparing each column’s skewness value against the defined maximum threshold of 1. The results table reports skewness values and pass/fail outcomes for eight numeric columns in the dataset. Observed skewness values range from -0.9034 to 1.2425, with seven columns marked as Pass and one column marked as Fail. The reported results cover CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited.

Key insights:

• One column exceeds threshold: NumOfProducts is the only feature that fails the test, with skewness of 1.2425, exceeding the maximum threshold of 1.
• Most columns show limited asymmetry: Seven of eight numeric columns pass the test, with skewness values remaining within the threshold range.
• EstimatedSalary is nearly symmetric: EstimatedSalary has skewness of 0.0002, which is the value closest to zero among the evaluated features.
• Largest negative skew remains within limit: HasCrCard records the most negative skewness at -0.9034 and still passes under the threshold.
• Several variables are close to symmetric: CreditScore (-0.0922), Tenure (0.0198), IsActiveMember (0.139), and Exited (0.0) all show skewness values near zero.

The results indicate that skewness is limited across most evaluated numeric features, with only NumOfProducts displaying asymmetry above the configured threshold. The remaining variables fall within the test limit, including features with moderate negative skew such as HasCrCard and Balance. Overall, the distributional asymmetry identified by this test is concentrated in a single numeric column rather than broadly present across the dataset.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.0922	Pass
Tenure	0.0198	Pass
Balance	-0.2645	Pass
NumOfProducts	1.2425	Fail
HasCrCard	-0.9034	Pass
IsActiveMember	0.1390	Pass
EstimatedSalary	0.0002	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

The UniqueRows test evaluates dataset diversity by comparing the proportion of unique values in each column against a prescribed minimum threshold. The results are reported at the column level, showing the number of unique values, the percentage of unique values, and a pass/fail outcome for each field. In this run, three columns passed the test and seven columns failed. Reported uniqueness ranges from 0.0619% to 100.0% across the evaluated columns.

Key insights:

• Uniqueness is concentrated in few columns: EstimatedSalary shows 3,232 unique values and 100.0% uniqueness, Balance shows 2,203 unique values and 68.1621% uniqueness, and CreditScore shows 431 unique values and 13.3354% uniqueness; these are the only columns that passed.
• Most columns have very low uniqueness: Geography, Gender, Tenure, NumOfProducts, HasCrCard, IsActiveMember, and Exited all failed, with uniqueness percentages between 0.0619% and 0.3403%.
• Several variables are binary: Gender, HasCrCard, IsActiveMember, and Exited each contain 2 unique values and 0.0619% uniqueness, and all four columns failed the threshold test.
• Lowest-cardinality non-binary columns also failed: Geography contains 3 unique values at 0.0928%, NumOfProducts contains 4 unique values at 0.1238%, and Tenure contains 11 unique values at 0.3403%, each below the threshold used in this run.

The results show a mixed uniqueness profile across the dataset, with passing outcomes limited to EstimatedSalary, Balance, and CreditScore, while the remaining columns exhibit low cardinality and fail the prescribed threshold. The strongest uniqueness is observed in EstimatedSalary and Balance, with CreditScore also exceeding the threshold at a lower level. Most failed columns are binary or otherwise low-cardinality fields, producing a concentration of failures among discrete variables.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	431	13.3354	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2203	68.1621	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

The TooManyZeroValues test evaluates numerical columns for zero-value frequency above a defined threshold. The result table reports, for each assessed variable, the total row count, the number of zero values, the corresponding zero-value percentage, and the pass/fail outcome. In this test run, all four evaluated numerical variables failed the threshold check, with zero-value percentages ranging from 4.4245% to 53.4653% across 3,232 rows.

Key insights:

• All assessed variables failed: Tenure, Balance, HasCrCard, and IsActiveMember each exceeded the configured zero-value threshold and were marked as Fail in the test output.
• IsActiveMember has the highest zero share: IsActiveMember contains 1,728 zero values out of 3,232 rows, corresponding to 53.4653%, which is the largest zero-value proportion among the evaluated variables.
• Balance and HasCrCard show substantial zero concentration: Balance records 1,030 zero values (31.8688%), and HasCrCard records 951 zero values (29.4245%), indicating that zeros account for roughly one-third of observations in both variables.
• Tenure has the lowest but still elevated zero incidence: Tenure contains 143 zero values, equal to 4.4245% of rows, making it the lowest zero-value percentage reported while still failing the test.

The results show that zero values are present above the test threshold in every numerical variable assessed. The concentration of zeros is most pronounced in IsActiveMember, followed by Balance and HasCrCard, while Tenure has a comparatively smaller but still failing zero-value share. Collectively, the output indicates that zero prevalence is a broad characteristic of the evaluated numerical fields in this dataset segment.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	143	4.4245	Fail
Balance	3232	1030	31.8688	Fail
HasCrCard	3232	951	29.4245	Fail
IsActiveMember	3232	1728	53.4653	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

The IQROutliersTable test identifies observations outside the interquartile range thresholds for numerical features and summarizes the magnitude of those outliers. In this result, outliers were detected for two variables: CreditScore and NumOfProducts. The table reports the total count of outliers for each variable together with the mean of the full variable and summary statistics for the outlier values, including their minimum, quartiles, median, and maximum.

Key insights:

• Outliers are concentrated in two variables: The test detected outliers only for CreditScore and NumOfProducts in the reported results, with 9 outliers for CreditScore and 44 for NumOfProducts.

• NumOfProducts outliers are uniform: All detected outliers for NumOfProducts take the same value. The minimum, 25th percentile, median, 75th percentile, and maximum outlier values are all 4, indicating no variation within the identified outlier set.

• CreditScore outliers are confined to the lower tail: CreditScore outliers range from 350 to 373, with the 25th percentile and median both at 350 and the 75th percentile at 365. Relative to the reported mean of 650.2101, the detected outliers are substantially lower than the overall variable average.

• Outlier counts differ materially across variables: NumOfProducts has nearly five times as many outliers as CreditScore, with 44 versus 9, indicating that the concentration of IQR-defined outliers is materially higher for NumOfProducts in this dataset.

The result shows that IQR-defined outliers are limited to two reported numerical variables, with a higher concentration in NumOfProducts and a smaller set in CreditScore. The NumOfProducts outlier pattern is fully concentrated at a single value, while CreditScore outliers form a narrow low-end range between 350 and 373. Collectively, the output indicates distinct outlier structures across the two variables: one discrete and uniform, the other continuous and confined to the lower tail.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	9	650.2101	350	350.0	350.0	365.0	373
NumOfProducts	44	1.5062	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

The IQROutliersBarPlot test evaluates the distribution of IQR-based outliers across percentile bands for numeric features. The results shown cover CreditScore and NumOfProducts, with bar heights representing the count of detected outliers in each percentile range. For CreditScore, outliers appear in the 50–75 and 75–100 bands, while for NumOfProducts, outliers are concentrated entirely in the 75–100 band. No outliers are displayed in the 0–25 or 25–50 bands for either feature.

Key insights:

• CreditScore outliers are upper-half concentrated: CreditScore shows 6 outliers in the 50–75 percentile band and 3 outliers in the 75–100 band, with no outliers in the lower two percentile ranges.

• NumOfProducts outliers are isolated to the top band: NumOfProducts shows outliers only in the 75–100 percentile range, with a count of 44 and no outliers in the other percentile bands.

• No lower-percentile outliers observed: Across both plotted features, the 0–25 and 25–50 percentile ranges show zero outlier counts.

The results indicate that detected outliers are concentrated in the upper portions of the distributions for both features. CreditScore exhibits a smaller number of outliers spread across the two highest percentile bands, whereas NumOfProducts shows a substantially larger concentration confined to the highest percentile band. The observed pattern is therefore dominated by upper-tail outlier behavior rather than broad outlier presence across the full range of values.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results are presented in separate summary tables for seven numerical variables and two categorical variables, showing counts, central tendency, dispersion, percentile values, and category concentration. All reported variables have a count of 3,232 observations, and the categorical summary reports the number of unique values together with the most frequent category and its share of the sample.

Key insights:

• Complete coverage across reported fields: Each numerical and categorical variable has a reported count of 3,232, indicating consistent observation counts across all summarized fields.

• Balance distribution is widely dispersed: Balance has a mean of 81,753.8424, a median of 102,713, a standard deviation of 61,360.9096, and a minimum of 0. The 25th percentile is also 0, while the 75th percentile reaches 129,294, showing a broad spread and concentration at the lower end of the distribution.

• CreditScore is centered near its median: CreditScore has a mean of 650.2101 and a median of 651.0, with quartiles at 582.0 and 720.0 and values spanning from 350.0 to 850.0. The close alignment of mean and median indicates a centered distribution in the reported sample.

• NumOfProducts is concentrated at low counts: NumOfProducts has a median of 1.0, a 75th percentile of 2.0, a 95th percentile of 3.0, and a maximum of 4.0. The mean of 1.5062 further indicates that most observations are concentrated in the lower product-count range.

• Binary indicators show uneven class proportions: HasCrCard has a mean of 0.7058 with median and upper quartiles equal to 1.0, indicating a majority of observations with value 1. IsActiveMember has a mean of 0.4653 and median of 0.0, indicating a lower share of observations with value 1 relative to HasCrCard.

• Categorical variables show moderate concentration: Geography contains 3 unique values, with France as the most frequent category at 1,504 observations or 46.53% of the sample. Gender contains 2 unique values, with Male as the top category at 1,650 observations or 51.05%.

The descriptive statistics show a fully populated summarized dataset with consistent counts across all reported variables. Numerical variables exhibit differing distribution shapes, with CreditScore and EstimatedSalary centered near their medians, while Balance shows substantial dispersion and a lower-end concentration reflected by zero values at both the minimum and 25th percentile. Discrete and binary variables are concentrated in a small number of levels, and the categorical variables display moderate dominance of their leading categories without a single category comprising an overwhelming majority of the sample.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	650.2101	98.8494	350.0	582.0	651.0	720.0	779.0	817.0	850.0
Tenure	3232.0	4.9861	2.9156	0.0	2.0	5.0	8.0	9.0	9.0	10.0
Balance	3232.0	81753.8424	61360.9096	0.0	0.0	102713.0	129294.0	150681.0	164053.0	250898.0
NumOfProducts	3232.0	1.5062	0.6708	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.7058	0.4558	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4653	0.4989	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	100010.8767	57371.8201	12.0	50688.0	99723.0	149483.0	179343.0	188525.0	199971.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1504.0	46.53
Gender	3232.0	2.0	Male	1650.0	51.05

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency among numerical variables by displaying pairwise Pearson correlation coefficients in a heat map. The result shows the correlation structure across CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.19 to 0.15 outside the diagonal. The diagonal entries are 1.0 by construction, while the off-diagonal cells are concentrated near zero, indicating generally weak linear relationships across the variables included in the test.

Key insights:

• Correlations are uniformly low: All off-diagonal correlation coefficients fall between -0.19 and 0.15, remaining well below the 0.70 threshold referenced in the test methodology.

• Strongest negative relationship is limited: The largest negative coefficient is -0.19 between IsActiveMember and Exited. A similarly sized negative relationship appears between Balance and NumOfProducts at -0.18.

• Strongest positive relationship is modest: The largest positive coefficient is 0.15 between Balance and Exited. Other positive correlations are smaller, including 0.07 between NumOfProducts and IsActiveMember and 0.04 between CreditScore and IsActiveMember.

• Several variables show near-zero association: EstimatedSalary has correlations close to zero with all other variables, including 0.00 with CreditScore and Exited, 0.01 with NumOfProducts and IsActiveMember, and 0.02 to 0.03 with Balance and Tenure.

The observed correlation matrix shows a low-correlation feature set, with no pairwise linear dependency approaching the high-correlation threshold defined for this test. The most pronounced relationships are limited to small negative associations involving IsActiveMember and NumOfProducts, and a modest positive association between Balance and Exited. Overall, the result indicates that the numerical variables in this dataset exhibit limited linear redundancy based on Pearson correlation.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant or highly collinear variable pairs. The results table lists the top feature pairs by absolute Pearson correlation coefficient together with Pass/Fail status based on the configured threshold of 0.3. In this test run, the reported coefficients range from -0.1923 to 0.1515, and all listed feature pairs are marked as Pass. The table therefore captures the strongest observed linear relationships in the dataset while showing that none exceed the specified threshold.

Key insights:

• No correlations exceed threshold: All reported feature pairs pass against the 0.3 threshold. The largest absolute coefficient is 0.1923 for the pair (IsActiveMember, Exited), which remains below the configured limit.

• Strongest observed relationship is weak: The highest absolute correlation in the table is the negative relationship between IsActiveMember and Exited at -0.1923. This is followed by (Balance, NumOfProducts) at -0.1792 and (Balance, Exited) at 0.1515.

• Top relationships are limited in magnitude: All ten reported coefficients are within approximately ±0.20. This indicates that even the strongest linear associations among the listed pairs are relatively modest in size.

• Both positive and negative associations appear: The reported pairs include both negative and positive coefficients, with negative values more prevalent among the top-ranked relationships. The observed values include negative associations such as (Balance, NumOfProducts) and positive associations such as (Balance, Exited).

The test results show that the strongest reported pairwise linear relationships in the dataset remain below the configured Pearson correlation threshold of 0.3. The largest absolute coefficients are concentrated in a narrow range, with the most pronounced associations involving IsActiveMember, Balance, NumOfProducts, and Exited. Overall, the reported top correlations do not indicate high pairwise linear dependence under the test criterion.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1923	Pass
(Balance, NumOfProducts)	-0.1792	Pass
(Balance, Exited)	0.1515	Pass
(NumOfProducts, IsActiveMember)	0.0663	Pass
(NumOfProducts, Exited)	-0.0507	Pass
(CreditScore, Exited)	-0.0502	Pass
(Tenure, IsActiveMember)	-0.0447	Pass
(CreditScore, IsActiveMember)	0.0438	Pass
(Tenure, EstimatedSalary)	0.0329	Pass
(Balance, IsActiveMember)	-0.0307	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships between features to identify potentially redundant variables and concentrated correlation structure in the dataset. The output lists the top correlation pairs ranked by absolute Pearson coefficient and classifies each pair against the configured threshold of 0.3. In this result, 10 feature pairs are shown, with coefficients ranging from -0.1923 to 0.3414. Only one pair exceeds the threshold and is marked as failing, while the remaining pairs are below the threshold and marked as passing.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) has the largest absolute correlation at 0.3414 and is the only relationship exceeding the 0.3 threshold, resulting in a fail classification.
• Remaining relationships are weak: The other nine reported pairs have absolute correlation magnitudes at or below 0.1923, indicating substantially weaker linear relationships than the top-ranked pair.
• Correlations are mixed in direction: The reported coefficients include both positive and negative values, ranging from -0.1923 for (IsActiveMember, Exited) to 0.1515 for (Balance, Exited) among the passing pairs.
• Exited appears in multiple top pairs: Several of the highest-ranked relationships involve Exited, including pairings with Age, IsActiveMember, Balance, NumOfProducts, and CreditScore, with only (Age, Exited) breaching the threshold.

Overall, the reported correlation structure is limited, with one feature pair above the configured threshold and all other listed relationships remaining below it. The strongest observed linear association is between Age and Exited, while the rest of the top-ranked correlations are comparatively small in magnitude. Across the displayed results, the dataset shows a predominantly low pairwise linear dependence pattern aside from this single flagged relationship.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3414	Fail
(IsActiveMember, Exited)	-0.1923	Pass
(Balance, NumOfProducts)	-0.1792	Pass
(Balance, Exited)	0.1515	Pass
(NumOfProducts, IsActiveMember)	0.0663	Pass
(NumOfProducts, Exited)	-0.0507	Pass
(CreditScore, Exited)	-0.0502	Pass
(Tenure, IsActiveMember)	-0.0447	Pass
(CreditScore, IsActiveMember)	0.0438	Pass
(Age, Tenure)	-0.0366	Pass

2026-09-24 21:47:11,178 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Learn more: Work with test results):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Development under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Running model evaluation tests

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
6013	578	France	Male	5	113226.47	1	1	0	56770.76	0
2703	623	Germany	Male	8	96759.42	1	1	1	174777.98	0
3594	669	France	Female	8	0.00	2	1	0	84049.16	0
4410	677	Germany	Male	3	126729.41	1	1	1	26106.39	1
7241	619	Spain	Female	4	175406.13	2	1	1	172792.43	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
6013	578	5	113226.47	1	1	0	56770.76	0	False	False	True
2703	623	8	96759.42	1	1	1	174777.98	0	True	False	True
3594	669	8	0.00	2	1	0	84049.16	0	False	False	False
4410	677	3	126729.41	1	1	1	26106.39	1	True	False	True
7241	619	4	175406.13	2	1	1	172792.43	1	False	True	False

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

Initialize ValidMind datasets

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Initialize a ValidMind model

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-09-24 21:47:12,306 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-24 21:47:12,308 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-24 21:47:12,308 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-24 21:47:12,310 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-09-24 21:47:12,311 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-09-24 21:47:12,312 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-09-24 21:47:12,312 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-09-24 21:47:12,313 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test compares model metadata fields to provide a standardized view of implementation attributes across models. The result is presented as a summary table covering modeling technique, modeling framework, framework version, and programming language. For this test run, the table reports a single model entry with values populated for each of these metadata fields.

Key insights:

• Single model metadata entry captured: The result contains one model record, indicating that the comparison output for this test run is based on a single metadata entry rather than multiple models.
• Metadata fields are fully populated: Values are present for modeling technique, modeling framework, framework version, and programming language, with no missing entries shown in the summary table.
• Implementation stack is clearly identified: The model is documented as using the SKlearnModel technique, the sklearn framework, framework version 1.7.2, and Python as the programming language.

The result provides a complete metadata snapshot for one model implementation. The reported fields consistently identify the model’s technique, framework, framework version, and programming language, and the table shows no missing values within the metadata elements included in this test output.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.7.2	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of records across the model’s available datasets. The result table reports the absolute size and share of the total population for each split. In this run, the documented datasets are train_dataset_final with 2,585 records and test_dataset_final with 647 records, for a total of 3,232 records. The corresponding proportions are 79.98% for training and 20.02% for testing.

Key insights:

• Training data is the dominant split: train_dataset_final contains 2,585 of 3,232 total records, representing 79.98% of the available data and forming the majority of the split structure.
• Test data comprises one-fifth of records: test_dataset_final contains 647 records, accounting for 20.02% of the total dataset.
• Two-way split is documented: The results include training and testing datasets only, with no validation dataset shown in the reported split table.

The reported dataset allocation is concentrated in the training split, with approximately four-fifths of records assigned to training and one-fifth assigned to testing. The full documented sample size is 3,232 records across the two reported datasets. No separate validation split is present in the test output.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of model prediction distributions by comparing the binned distributions of the initial and new datasets. The results are presented across 10 bins, showing counts, population shares, and bin-level PSI contributions for train_dataset_final versus test_dataset_final. The initial dataset contains 2,585 observations and the new dataset contains 647 observations, with total PSI reported as 0.0132. The accompanying plot shows that the population ratios in corresponding bins are closely aligned, with only modest differences in selected bins.

Key insights:

• Overall PSI is low: The total PSI is 0.0132, reflecting a small aggregate difference between the initial and new distributions across all bins.

• Distribution shares are closely aligned: Most bin population shares differ by less than 1 percentage point between datasets. Examples include Bin 4 at 16.8279% versus 16.5379% and Bin 9 at 8.4720% versus 8.1917%.

• Largest shifts are concentrated in a few bins: The highest bin-level PSI contributions occur in Bin 3 (0.0046) and Bin 6 (0.0045). These correspond to share differences of 10.5996% versus 8.5008% in Bin 3 and 10.6770% versus 12.9830% in Bin 6.

• Upper-tail bin remains small in both datasets: Bin 8 contains 1.1219% of the initial sample and 1.5456% of the new sample, with a PSI contribution of 0.0014. This indicates some relative change, but within a low-frequency portion of the distribution.

The PSI results indicate that the prediction distribution is highly similar between train_dataset_final and test_dataset_final at the aggregate level. Observed differences are limited and are primarily concentrated in Bin 3 and Bin 6, while the remaining bins contribute minimally to the overall PSI. Taken together, the table and plot show a stable distributional profile across the two datasets.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	241	9.3230	58	8.9645	0.0001
1	198	7.6596	56	8.6553	0.0012
2	379	14.6615	92	14.2195	0.0001
3	274	10.5996	55	8.5008	0.0046
4	435	16.8279	107	16.5379	0.0001
5	236	9.1296	63	9.7372	0.0004
6	276	10.6770	84	12.9830	0.0045
7	298	11.5280	69	10.6646	0.0007
8	29	1.1219	10	1.5456	0.0014
9	219	8.4720	53	8.1917	0.0001
Total	2585	100.0000	647	100.0000	0.0132

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted class labels with observed class labels and displaying the counts of correct and incorrect classifications. The heatmap shows outcomes for the two classes across the four standard confusion matrix cells. In this result, the model produced 196 true positives, 193 true negatives, 129 false positives, and 129 false negatives. The visualization therefore presents both correctly classified cases and the distribution of errors by error type.

Key insights:

• Correct classifications exceed errors: The model records 196 true positives and 193 true negatives, compared with 129 false positives and 129 false negatives, indicating that correct classifications are more frequent than misclassifications in both classes combined.

• Error types are balanced: False positives and false negatives are identical at 129 each. This indicates that the model’s two primary error types occur at the same observed frequency.

• True outcomes are similarly distributed: True positives and true negatives are close in magnitude, at 196 and 193 respectively. This indicates similar observed counts of correct classification for the positive and negative classes.

• Positive-class captures exceed missed positives: Within the observations with true label 1, the model identifies 196 as positive and misses 129 as negative. This shows more correctly detected positives than false negatives.

The confusion matrix shows that the model classifies both classes with a similar number of correct outcomes, with true positives and true negatives nearly equal. Misclassification is also evenly split between false positives and false negatives, each occurring 129 times. Taken together, the result indicates a relatively balanced classification pattern across classes, with correct predictions occurring more often than errors.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported in-sample results include class-level precision, recall, and F1 metrics for classes 0 and 1, along with macro and weighted averages. Overall summary metrics show accuracy of 0.6453 and ROC AUC of 0.694. The class-level table shows modest differences between the two classes, with class 0 having higher recall and F1 than class 1.

Key insights:

• Overall performance is moderate: Accuracy is 0.6453, weighted-average F1 is 0.6449, and macro-average F1 is 0.6448. These values are closely aligned, indicating similar aggregate performance across the reported summary metrics.
• Class-level performance is relatively balanced: Precision is 0.6367 for class 0 and 0.6551 for class 1, while F1 is 0.6569 for class 0 and 0.6328 for class 1. Macro and weighted averages are nearly identical, which indicates limited divergence between class-specific and aggregate results.
• Recall is stronger for class 0: Recall for class 0 is 0.6785 versus 0.6119 for class 1. This produces the largest observed class-level gap among the reported metrics, at 0.0666.
• ROC AUC exceeds accuracy-based metrics: ROC AUC is 0.694, compared with accuracy of 0.6453 and average F1 values near 0.645. This indicates stronger rank-order discrimination than the threshold-dependent classification metrics.

The in-sample performance results show moderate classification effectiveness across both classes, with aggregate precision, recall, F1, and accuracy clustered around 0.645. Class-level differences are present but limited, with the largest separation appearing in recall, where class 0 performs better than class 1. ROC AUC of 0.694 is higher than the threshold-based summary metrics, indicating that discrimination is somewhat stronger than the final class assignment performance reflected by accuracy and F1.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6367	0.6785	0.6569
1	0.6551	0.6119	0.6328
Weighted Average	0.6459	0.6453	0.6449
Macro Average	0.6459	0.6452	0.6448

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6453
ROC AUC	0.6940

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported out-of-sample results present class-level precision, recall, and F1 values for classes 0 and 1, along with macro and weighted averages. Class 0 records 0.5994 across precision, recall, and F1, while class 1 records 0.6031 for the same metrics. Overall accuracy is 0.6012, and ROC AUC is 0.6498.

Key insights:

• Nearly identical class-level performance: Class 0 and class 1 show very similar results, with precision, recall, and F1 equal to 0.5994 for class 0 and 0.6031 for class 1. This indicates limited variation in performance across the two classes in the out-of-sample evaluation.
• Averages match overall accuracy: Weighted average and macro average precision, recall, and F1 are all 0.6012, which is also equal to the reported accuracy. This reflects a highly uniform aggregate performance profile across the reported classification metrics.
• ROC AUC exceeds classification metrics: ROC AUC is 0.6498, which is higher than the point-estimate classification metrics reported at approximately 0.60. The ranking-based discrimination measure is therefore stronger than the threshold-dependent metrics shown in the classification report.

The out-of-sample results show a consistent performance pattern across both classes and across aggregate summary metrics. Precision, recall, F1, and accuracy are tightly clustered around 0.60, while ROC AUC is higher at 0.6498. Collectively, these results indicate stable but moderate observed classification performance in the evaluated sample.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.5994	0.5994	0.5994
1	0.6031	0.6031	0.6031
Weighted Average	0.6012	0.6012	0.6012
Macro Average	0.6012	0.6012	0.6012

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6012
ROC AUC	0.6498

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for this binary classification model. The result is presented as a precision-recall curve with recall on the x-axis and precision on the y-axis, showing how precision changes as progressively more positive cases are captured. In this plot, precision is highest at very low recall, then declines with fluctuations through the mid-recall range, and continues downward as recall approaches 1.0.

Key insights:

• High precision at low recall: Precision begins near 1.0 at the lowest recall levels and remains above 0.8 only in the earliest portion of the curve, indicating the strongest precision is concentrated at very selective thresholds.

• Mid-range plateau around 0.65-0.70: Across approximately the 0.15 to 0.40 recall range, precision stabilizes in a relatively narrow band around 0.67 to 0.70, indicating a moderate precision region over a meaningful portion of the threshold range.

• Precision declines with broader capture: After the mid-recall region, precision trends downward as recall increases, moving from roughly the mid-0.60s toward about 0.50 by the highest recall levels.

• Some threshold sensitivity at low recall: The left side of the curve shows visible oscillation, with rapid changes in precision at small increases in recall, indicating less smooth threshold behavior in the low-recall region.

The curve shows a clear trade-off between precision and recall across thresholds. Performance is strongest in precision terms at very low recall, transitions into a moderate and relatively stable middle region, and weakens progressively as recall increases further. Overall, the observed pattern indicates that capturing a larger share of positives is associated with a gradual reduction in precision, with the steepest variability concentrated at the lowest recall levels.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates classification performance by plotting the Receiver Operating Characteristic curve and reporting the associated Area Under the Curve (AUC). For log_reg_model_v1 on test_dataset_final, the figure shows a single binary ROC curve and a random-classifier reference line. The plotted ROC curve remains above the diagonal reference across the threshold range, and the legend reports an AUC value of 0.65.

Key insights:

• AUC is 0.65: The reported AUC indicates the model’s discrimination performance as measured over all classification thresholds on the test dataset.
• Curve remains above random baseline: The ROC curve lies above the dashed random-reference line (AUC = 0.5) throughout the plotted range, indicating separation between classes that is better than random ranking.
• Performance is moderate rather than strong: The visual gap between the ROC curve and the random baseline is present but limited, consistent with the reported AUC of 0.65.

The ROC result shows that log_reg_model_v1 demonstrates positive discriminatory ability on test_dataset_final, with performance exceeding the random baseline. At the same time, the reported AUC of 0.65 and the modest elevation of the curve above the diagonal indicate that the degree of class separation is moderate.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

❌ Training Test Degradation

The TrainingTestDegradation test evaluates whether classification performance declines from the training dataset to the test dataset beyond a predefined threshold. The results table reports train and test scores for precision, recall, and F1-score separately for classes 1 and 0, along with the corresponding degradation percentages and pass/fail outcomes. Observed degradation values range from 1.4465% to 11.6633%, with five metrics marked as Pass and one metric marked as Fail.

Key insights:

• Single failure in class 0 recall: Class 0 recall declines from 0.6785 on the training dataset to 0.5994 on the test dataset, corresponding to 11.6633% degradation. This is the only metric exceeding the 10% threshold and is the only result marked Fail.
• Class 1 metrics remain within threshold: For class 1, all three evaluated metrics pass the test. Degradation is 7.9353% for precision, 1.4465% for recall, and 4.6909% for F1-score.
• Most degradation is moderate: Excluding class 0 recall, all remaining degradation values fall between 1.4465% and 8.7620%. These results remain below the stated maximum threshold.
• Test scores are uniform within class: For class 1, the reported test score is 0.6031 across precision, recall, and F1-score, while for class 0 the reported test score is 0.5994 across all three metrics. The variation across metrics is therefore reflected in the training scores and calculated degradation percentages.

The results show limited train-to-test degradation across most evaluated classification metrics, with five of six measures remaining below the stated threshold. The only exception is recall for class 0, which records 11.6633% degradation and fails the test. Overall, the observed train-test performance gap is concentrated in one metric rather than broadly distributed across both classes and all measures.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6551	0.6031	7.9353	Pass
1	Recall	0.6119	0.6031	1.4465	Pass
1	F1-Score	0.6328	0.6031	4.6909	Pass
0	Precision	0.6367	0.5994	5.8607	Pass
0	Recall	0.6785	0.5994	11.6633	Fail
0	F1-Score	0.6569	0.5994	8.7620	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a predefined threshold. The result table reports an accuracy score of 0.6012 against a threshold of 0.7, with an overall test outcome of Fail. The output is presented as a direct comparison between the observed score and the configured minimum criterion.

Key insights:

• Accuracy below threshold: The observed accuracy score is 0.6012, which is lower than the threshold value of 0.7 used in this test.
• Test outcome is fail: The result is recorded as Fail, reflecting that the measured accuracy did not meet the minimum required level in this evaluation.
• Gap to minimum criterion: The difference between the observed score and the threshold is 0.0988, indicating the extent to which the result fell short of the configured minimum.

The test result shows that the model did not satisfy the minimum accuracy criterion under this evaluation. The measured accuracy of 0.6012 remained below the specified threshold of 0.7, and the test was therefore marked as failed. Collectively, the result indicates that the model’s overall classification accuracy, as measured in this run, did not reach the configured acceptance level.

Tables

Score	Threshold	Pass/Fail
0.6012	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s validation F1 score meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6031, alongside a threshold value of 0.5 and an overall outcome of Pass. This output presents the observed F1 score directly against the configured minimum standard used for the test.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6031, which is above the minimum threshold of 0.5 by 0.1031.
• Test outcome is pass: The recorded Pass/Fail status is Pass, indicating the test criterion was met for the evaluated validation result.

The test result shows that the model achieved an F1 score of 0.6031 on the validation dataset and that this value exceeded the configured minimum threshold of 0.5. The reported pass status is consistent with the observed score-threshold comparison.

Tables

Score	Threshold	Pass/Fail
0.6031	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC on the validation dataset meets or exceeds a predefined minimum threshold. The result table reports a ROC AUC score of 0.6498, a threshold of 0.5, and an overall outcome of Pass. These values indicate the measured classification performance level recorded by this test and the benchmark used for assessment.

Key insights:

• Threshold exceeded: The recorded ROC AUC score of 0.6498 is above the configured minimum threshold of 0.5 used in this test.
• Test passed: The result is explicitly marked as Pass, consistent with the score exceeding the stated threshold.
• Positive margin to threshold: The observed score exceeds the threshold by 0.1498, showing a measurable buffer above the minimum acceptance level.

The test result shows that the model satisfied the minimum ROC AUC criterion on the validation dataset. The recorded score of 0.6498 exceeded the threshold of 0.5, and the test outcome was therefore documented as Pass. Within the scope of this test, the model met the defined minimum discrimination benchmark.

Tables

Score	Threshold	Pass/Fail
0.6498	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative contribution of each input feature by measuring the change in model performance after randomly permuting that feature’s values. The result is presented as a horizontal bar chart of permutation importance values across the model inputs. The plot shows two features with clearly larger positive importance values, a small group with near-zero positive or negative values, and several features with importance effectively at or very near zero.

Key insights:

• Geography_Germany dominates importance: Geography_Germany has the largest permutation importance in the chart, at approximately 0.065, indicating the largest observed performance decrease when this feature is shuffled.

• IsActiveMember is the second-largest driver: IsActiveMember shows the next highest importance, at approximately 0.046, and is the only other feature with a clearly material positive value.

• Most remaining features contribute minimally: Balance and Gender_Male have only small positive importance values, both close to zero relative to the top two features.

• Several features show near-zero impact: Geography_Spain, HasCrCard, and Tenure appear at or essentially indistinguishable from zero importance in the plot.

• Some features have negative importance: NumOfProducts, EstimatedSalary, and CreditScore display slightly negative permutation importance values, with NumOfProducts the most negative among them.

The results show that model performance is primarily influenced by Geography_Germany and IsActiveMember, with a large separation between these two features and the rest of the input set. Most other variables have limited measured contribution under this permutation-based assessment, and a small subset exhibits slightly negative importance values, indicating that permuting them did not reduce performance in this test result.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance by measuring the average absolute SHAP contribution of each input variable and visualizing both the magnitude and direction of those contributions. The results include a normalized feature importance bar chart and a SHAP summary plot. The bar chart ranks features by relative importance, while the summary plot shows how low and high feature values align with negative or positive SHAP impacts on model output across observations.

Key insights:

• Importance is concentrated in four features: IsActiveMember, Geography_Germany, Gender_Male, and Balance have materially larger normalized SHAP importance than the remaining variables. The first three are the highest-ranked features, and Balance forms a second tier below them.

• IsActiveMember is the strongest driver: IsActiveMember has the highest normalized importance in the bar chart. In the summary plot, low feature values are associated with positive SHAP values and high feature values with negative SHAP values, indicating a clear directional separation in contribution.

• Geography_Germany and Gender_Male show strong directional effects: Both features are among the top three by normalized importance. In the summary plot, higher feature values align with positive SHAP contributions, while lower values align with negative contributions, producing distinct separation across the SHAP axis.

• Balance has broad impact dispersion: Balance is the fourth most important feature and shows the widest spread of SHAP values among the continuous variables. Higher Balance values are concentrated on the positive side of the SHAP axis, while lower values appear more often on the negative side.

• Remaining features contribute marginally: EstimatedSalary has modest importance relative to the top four features, while CreditScore and NumOfProducts are near zero in the normalized importance chart. Geography_Spain, HasCrCard, and Tenure show essentially no visible global importance and their SHAP values are tightly clustered around zero.

The SHAP results show that model behavior is dominated by a small set of features, with IsActiveMember, Geography_Germany, Gender_Male, and Balance accounting for most of the observed global importance. The summary plot indicates clear directional patterns for the top-ranked features, particularly for IsActiveMember and the binary geography and gender indicators, while Balance contributes with a broader range of impacts. Other variables have limited visible influence on model output in this test, with SHAP contributions concentrated near zero.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance within localized regions of the feature space by comparing accuracy, precision, recall, and F1 across binned slices of individual features on the training and test datasets. The results are reported for Balance, CreditScore, EstimatedSalary, Tenure, Gender_Male, Geography_Germany, Geography_Spain, HasCrCard, IsActiveMember, and NumOfProducts, with thresholds shown in the plots for each metric. Across the reported slices, metric values vary materially by feature segment, and the charts highlight whether test-set performance in each slice falls above or below the stated thresholds. The results also allow direct comparison between training and test performance within the same slice.

Key insights:

• Accuracy is below threshold throughout: In the plotted features, the accuracy threshold is 0.75, and all reported test accuracy values for CreditScore, Tenure, EstimatedSalary, HasCrCard, IsActiveMember, Geography_Germany, Geography_Spain, and Gender_Male remain below this level. Balance shows only a few test slices at or above 0.75, including 25,089.809–50,179.618 (0.8333), 175,628.663–200,718.472 (0.7500), and 200,718.472–225,808.281 (0.7500).

• IsActiveMember shows the clearest weak spot: For the active-member slice (0.9, 1.0], test recall is 0.2975 and test F1 is 0.3892, versus 0.7843 recall and 0.6882 F1 for the non-active slice (-0.001, 0.1]. The same pattern is present in training data, where recall is 0.3319 and F1 is 0.4389 for active members compared with 0.7755 and 0.7113 for non-active members.

• Gender split has sharply different recall/F1 profiles: For Gender_Male, the test slice (-0.001, 0.1] has recall 0.8171 and F1 0.6872, while the test slice (0.9, 1.0] has recall 0.3851 and F1 0.4769. Training data shows a similar separation, with recall 0.7883 vs. 0.3810 and F1 0.7249 vs. 0.4707 across the same two slices.

• Geography slices show asymmetric behavior: In Geography_Germany, the Germany slice (0.9, 1.0] has substantially higher test recall and F1 (0.8779 and 0.7468) than the non-Germany slice (0.4175 and 0.4737). In Geography_Spain, the pattern reverses, with the Spain slice (0.9, 1.0] showing lower test recall and F1 (0.3625 and 0.4427) than the non-Spain slice (0.6816 and 0.6435).

• NumOfProducts is weakest for two-product accounts: The NumOfProducts slice (1.9, 2.2] records the lowest test F1 at 0.4133 and the lowest test precision at 0.3196, compared with F1 values of 0.6478, 0.7188, and 0.7692 in the other slices. This pattern is also present in training, where the same slice has F1 0.4496 and precision 0.3660.

• CreditScore performance is uneven by band: On the test set, CreditScore slices range from F1 0.5217 to 0.8000 and from accuracy 0.5000 to 0.7368. The weakest test F1 values occur at 500–550 (0.5217) and 800–850 (0.5217), while the strongest occur at 349.5–400.0 (0.8000, based on 3 records) and 400–450 (0.7368).

• Tenure improves in selected mid-range bins: Test performance is weakest in lower-tenure slices, with accuracy between 0.5000 and 0.5811 and F1 between 0.5098 and 0.5974 for tenure up to 4.0. Higher values appear in selected mid-range bins, including 4.0–5.0 (F1 0.6923) and 6.0–7.0 (F1 0.6970), before moderating again in later bins.

• Balance results include sparse extreme bins: Several Balance slices have very small record counts on the test set, including 6, 4, and 0 records in the upper ranges. The slice 225,808.281–250,898.09 has 0 test records and zero-valued metrics, while the 25,089.809–50,179.618 slice has 6 test records and high reported metrics, including accuracy 0.8333 and F1 0.8571.

• EstimatedSalary is comparatively stable but still sub-threshold on F1: Test F1 for EstimatedSalary ranges from 0.5116 to 0.7692, with the strongest result in the 179,927.563–199,909.32 slice. Most other salary bins cluster more tightly, with test accuracy generally between 0.5469 and 0.6364 and test precision between 0.5000 and 0.6667.

Overall, the test identifies substantial heterogeneity in model performance across feature-space slices rather than a uniform error profile. The most pronounced weak regions are associated with IsActiveMember, Gender_Male, Geography_Spain, Geography_Germany, and the two-product NumOfProducts slice, where recall and F1 differ sharply across segments. Balance and some low-count CreditScore slices show more volatile results because several bins contain very few test observations, while EstimatedSalary and parts of Tenure exhibit comparatively narrower variation despite remaining below the plotted accuracy threshold in most slices.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-250.898, 25089.809]	202	Balance	0.6337	0.4909	0.3699	0.4219	test_dataset_final
(25089.809, 50179.618]	6	Balance	0.8333	1.0000	0.7500	0.8571	test_dataset_final
(50179.618, 75269.427]	23	Balance	0.5652	0.7143	0.3846	0.5000	test_dataset_final
(75269.427, 100359.236]	77	Balance	0.4805	0.4865	0.4615	0.4737	test_dataset_final
(100359.236, 125449.045]	146	Balance	0.6233	0.7143	0.6915	0.7027	test_dataset_final
(125449.045, 150538.854]	122	Balance	0.6148	0.6098	0.7692	0.6803	test_dataset_final
(150538.854, 175628.663]	55	Balance	0.5091	0.5000	0.7037	0.5846	test_dataset_final
(175628.663, 200718.472]	12	Balance	0.7500	0.7500	0.8571	0.8000	test_dataset_final
(200718.472, 225808.281]	4	Balance	0.7500	0.7500	1.0000	0.8571	test_dataset_final
(225808.281, 250898.09]	0	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(-250.898, 25089.809]	831	Balance	0.6426	0.5741	0.3769	0.4550	train_dataset_final
(25089.809, 50179.618]	20	Balance	0.5000	0.5000	0.2000	0.2857	train_dataset_final
(50179.618, 75269.427]	109	Balance	0.5505	0.5250	0.4118	0.4615	train_dataset_final
(75269.427, 100359.236]	293	Balance	0.6382	0.6286	0.6197	0.6241	train_dataset_final
(100359.236, 125449.045]	593	Balance	0.7133	0.7642	0.7378	0.7507	train_dataset_final
(125449.045, 150538.854]	483	Balance	0.6563	0.6821	0.7464	0.7128	train_dataset_final
(150538.854, 175628.663]	189	Balance	0.5450	0.5397	0.7083	0.6126	train_dataset_final
(175628.663, 200718.472]	49	Balance	0.5306	0.5312	0.6800	0.5965	train_dataset_final
(200718.472, 225808.281]	16	Balance	0.4375	0.7000	0.5385	0.6087	train_dataset_final
(225808.281, 250898.09]	2	Balance	0.5000	1.0000	0.5000	0.6667	train_dataset_final
(349.5, 400.0]	3	CreditScore	0.6667	1.0000	0.6667	0.8000	test_dataset_final
(400.0, 450.0]	19	CreditScore	0.7368	0.7778	0.7000	0.7368	test_dataset_final
(450.0, 500.0]	26	CreditScore	0.5769	0.5714	0.6154	0.5926	test_dataset_final
(500.0, 550.0]	66	CreditScore	0.5000	0.5625	0.4865	0.5217	test_dataset_final
(550.0, 600.0]	78	CreditScore	0.6026	0.5676	0.5833	0.5753	test_dataset_final
(600.0, 650.0]	130	CreditScore	0.5923	0.6000	0.5909	0.5954	test_dataset_final
(650.0, 700.0]	118	CreditScore	0.6441	0.6094	0.6964	0.6500	test_dataset_final
(700.0, 750.0]	112	CreditScore	0.6161	0.6939	0.5484	0.6126	test_dataset_final
(750.0, 800.0]	64	CreditScore	0.5625	0.5500	0.6875	0.6111	test_dataset_final
(800.0, 850.0]	31	CreditScore	0.6452	0.4615	0.6000	0.5217	test_dataset_final
(349.5, 400.0]	11	CreditScore	0.4545	1.0000	0.4545	0.6250	train_dataset_final
(400.0, 450.0]	39	CreditScore	0.5128	0.6316	0.5000	0.5581	train_dataset_final
(450.0, 500.0]	115	CreditScore	0.6522	0.6721	0.6721	0.6721	train_dataset_final
(500.0, 550.0]	261	CreditScore	0.6207	0.6587	0.5971	0.6264	train_dataset_final
(550.0, 600.0]	376	CreditScore	0.6436	0.6941	0.5900	0.6378	train_dataset_final
(600.0, 650.0]	481	CreditScore	0.6237	0.6129	0.5783	0.5951	train_dataset_final
(650.0, 700.0]	491	CreditScore	0.6395	0.6391	0.6100	0.6242	train_dataset_final
(700.0, 750.0]	378	CreditScore	0.6825	0.6805	0.6354	0.6571	train_dataset_final
(750.0, 800.0]	240	CreditScore	0.6875	0.6613	0.7130	0.6862	train_dataset_final
(800.0, 850.0]	193	CreditScore	0.6580	0.6353	0.6067	0.6207	train_dataset_final
(-108.068, 20073.507]	82	EstimatedSalary	0.5610	0.5714	0.5714	0.5714	test_dataset_final
(20073.507, 40055.264]	64	EstimatedSalary	0.5469	0.5625	0.5455	0.5538	test_dataset_final
(40055.264, 60037.021]	66	EstimatedSalary	0.6364	0.6471	0.6471	0.6471	test_dataset_final
(60037.021, 80018.778]	70	EstimatedSalary	0.6143	0.6000	0.6857	0.6400	test_dataset_final
(80018.778, 100000.535]	68	EstimatedSalary	0.5588	0.5806	0.5143	0.5455	test_dataset_final
(100000.535, 119982.292]	55	EstimatedSalary	0.6182	0.5000	0.5714	0.5333	test_dataset_final
(119982.292, 139964.049]	49	EstimatedSalary	0.5714	0.5238	0.5000	0.5116	test_dataset_final
(139964.049, 159945.806]	61	EstimatedSalary	0.5902	0.6667	0.6111	0.6377	test_dataset_final
(159945.806, 179927.563]	65	EstimatedSalary	0.6154	0.6250	0.4839	0.5455	test_dataset_final
(179927.563, 199909.32]	65	EstimatedSalary	0.7231	0.7143	0.8333	0.7692	test_dataset_final
(-108.068, 20073.507]	232	EstimatedSalary	0.6595	0.7238	0.6032	0.6580	train_dataset_final
(20073.507, 40055.264]	249	EstimatedSalary	0.6506	0.6636	0.5935	0.6266	train_dataset_final
(40055.264, 60037.021]	267	EstimatedSalary	0.6330	0.6036	0.5537	0.5776	train_dataset_final
(60037.021, 80018.778]	268	EstimatedSalary	0.6493	0.7019	0.5368	0.6083	train_dataset_final
(80018.778, 100000.535]	256	EstimatedSalary	0.6602	0.6240	0.6610	0.6420	train_dataset_final
(100000.535, 119982.292]	266	EstimatedSalary	0.5865	0.6165	0.5816	0.5985	train_dataset_final
(119982.292, 139964.049]	278	EstimatedSalary	0.6619	0.6691	0.6596	0.6643	train_dataset_final
(139964.049, 159945.806]	246	EstimatedSalary	0.6748	0.6525	0.6638	0.6581	train_dataset_final
(159945.806, 179927.563]	281	EstimatedSalary	0.6441	0.6690	0.6419	0.6552	train_dataset_final
(179927.563, 199909.32]	242	EstimatedSalary	0.6364	0.6387	0.6281	0.6333	train_dataset_final
(-0.001, 0.1]	323	Gender_Male	0.6223	0.5929	0.8171	0.6872	test_dataset_final
(0.9, 1.0]	324	Gender_Male	0.5802	0.6263	0.3851	0.4769	test_dataset_final
(-0.001, 0.1]	1259	Gender_Male	0.6521	0.6709	0.7883	0.7249	train_dataset_final
(0.9, 1.0]	1326	Gender_Male	0.6388	0.6156	0.3810	0.4707	train_dataset_final
(-0.001, 0.1]	449	Geography_Germany	0.5991	0.5473	0.4175	0.4737	test_dataset_final
(0.9, 1.0]	198	Geography_Germany	0.6061	0.6497	0.8779	0.7468	test_dataset_final
(-0.001, 0.1]	1800	Geography_Germany	0.6256	0.5887	0.4062	0.4807	train_dataset_final
(0.9, 1.0]	785	Geography_Germany	0.6904	0.7071	0.9140	0.7973	train_dataset_final
(-0.001, 0.1]	478	Geography_Spain	0.6130	0.6095	0.6816	0.6435	test_dataset_final
(0.9, 1.0]	169	Geography_Spain	0.5680	0.5686	0.3625	0.4427	test_dataset_final
(-0.001, 0.1]	2009	Geography_Spain	0.6461	0.6583	0.6590	0.6587	train_dataset_final
(0.9, 1.0]	576	Geography_Spain	0.6424	0.6341	0.4160	0.5024	train_dataset_final
(-0.001, 0.1]	190	HasCrCard	0.5632	0.5930	0.5152	0.5514	test_dataset_final
(0.9, 1.0]	457	HasCrCard	0.6171	0.6067	0.6416	0.6237	test_dataset_final
(-0.001, 0.1]	761	HasCrCard	0.6229	0.6508	0.5899	0.6189	train_dataset_final
(0.9, 1.0]	1824	HasCrCard	0.6546	0.6568	0.6217	0.6388	train_dataset_final
(-0.001, 0.1]	352	IsActiveMember	0.5881	0.6130	0.7843	0.6882	test_dataset_final
(0.9, 1.0]	295	IsActiveMember	0.6169	0.5625	0.2975	0.3892	test_dataset_final
(-0.001, 0.1]	1376	IsActiveMember	0.6272	0.6570	0.7755	0.7113	train_dataset_final
(0.9, 1.0]	1209	IsActiveMember	0.6658	0.6475	0.3319	0.4389	train_dataset_final
(0.997, 1.3]	376	NumOfProducts	0.6037	0.6850	0.6143	0.6478	test_dataset_final
(1.9, 2.2]	221	NumOfProducts	0.6018	0.3196	0.5849	0.4133	test_dataset_final
(2.8, 3.1]	42	NumOfProducts	0.5714	1.0000	0.5610	0.7188	test_dataset_final
(3.7, 4.0]	8	NumOfProducts	0.6250	1.0000	0.6250	0.7692	test_dataset_final
(0.997, 1.3]	1499	NumOfProducts	0.6464	0.7447	0.6236	0.6788	train_dataset_final
(1.9, 2.2]	901	NumOfProducts	0.6548	0.3660	0.5826	0.4496	train_dataset_final
(2.8, 3.1]	149	NumOfProducts	0.6040	0.9545	0.6043	0.7401	train_dataset_final
(3.7, 4.0]	36	NumOfProducts	0.5278	1.0000	0.5278	0.6909	train_dataset_final
(-0.01, 1.0]	88	Tenure	0.5000	0.5556	0.5102	0.5319	test_dataset_final
(1.0, 2.0]	58	Tenure	0.5690	0.4815	0.5417	0.5098	test_dataset_final
(2.0, 3.0]	67	Tenure	0.5224	0.5143	0.5455	0.5294	test_dataset_final
(3.0, 4.0]	74	Tenure	0.5811	0.6765	0.5349	0.5974	test_dataset_final
(4.0, 5.0]	69	Tenure	0.6522	0.7105	0.6750	0.6923	test_dataset_final
(5.0, 6.0]	64	Tenure	0.6406	0.5312	0.6800	0.5965	test_dataset_final
(6.0, 7.0]	68	Tenure	0.7059	0.7188	0.6765	0.6970	test_dataset_final
(7.0, 8.0]	67	Tenure	0.6269	0.6364	0.6176	0.6269	test_dataset_final
(8.0, 9.0]	55	Tenure	0.6545	0.6207	0.6923	0.6545	test_dataset_final
(9.0, 10.0]	37	Tenure	0.5946	0.5500	0.6471	0.5946	test_dataset_final
(-0.01, 1.0]	402	Tenure	0.6667	0.6806	0.6404	0.6599	train_dataset_final
(1.0, 2.0]	273	Tenure	0.6190	0.6136	0.6045	0.6090	train_dataset_final
(2.0, 3.0]	260	Tenure	0.6269	0.6930	0.5603	0.6196	train_dataset_final
(3.0, 4.0]	249	Tenure	0.6305	0.6607	0.5781	0.6167	train_dataset_final
(4.0, 5.0]	255	Tenure	0.6745	0.6391	0.7083	0.6719	train_dataset_final
(5.0, 6.0]	247	Tenure	0.6235	0.6731	0.5426	0.6009	train_dataset_final
(6.0, 7.0]	233	Tenure	0.6395	0.5922	0.5922	0.5922	train_dataset_final
(7.0, 8.0]	273	Tenure	0.6850	0.6639	0.6429	0.6532	train_dataset_final
(8.0, 9.0]	275	Tenure	0.6364	0.6439	0.6159	0.6296	train_dataset_final
(9.0, 10.0]	118	Tenure	0.6271	0.6984	0.6377	0.6667	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences in training and test performance across feature-based slices to identify regions where the AUC gap exceeds the configured 0.04 threshold. The reported results list feature slices with training and test record counts, slice-level training AUC, test AUC, and the resulting gap. Over-threshold gaps are present across multiple features, including CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, Geography_Germany, and Geography_Spain. The largest observed gaps are concentrated in selected bins of Balance, CreditScore, and EstimatedSalary, with additional moderate gaps appearing in several binary and tenure-related segments.

Key insights:

• Largest gap occurs in a very small Balance slice: The Balance slice (200718.472, 225808.281] shows the largest gap at 0.2051, with training AUC 0.2051 and test AUC 0.0. This slice contains 16 training records and 4 test records.

• CreditScore shows multiple over-threshold regions: Five CreditScore slices exceed the threshold, with the largest gap in (500.0, 550.0] at 0.1790 (0.6897 train AUC vs 0.5107 test AUC). Additional over-threshold gaps appear in (550.0, 600.0], (600.0, 650.0], (750.0, 800.0], and (800.0, 850.0].

• EstimatedSalary has repeated localized gaps: Six EstimatedSalary slices exceed the threshold. The largest is (20073.507, 40055.264] with a gap of 0.1632, followed by (80018.778, 100000.535] at 0.1223; other affected salary ranges span low, middle, and upper-mid intervals.

• Tenure gaps are concentrated in early and mid tenure bins: The largest Tenure gaps occur in (-0.01, 1.0] at 0.1326 and (2.0, 3.0] at 0.1225. Additional above-threshold gaps are present in (3.0, 4.0], (7.0, 8.0], and (9.0, 10.0].

• Balance contains several high-gap segments: Beyond the smallest high-balance slice, the Balance intervals (75269.427, 100359.236] and (100359.236, 125449.045] show gaps of 0.1353 and 0.0926, respectively. The lowest balance interval (-250.898, 25089.809] also exceeds the threshold at 0.0509.

• Binary features show modest but consistent gaps: HasCrCard exceeds the threshold in both categories, with gaps of 0.0458 for (-0.001, 0.1] and 0.0438 for (0.9, 1.0]. IsActiveMember exceeds the threshold in the active-member category (0.9, 1.0] with a gap of 0.0643, while Geography_Germany (0.9, 1.0] and Geography_Spain (0.9, 1.0] show gaps of 0.1133 and 0.0666.

The test results show that training-test AUC differences above the 0.04 threshold are distributed across several features rather than isolated to a single variable. The most pronounced gaps appear in selected Balance, CreditScore, EstimatedSalary, and Tenure slices, with the largest value occurring in a very small Balance segment. Smaller but still above-threshold differences are also present in binary indicator features, indicating that the observed performance separation spans both continuous-variable bins and categorical segments.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(500.0, 550.0]	261	66	0.6897	0.5107	0.1790
CreditScore	(550.0, 600.0]	376	78	0.7126	0.6660	0.0466
CreditScore	(600.0, 650.0]	481	130	0.6787	0.6151	0.0637
CreditScore	(750.0, 800.0]	240	64	0.7041	0.6455	0.0586
CreditScore	(800.0, 850.0]	193	31	0.7180	0.6429	0.0752
Tenure	(-0.01, 1.0]	402	88	0.7082	0.5756	0.1326
Tenure	(2.0, 3.0]	260	67	0.6733	0.5508	0.1225
Tenure	(3.0, 4.0]	249	74	0.6694	0.6234	0.0460
Tenure	(7.0, 8.0]	273	67	0.7352	0.6684	0.0667
Tenure	(9.0, 10.0]	118	37	0.6838	0.6353	0.0485
Balance	(-250.898, 25089.809]	831	202	0.6571	0.6061	0.0509
Balance	(75269.427, 100359.236]	293	77	0.6556	0.5202	0.1353
Balance	(100359.236, 125449.045]	593	146	0.7633	0.6706	0.0926
Balance	(200718.472, 225808.281]	16	4	0.2051	0.0000	0.2051
NumOfProducts	(0.997, 1.3]	1499	376	0.6735	0.6268	0.0467
HasCrCard	(-0.001, 0.1]	761	190	0.6621	0.6163	0.0458
HasCrCard	(0.9, 1.0]	1824	457	0.7069	0.6632	0.0438
IsActiveMember	(0.9, 1.0]	1209	295	0.6768	0.6125	0.0643
EstimatedSalary	(-108.068, 20073.507]	232	82	0.7264	0.6298	0.0967
EstimatedSalary	(20073.507, 40055.264]	249	64	0.7233	0.5601	0.1632
EstimatedSalary	(60037.021, 80018.778]	268	70	0.6765	0.6041	0.0724
EstimatedSalary	(80018.778, 100000.535]	256	68	0.7101	0.5879	0.1223
EstimatedSalary	(119982.292, 139964.049]	278	49	0.7242	0.6380	0.0861
EstimatedSalary	(139964.049, 159945.806]	246	61	0.7132	0.6200	0.0932
Geography_Germany	(0.9, 1.0]	785	198	0.6429	0.5296	0.1133
Geography_Spain	(0.9, 1.0]	576	169	0.6784	0.6118	0.0666

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model robustness by measuring AUC changes after adding Gaussian noise to numeric input features at increasing perturbation sizes. The results are reported for both train_dataset_final and test_dataset_final from the baseline condition through perturbation sizes of 0.1 to 0.5 standard deviations. The table and plot show AUC values and corresponding performance decay at each noise level, together with pass/fail status for each dataset. The perturbed features listed in the figure are CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, Geography_Germany, Geography_Spain, and Gender_Male.

Key insights:

• AUC declines as noise increases: Training AUC decreases from 0.694 at baseline to 0.6757 at perturbation size 0.5, while test AUC decreases from 0.6498 to 0.6328 over the same range. This corresponds to monotonic degradation on the training set and an overall downward pattern on the test set.

• All perturbation levels passed: Every evaluated condition is marked Passed = true for both training and test datasets, including the highest perturbation size of 0.5.

• Train performance decays gradually: Training performance decay rises from 0.0014 at perturbation size 0.1 to 0.0183 at 0.5. The increase is incremental across all tested noise levels.

• Test response shows minor variability at low noise: On the test dataset, AUC increases slightly from 0.6498 at baseline to 0.6522 at perturbation size 0.1, producing a performance decay of -0.0024. After that point, test AUC declines to 0.6477, 0.6447, 0.6367, and 0.6328 as perturbation size increases from 0.2 to 0.5.

• Largest observed decay occurs at highest noise: The maximum performance decay is observed at perturbation size 0.5 for both datasets, with 0.0183 on the training set and 0.0170 on the test set.

The robustness results show that model discrimination declines under progressively stronger Gaussian perturbations, with the largest reductions occurring at the highest tested noise level. The training set exhibits a steady monotonic decrease in AUC, while the test set shows a small improvement at the lowest perturbation level before declining thereafter. Across all evaluated perturbation sizes, both datasets remain within the test’s passing criteria.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6940	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6498	0.0000	True
0.1	train_dataset_final	2585	0.6925	0.0014	True
0.1	test_dataset_final	647	0.6522	-0.0024	True
0.2	train_dataset_final	2585	0.6891	0.0049	True
0.2	test_dataset_final	647	0.6477	0.0021	True
0.3	train_dataset_final	2585	0.6855	0.0085	True
0.3	test_dataset_final	647	0.6447	0.0052	True
0.4	train_dataset_final	2585	0.6829	0.0110	True
0.4	test_dataset_final	647	0.6367	0.0131	True
0.5	train_dataset_final	2585	0.6757	0.0183	True
0.5	test_dataset_final	647	0.6328	0.0170	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial