ValidMind for development 2 — Start the development process

Learn how to use ValidMind for your end-to-end documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a record (model) is being built appropriately.

For a full list of out-of-the-box tests and descriptions, use the interactive ValidMind test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. On the left sidebar that appears for your model, select Getting Started and select Development from the DOCUMENT drop-down menu.

2. Click Copy snippet to clipboard.

3. Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Note: you may need to restart the kernel to use updated packages.

2026-05-26 22:03:12,559 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outlier',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value percentage across all features does not exceed a set threshold....	False	True	['dataset']	{'min_percentage_threshold': {'type': 'float', 'default': 1.0}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind dataset

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests on datasets

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results are presented in separate summary tables for eight numerical variables and two categorical variables, including counts, central tendency, dispersion, percentile ranges, and category concentration. All reported variables have a count of 8,000 observations, and the tables show the observed ranges and frequency distributions for each field. The numerical summaries include percentiles through the 95th percentile, while the categorical summaries report the number of unique values and the most frequent category with its frequency share.

Key insights:

• Complete coverage across all variables: Every numerical and categorical variable reports a count of 8,000, indicating consistent observation counts across the fields included in this test.

• Balance shows pronounced dispersion: Balance has a mean of 76,434.0965, a median of 97,264.0, and a standard deviation of 62,612.2513, with the 25th percentile at 0.0 and the maximum at 250,898.0. This reflects a wide spread of observed values and a substantial concentration at zero in the lower quartile.

• Salary is broadly distributed: EstimatedSalary spans from 12.0 to 199,992.0, with a mean of 99,790.188 and median of 99,505.0. The standard deviation of 57,520.5089 indicates substantial dispersion across the observed salary range.

• Credit score and age are centered with broad ranges: CreditScore has a mean of 650.1596 and median of 652.0 over a range of 350.0 to 850.0, while Age has a mean of 38.9489 and median of 37.0 over a range of 18.0 to 92.0. In both variables, mean and median are relatively close compared with the full observed range.

• Product holdings are concentrated at low counts: NumOfProducts has a median of 1.0, a 75th percentile of 2.0, and a maximum of 4.0, with a mean of 1.5325. This indicates that most observations are concentrated in the lower product-count categories.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the top category at 4,010 observations (50.12%), and Gender contains 2 unique values, with Male as the top category at 4,396 observations (54.95%). Neither categorical variable is dominated by a single category far beyond half of the sample.

The descriptive statistics show a dataset with complete reported coverage across all variables and a mix of compact and widely dispersed distributions. Balance is the most unevenly distributed numerical variable in the summary, with a zero-valued lower quartile and large spread, while EstimatedSalary also exhibits a wide range with mean and median closely aligned. CreditScore and Age show central tendency measures that are relatively close, and the categorical variables display moderate concentration rather than extreme dominance by a single category.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a minimum percentage threshold. In this run, the target variable Exited contains two classes, with class 0 representing 79.80% of rows and class 1 representing 20.20% of rows. The applied threshold is 30%, and the result table reports a pass/fail outcome for each class based on that cutoff. The accompanying bar chart visually reflects this distribution, with a substantially taller bar for class 0 than for class 1.

Key insights:

• Majority class dominates distribution: Exited = 0 accounts for 79.80% of observations, while Exited = 1 accounts for 20.20%, indicating a pronounced difference in class representation.
• Minority class fails threshold: With the minimum percentage threshold set to 30%, class 1 falls below the cutoff at 20.20% and is marked as Fail.
• Only one class passes: Class 0 exceeds the threshold at 79.80% and is marked as Pass, while the other class does not meet the configured minimum.

The test result shows that the target distribution is uneven under the configured 30% threshold. One class comprises nearly four-fifths of the dataset and passes the test, while the other class represents roughly one-fifth of observations and fails. Taken together, the table and plot document that the dataset contains a materially underrepresented target class relative to the threshold used in this assessment.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the validmind.data_validation.ClassImbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring the share of records in each class against a specified minimum percentage threshold. In this run, the target variable Exited contains two classes, 0 and 1, and the results table reports the percentage of rows and pass/fail status for each class. Both classes account for 50.00% of records, and both are assessed against the configured minimum percentage threshold of 30%. The accompanying bar chart visually confirms an equal split between the two classes.

Key insights:

• Classes are evenly distributed: Exited = 0 and Exited = 1 each represent 50.00% of the dataset, indicating a perfectly balanced binary class distribution in the observed sample.
• Both classes pass the threshold: Each class exceeds the configured 30% minimum percentage threshold and is marked as Pass in the test output.
• No minority class is present: The reported class proportions show no under-represented target class within the tested dataset.

The test results show that the target variable is evenly split across its two classes, with each class materially above the configured minimum threshold. The tabular output and chart are aligned in showing a 50.00% / 50.00% class distribution, and no class is flagged as imbalanced under the parameters used for this test.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among dataset features to identify highly correlated variables that may indicate redundancy or multicollinearity. The results table lists the strongest reported feature pairs, their Pearson correlation coefficients, and a Pass/Fail status based on the configured absolute correlation threshold of 0.3. Among the ten reported pairs, coefficients range from -0.1837 to 0.3379, with one pair exceeding the threshold and the remaining pairs falling below it.

Key insights:

• One pair exceeds threshold: The pair (Age, Exited) has a Pearson correlation coefficient of 0.3379, which is the only reported relationship marked Fail against the 0.3 threshold.
• Remaining correlations are modest: The other nine reported feature pairs all pass the test, with absolute correlation values ranging from 0.0391 to 0.1837, remaining below the configured threshold.
• Largest negative correlation is limited: The most negative reported relationship is (IsActiveMember, Exited) at -0.1837, which remains within the passing range.
• Top reported relationships are concentrated at low magnitudes: Aside from (Age, Exited), the reported coefficients are relatively small in magnitude, including (Balance, NumOfProducts) at -0.1697 and (Balance, Exited) at 0.1429.

The reported correlation structure is characterized by a single feature pair above the configured threshold and a broader set of pairwise relationships with low absolute magnitudes. The strongest observed relationship is between Age and Exited, while all other listed pairs remain below the test limit. Overall, the output indicates limited high linear correlation among the reported top feature pairs, with the exception of the single failed pair.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3379	Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1697	Pass
(Balance, Exited)	0.1429	Pass
(NumOfProducts, Exited)	-0.0609	Pass
(Balance, HasCrCard)	-0.0510	Pass
(Age, NumOfProducts)	-0.0488	Pass
(Tenure, IsActiveMember)	-0.0476	Pass
(NumOfProducts, IsActiveMember)	0.0441	Pass
(CreditScore, IsActiveMember)	0.0391	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3379	Fail
1	(IsActiveMember, Exited)	-0.1837	Pass
2	(Balance, NumOfProducts)	-0.1697	Pass
3	(Balance, Exited)	0.1429	Pass
4	(NumOfProducts, Exited)	-0.0609	Pass
5	(Balance, HasCrCard)	-0.0510	Pass
6	(Age, NumOfProducts)	-0.0488	Pass
7	(Tenure, IsActiveMember)	-0.0476	Pass
8	(NumOfProducts, IsActiveMember)	0.0441	Pass
9	(CreditScore, IsActiveMember)	0.0391	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or signs of multicollinearity. The result table reports the top 10 feature pairs by absolute Pearson correlation coefficient, along with pass/fail status against the configured threshold of 0.3. In this run, all reported coefficients are below the threshold and are classified as Pass. Observed coefficients range from -0.1837 to 0.1429, covering both positive and negative relationships.

Key insights:

• No correlations exceed threshold: All 10 reported feature pairs pass the test against the 0.3 threshold, with no absolute correlation coefficient reaching the configured limit.
• Largest relationship is modest: The strongest observed pair is (IsActiveMember, Exited) with a coefficient of -0.1837, which remains below the threshold and indicates a limited linear relationship in magnitude.
• Top positive correlation is low: The largest positive coefficient is 0.1429 for (Balance, Exited), followed by 0.0441 for (NumOfProducts, IsActiveMember) and 0.0391 for (CreditScore, IsActiveMember).
• Negative correlations dominate the top pairs: Seven of the 10 listed feature pairs have negative coefficients, including (Balance, NumOfProducts) at -0.1697 and (NumOfProducts, Exited) at -0.0609, indicating that the strongest observed linear relationships are more frequently negative than positive.

The reported correlation structure shows limited pairwise linear dependence among the listed features. All top-ranked correlations remain comfortably below the configured 0.3 threshold, and the strongest observed relationships are modest in magnitude. Based on the reported pairs, the test does not identify feature combinations with high Pearson correlation under the specified criterion.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1697	Pass
(Balance, Exited)	0.1429	Pass
(NumOfProducts, Exited)	-0.0609	Pass
(Balance, HasCrCard)	-0.0510	Pass
(Tenure, IsActiveMember)	-0.0476	Pass
(NumOfProducts, IsActiveMember)	0.0441	Pass
(CreditScore, IsActiveMember)	0.0391	Pass
(HasCrCard, IsActiveMember)	-0.0372	Pass
(CreditScore, Balance)	0.0346	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency among numerical variables by computing pairwise Pearson correlation coefficients and presenting them in a heat map. The result shows correlations across CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.18 to 0.14 outside the diagonal. The diagonal entries are 1.0 by construction, and the off-diagonal cells are concentrated near zero, with no pairwise correlation approaching the stated high-correlation threshold of 0.7 in absolute value.

Key insights:

• No high pairwise correlations: All off-diagonal correlation coefficients remain well below the 0.7 absolute threshold. The largest observed absolute correlation is 0.18 between IsActiveMember and Exited.
• Exited shows weak linear relationships: Correlations between Exited and the other variables are small in magnitude, ranging from -0.18 for IsActiveMember to 0.14 for Balance. Other associations with Exited include -0.06 for NumOfProducts and values near zero for CreditScore, Tenure, HasCrCard, and EstimatedSalary.
• Feature-to-feature dependencies are limited: Relationships among predictors are also weak, with the largest observed magnitudes being -0.17 between Balance and NumOfProducts, -0.05 between Balance and HasCrCard, and -0.05 between Tenure and IsActiveMember. Most remaining coefficients are between approximately -0.04 and 0.04.
• Correlation structure is broadly neutral: The matrix contains a mix of small positive and negative values without concentrated clusters of stronger association. This indicates an overall correlation pattern centered close to zero across the numerical variables shown.

The correlation matrix indicates that the numerical variables in this dataset exhibit weak linear dependence overall. Both predictor-to-predictor relationships and correlations with Exited remain small in magnitude, and no off-diagonal coefficient approaches the threshold identified in the test description for high correlation. Collectively, the result reflects a low level of linear redundancy among the variables included in the matrix.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the documentation.

To demonstrate how to add test results to your documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

The DatasetDescription test evaluates the structure, completeness, and cardinality of each dataset column. The results summarize 10 variables across numeric and categorical types, reporting count, missingness, and distinct-value levels for each field. All columns contain 3,232 observations, and the table shows the number and proportion of distinct values for variables including CreditScore, Balance, EstimatedSalary, and the categorical indicators Geography, Gender, HasCrCard, IsActiveMember, and Exited.

Key insights:

• No missing values observed: All 10 columns have a count of 3,232 with 0 missing values and 0.0% missingness, indicating full completeness in the analyzed dataset.
• EstimatedSalary is fully unique: EstimatedSalary has 3,232 distinct values, corresponding to a distinct proportion of 1.0, making it the highest-cardinality field in the dataset.
• Balance shows high numeric cardinality: Balance contains 2,211 distinct values, or 68.41% of observations, which is substantially higher than the other numeric variables except EstimatedSalary.
• Several fields have very low cardinality: Geography has 3 distinct values, Gender, HasCrCard, IsActiveMember, and Exited each have 2, and NumOfProducts has 4, indicating multiple low-cardinality discrete fields.
• Numeric fields vary in granularity: CreditScore has 438 distinct values, Tenure has 11, NumOfProducts has 4, Balance has 2,211, and EstimatedSalary has 3,232, showing substantial variation in value granularity across numeric inputs.

The dataset description indicates complete population coverage across all reported variables, with no observed missingness in any column. The column set combines low-cardinality categorical and discrete variables with higher-cardinality numeric fields, most notably Balance and EstimatedSalary. Overall, the results show a structurally complete dataset with heterogeneous feature granularity across inputs and the target column Exited.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	438	0.1355
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2211	0.6841
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

The Class Imbalance test evaluates the distribution of target classes in the dataset by measuring each class’s share of total records against a minimum percentage threshold. For the Exited target, the results table and accompanying bar chart show two classes, 0 and 1, each representing 50.00% of rows. The applied minimum percentage threshold is 30, and both classes are marked as Pass.

Key insights:

• Perfectly even class split: The Exited target is divided evenly between class 0 and class 1, with each class accounting for 50.00% of observations.
• All classes exceed threshold: Both target classes are above the 30% minimum percentage threshold used in this test and therefore receive a Pass result.
• No class under-representation detected: The observed distribution does not contain any class with a smaller share than the configured threshold, as confirmed by both the tabular output and the plot.

The test results show a balanced binary target distribution for Exited, with equal representation across both classes. Relative to the configured 30% threshold, neither class is under-represented, and both classes pass the class imbalance assessment.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

The Duplicates test evaluates whether the dataset contains exact duplicate rows that could affect data quality and training behavior. The result table reports the total number of duplicate rows and the corresponding share of the dataset. In this run, the dataset shows 0 duplicate rows, representing 0.0% of all rows.

Key insights:

• No duplicate rows detected: The test identified 0 duplicate rows in the dataset, indicating that no exact row-level repetitions were found.
• Duplicate share is zero: The percentage of rows classified as duplicates is 0.0%, showing that duplicates do not contribute to the observed dataset composition.

The test results indicate that exact duplicate entries were not present in the evaluated dataset. Both the absolute count and percentage measure are zero, showing a clean outcome under this duplicate-row check.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

The High Cardinality test evaluates the number of unique values in categorical columns to identify features with large counts of distinct categories. The result table reports, for each categorical column assessed, the number of distinct values, the percentage of distinct values, and the corresponding pass/fail outcome. In this run, two categorical columns were evaluated: Geography and Gender. Both columns are shown with low distinct-value counts and both received a passing result.

Key insights:

• All assessed categorical columns passed: Both Geography and Gender received a pass result, indicating that neither evaluated categorical feature exceeded the test threshold used in this run.
• Distinct counts are low: Geography has 3 distinct values and Gender has 2 distinct values, showing limited category breadth across the assessed categorical fields.
• Distinct-value percentages remain minimal: The percentage of distinct values is 0.0928% for Geography and 0.0619% for Gender, with Geography higher than Gender on both the absolute and percentage measures.

The result indicates that the categorical columns evaluated in this test do not exhibit high cardinality under the applied threshold. Geography has the higher distinct count and distinct-value percentage of the two assessed fields, but both remain within passing bounds. Across the evaluated categorical inputs, the observed category structure is limited and consistent with the test outcome.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

The Missing Values test evaluates dataset completeness by measuring the proportion of missing values in each feature against the configured threshold. The results are presented at the column level, showing the number of missing values, the corresponding missingness percentage, and the pass/fail status for each feature. In this dataset, the table covers 10 columns: CreditScore, Geography, Gender, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited. Each column reports 0 missing values, 0.0% missingness, and a passing result.

Key insights:

• No missing values detected: All 10 evaluated columns report 0 missing values and 0.0% missingness.
• All features passed the test: Every listed feature received a Pass result, indicating no column exceeded the missing value threshold used in the test.
• Completeness is uniform across variables: Missingness results are identical across predictor fields and the Exited target column, with no variation in completeness across the dataset.

The test results show complete observed data coverage across all evaluated fields in this dataset. No feature-level missingness was identified, and no column breached the test threshold. The completeness profile is fully uniform across the assessed predictors and target variable.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

The Skewness test evaluates asymmetry in the distribution of numerical variables by comparing each column’s skewness value against the configured maximum threshold of 1. The results table reports skewness values and pass/fail outcomes for eight numeric columns in the dataset. Observed skewness values range from -0.8646 to 1.2187, with seven columns passing the test and one column failing. The reported results cover CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited.

Key insights:

• Single threshold breach observed: NumOfProducts is the only column that fails the test, with skewness of 1.2187, exceeding the threshold of 1.
• Most variables show limited asymmetry: Seven of eight numeric columns pass the test, with skewness values within the acceptable range defined by the test.
• Negative skewness is present but contained: HasCrCard shows the largest negative skewness at -0.8646, followed by Balance at -0.2816 and CreditScore at -0.1178; each remains within the threshold and receives a pass result.
• Several variables are near symmetric: Exited has skewness of 0.0, while EstimatedSalary (−0.0232) and Tenure (0.0320) are also very close to zero.

The results indicate that skewness is limited across most numeric inputs in the dataset, with only one variable exceeding the test threshold. The clearest concentration of asymmetry appears in NumOfProducts, while the remaining columns exhibit either modest negative or positive skewness or are close to symmetric. Overall, the test outcome reflects predominantly acceptable distributional shape under the specified skewness criterion.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.1178	Pass
Tenure	0.0320	Pass
Balance	-0.2816	Pass
NumOfProducts	1.2187	Fail
HasCrCard	-0.8646	Pass
IsActiveMember	0.1565	Pass
EstimatedSalary	-0.0232	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

The UniqueRows test evaluates dataset diversity by measuring the number and percentage of distinct values observed in each column relative to the total row count. The results are reported at the column level, showing both unique value counts and unique value percentages alongside a pass/fail outcome. Across the evaluated fields, unique value percentages range from 0.0619% to 100.0%, with three columns passing the test and seven columns failing it. The highest observed uniqueness is in EstimatedSalary, followed by Balance and CreditScore, while the remaining columns show comparatively low distinct value counts.

Key insights:

• Uniqueness is concentrated in three columns: EstimatedSalary records 3,232 unique values and 100.0% uniqueness, Balance records 2,211 unique values and 68.4097%, and CreditScore records 438 unique values and 13.552%. These are the only columns marked as passing.
• Most columns have very low uniqueness: Seven columns fail the test, with uniqueness percentages below 0.5% in all cases. These include Geography (0.0928%), Gender (0.0619%), Tenure (0.3403%), NumOfProducts (0.1238%), HasCrCard (0.0619%), IsActiveMember (0.0619%), and Exited (0.0619%).
• Several fields are binary: Gender, HasCrCard, IsActiveMember, and Exited each contain 2 unique values, corresponding to 0.0619% unique values. This produces identical uniqueness percentages and identical fail outcomes across these columns.
• Distinct-value counts vary substantially across features: Unique value counts span from 2 to 3,232 across the tested columns. The spread between high-cardinality fields (EstimatedSalary, Balance) and low-cardinality fields (Gender, HasCrCard, IsActiveMember, Exited) is pronounced in both absolute counts and percentage terms.

Overall, the test result shows a mixed uniqueness profile across the dataset. High uniqueness is observed in EstimatedSalary, Balance, and CreditScore, while the majority of evaluated columns exhibit very limited distinct-value counts and fail the test threshold. The observed diversity is therefore concentrated in a small subset of variables rather than distributed broadly across all columns.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	438	13.5520	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2211	68.4097	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

The TooManyZeroValues test evaluates numerical columns for an excessive proportion of zero values relative to the total row count. The results table reports, for each evaluated variable, the row count, the number of zero values, the percentage of zero values, and the pass/fail outcome against the configured threshold. Four numerical variables are listed in the output, and each is shown as failing the test. Reported zero-value percentages range from 4.2389% to 53.8985%.

Key insights:

• All evaluated variables failed: Tenure, Balance, HasCrCard, and IsActiveMember each exceeded the test threshold and are marked as Fail in the results.
• IsActiveMember has the highest zero concentration: IsActiveMember contains 1,742 zero values out of 3,232 rows, corresponding to 53.8985%, which is the highest percentage among the evaluated variables.
• Balance and HasCrCard show substantial zero prevalence: Balance records 1,022 zero values (31.6213%) and HasCrCard records 975 zero values (30.1671%), indicating that roughly three out of ten observations are zero in both variables.
• Tenure has the lowest reported zero share: Tenure contains 137 zero values out of 3,232 rows, equal to 4.2389%, which is the smallest zero-value percentage in the reported set, though it still exceeds the test threshold.

The test results show that every numerical variable included in this output exceeded the configured zero-value threshold. Zero prevalence varies materially across variables, with the highest concentration observed in IsActiveMember, followed by Balance and HasCrCard, while Tenure has a comparatively lower but still failing proportion. Collectively, the results indicate that zero values are a prominent characteristic across the reported numerical fields.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	137	4.2389	Fail
Balance	3232	1022	31.6213	Fail
HasCrCard	3232	975	30.1671	Fail
IsActiveMember	3232	1742	53.8985	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

The Interquartile Range Outliers Table test identifies numerical observations falling outside the IQR-based bounds and summarizes the count and distribution of those outlier values by variable. The results show detected outliers for two numerical features: CreditScore and NumOfProducts. For each feature, the table reports the total number of outliers, the variable mean, and the minimum, quartile, median, and maximum values among the outlier subset. CreditScore has 9 detected outliers, while NumOfProducts has 44.

Key insights:

• Outliers are concentrated in NumOfProducts: NumOfProducts contains 44 outliers, compared with 9 for CreditScore, making it the feature with the larger concentration of IQR-defined outlying observations in this result set.
• NumOfProducts outliers are uniform: All reported outlier summary statistics for NumOfProducts are equal to 4, indicating every detected outlier for this feature takes the same value.
• CreditScore outliers cluster at the low end: The outlier values for CreditScore range from 350 to 373, with the 25th percentile and median both at 350 and the 75th percentile at 365, showing that detected outliers are concentrated within a narrow lower-value band relative to the feature mean of 648.1399.
• Outlier spreads differ across features: CreditScore exhibits variation within its outlier subset, while NumOfProducts shows no dispersion among outlier values based on identical minimum, quartile, median, and maximum statistics.

The IQR outlier summary indicates that detected outliers are limited to two reported numerical features and are distributed differently across them. NumOfProducts contributes the majority of outlier cases, with all such observations occurring at a single value of 4. CreditScore shows fewer outliers, and those values are concentrated within a relatively narrow low-end range from 350 to 373.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	9	648.1399	350	350.0	350.0	365.0	373
NumOfProducts	44	1.5130	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

The IQROutliersBarPlot test evaluates the distribution of IQR-based outliers across percentile bands for numeric features. The result shows outlier counts for CreditScore and NumOfProducts across the 0–25, 25–50, 50–75, and 75–100 percentile ranges. In CreditScore, outliers appear only in the upper half of the distribution, while in NumOfProducts, all observed outliers are concentrated in the highest percentile band. The bars report counts rather than proportions, allowing direct comparison of where outlier observations are located within each feature’s distribution.

Key insights:

• CreditScore outliers are upper-tail only: CreditScore shows no outliers in the 0–25 or 25–50 percentile bands. Outliers are present in the 50–75 band with a count of 6 and in the 75–100 band with a count of 3.

• NumOfProducts outliers are fully concentrated at the top: NumOfProducts has no outliers in the 0–25, 25–50, or 50–75 percentile bands. All observed outliers occur in the 75–100 percentile band, with a count of 44.

• Outlier concentration differs materially by feature: The total observed outlier count is substantially higher for NumOfProducts than for CreditScore. CreditScore exhibits a split across two upper percentile bands, whereas NumOfProducts shows a single concentrated upper-percentile pattern.

The plotted results indicate that detected IQR outliers are confined to the upper portions of both feature distributions. CreditScore has a limited number of outliers distributed across the 50–75 and 75–100 bands, while NumOfProducts shows a much larger outlier concentration entirely within the highest percentile band. Collectively, the test identifies upper-tail extremity as the observed outlier pattern for both variables, with the concentration markedly stronger for NumOfProducts.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

The Descriptive Statistics test evaluates the distributional characteristics of numerical and categorical variables in the dataset. The results are presented in separate summary tables for seven numerical variables and two categorical variables, covering counts, central tendency, dispersion, percentile ranges, and category concentration. All reported variables have a count of 3,232 observations, with numerical summaries spanning minimum through maximum values and categorical summaries showing the number of unique values, top category, and top-category frequency share.

Key insights:

• Balance shows the widest dispersion: Balance has a mean of 82,278.1433 and a standard deviation of 61,368.5913, the largest spread among the numerical variables. Its 25th percentile is 0.0 while the median is 103,606.0, indicating substantial dispersion across observations.

• EstimatedSalary is broadly distributed: EstimatedSalary ranges from 12.0 to 199,909.0, with a mean of 100,827.2467 and a median of 101,317.0. The interquartile range extends from 51,228.0 to 150,104.0, showing a wide distribution around the center.

• CreditScore is centered near its median: CreditScore has a mean of 648.1399 and a median of 650.0, with values ranging from 350.0 to 850.0. The proximity of mean and median indicates limited difference between these two central tendency measures in the reported summary.

• NumOfProducts is concentrated at lower values: NumOfProducts has a median of 1.0, a 75th percentile of 2.0, and a maximum of 4.0. The mean of 1.513 indicates most observations are concentrated in the lower end of the available range.

• Binary indicators are unevenly distributed: HasCrCard has a mean of 0.6983, with the median and upper percentiles equal to 1.0, while IsActiveMember has a mean of 0.461 and a median of 0.0. These summaries show different class balances across the two binary variables.

• Categorical concentration is moderate: Geography contains 3 unique values, with France as the top category at 1,510 observations (46.72%). Gender contains 2 unique values, with Male as the top category at 1,663 observations (51.45%).

The descriptive statistics show full reported coverage across all summarized variables, with notable variation in spread across the numerical features. Balance and EstimatedSalary exhibit the broadest ranges and dispersion, while CreditScore appears more centered around its median and NumOfProducts is concentrated at lower values. The categorical variables do not show extreme dominance by a single category, although both are led by one category with shares of 46.72% and 51.45%, respectively.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	648.1399	98.0723	350.0	580.0	650.0	716.0	778.0	811.0	850.0
Tenure	3232.0	4.9913	2.9124	0.0	2.0	5.0	8.0	9.0	10.0	10.0
Balance	3232.0	82278.1433	61368.5913	0.0	0.0	103606.0	129375.0	150212.0	163618.0	250898.0
NumOfProducts	3232.0	1.5130	0.6712	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.6983	0.4591	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4610	0.4986	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	100827.2467	57591.1937	12.0	51228.0	101317.0	150104.0	179673.0	189648.0	199909.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1510.0	46.72
Gender	3232.0	2.0	Male	1663.0	51.45

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

The PearsonCorrelationMatrix test evaluates linear dependency between numerical variables by calculating pairwise Pearson correlation coefficients and presenting them in a heat map. The result shows correlations among CreditScore, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, and Exited, with coefficients ranging from -0.18 to 0.14 outside the diagonal. Most pairwise relationships are near zero, and the matrix is symmetric with self-correlations of 1.0 on the diagonal. The largest observed magnitudes are visible between Exited and IsActiveMember (-0.18), Balance and NumOfProducts (-0.17), and Exited and Balance (0.14).

Key insights:

• Correlations are uniformly low: All off-diagonal correlation coefficients fall between -0.18 and 0.14, indicating weak linear relationships across the variables included in the matrix.
• No high-correlation pairs observed: No pairwise correlation approaches the 0.7 absolute-value threshold described for highlighting high dependency in the heat map.
• Exited shows limited linear association: Exited has weak correlations with the other variables, with the largest magnitudes at -0.18 for IsActiveMember, 0.14 for Balance, and -0.06 for NumOfProducts.
• Balance and NumOfProducts are the strongest feature pair: The strongest non-target feature relationship is the correlation between Balance and NumOfProducts at -0.17, which remains weak in magnitude.
• EstimatedSalary is largely uncorrelated: EstimatedSalary exhibits near-zero correlations with the other variables, ranging from -0.03 to 0.03, including -0.01 with Exited.

The correlation structure indicates a dataset with minimal linear dependency among the numerical variables assessed. No material pairwise redundancy is evident from the Pearson coefficients, as all observed relationships remain weak and substantially below the high-correlation threshold referenced in the test methodology. The result therefore reflects a broadly low-collinearity feature set within the variables shown in the matrix.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or multicollinearity. The result table lists the 10 strongest feature-pair correlations returned by the test, along with each Pearson coefficient and its Pass/Fail status under the configured absolute threshold of 0.3. In this run, all reported coefficients are below the threshold, and the displayed relationships range from -0.1837 to 0.1429 in magnitude. The table includes both positive and negative associations among predictor-target and predictor-predictor pairs.

Key insights:

• No correlations exceed threshold: All 10 reported feature pairs are marked Pass, with absolute correlation values remaining below the 0.3 threshold used by the test.
• Strongest observed relationship is weak: The largest absolute correlation is between IsActiveMember and Exited at -0.1837, which remains well below the configured cutoff.
• Top feature-pair associations are limited: Among the highest-ranked pairs, Balance shows modest relationships with NumOfProducts (-0.1697), Exited (0.1429), HasCrCard (-0.0510), and CreditScore (0.0346), all within a low correlation range.
• Reported relationships are concentrated near zero: Most coefficients in the table are small in magnitude, including NumOfProducts and Exited (-0.0609), Tenure and IsActiveMember (-0.0476), and HasCrCard and IsActiveMember (-0.0372), indicating limited linear association among the listed pairs.

The reported correlation structure is uniformly low across the top-ranked feature pairs returned by the test. No displayed pair breaches the configured threshold, and the largest observed absolute coefficient remains below 0.2. Collectively, the results indicate that the strongest linear relationships identified in this run are weak in magnitude within the set of reported pairs.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1697	Pass
(Balance, Exited)	0.1429	Pass
(NumOfProducts, Exited)	-0.0609	Pass
(Balance, HasCrCard)	-0.0510	Pass
(Tenure, IsActiveMember)	-0.0476	Pass
(NumOfProducts, IsActiveMember)	0.0441	Pass
(CreditScore, IsActiveMember)	0.0391	Pass
(HasCrCard, IsActiveMember)	-0.0372	Pass
(CreditScore, Balance)	0.0346	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

The High Pearson Correlation test evaluates pairwise linear relationships among features to identify potentially redundant variables or signs of multicollinearity. The result table lists the top correlations observed in the balanced raw dataset, including each feature pair, its Pearson correlation coefficient, and a Pass/Fail outcome based on the configured absolute-threshold value of 0.3. Among the ten reported pairs, coefficient values range from -0.1837 to 0.3379, with one pair exceeding the threshold and the remaining nine falling below it.

Key insights:

• Single threshold breach observed: The pair (Age, Exited) records the largest absolute correlation at 0.3379 and is the only result marked Fail, indicating it exceeds the configured threshold of 0.3.
• Most reported correlations are weak: The other nine listed feature pairs have absolute correlation values at or below 0.1837, with several close to zero, including (NumOfProducts, Exited) at -0.0609, (Balance, HasCrCard) at -0.0510, and (CreditScore, IsActiveMember) at 0.0391.
• Top non-failing relationships remain modest: The largest correlations that still pass are (IsActiveMember, Exited) at -0.1837, (Balance, NumOfProducts) at -0.1697, and (Balance, Exited) at 0.1429, all remaining materially below the threshold.
• Reported relationships include both directions: The table contains both positive and negative coefficients, with the strongest positive value at 0.3379 for (Age, Exited) and the strongest negative value at -0.1837 for (IsActiveMember, Exited).

The reported correlation structure is dominated by low-magnitude linear relationships, with only one feature pair crossing the configured threshold. The most pronounced relationship is between Age and Exited, while all other listed pairs remain below the limit and are substantially smaller in magnitude. Based on the reported top correlations, elevated linear association is concentrated in a single pair rather than broadly distributed across the features shown.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3379	Fail
(IsActiveMember, Exited)	-0.1837	Pass
(Balance, NumOfProducts)	-0.1697	Pass
(Balance, Exited)	0.1429	Pass
(NumOfProducts, Exited)	-0.0609	Pass
(Balance, HasCrCard)	-0.0510	Pass
(Age, NumOfProducts)	-0.0488	Pass
(Tenure, IsActiveMember)	-0.0476	Pass
(NumOfProducts, IsActiveMember)	0.0441	Pass
(CreditScore, IsActiveMember)	0.0391	Pass

2026-05-26 22:05:06,581 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Learn more: Work with test results):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Development under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Running model evaluation tests

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
5169	841	France	Female	3	162701.65	2	1	1	126794.56	0
1144	611	Spain	Male	4	0.00	2	1	0	170950.58	0
5059	598	France	Male	8	114212.60	1	1	1	74322.85	0
20	510	Spain	Female	4	0.00	1	1	0	118913.53	1
3997	769	Germany	Female	7	137239.17	1	1	1	71379.92	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
5169	841	3	162701.65	2	1	1	126794.56	0	False	False	False
1144	611	4	0.00	2	1	0	170950.58	0	False	True	True
5059	598	8	114212.60	1	1	1	74322.85	0	False	False	True
20	510	4	0.00	1	1	0	118913.53	1	False	True	False
3997	769	7	137239.17	1	1	1	71379.92	1	True	False	False

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.
  warnings.warn(
/opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning: Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.
  warnings.warn(

Initialize ValidMind datasets

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

Initialize a ValidMind model

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

• Despite the naming convention, ValidMind model objects can be any type of record you want to test, document, validate, or monitor with the ValidMind Library.
• From classical statistical and machine learning models, to generative and agentic AI systems and more, the ValidMind model object provides a consistent wrapper around your record so it can be passed as a unified input to any ValidMind test or test suite, with results sent directly to the ValidMind Platform.

Initialize your model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2026-05-26 22:05:07,840 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-05-26 22:05:07,841 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-05-26 22:05:07,842 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-05-26 22:05:07,844 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2026-05-26 22:05:07,848 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2026-05-26 22:05:07,849 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2026-05-26 22:05:07,852 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2026-05-26 22:05:07,853 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

The ModelMetadata test evaluates model metadata consistency by comparing core implementation attributes across models. The result is presented as a summary table containing the modeling technique, modeling framework, framework version, and programming language. In this execution, the table reports a single model entry with values for each of these metadata fields.

Key insights:

• Single model entry reported: The result contains one metadata record, indicating that the comparison output for this test run is limited to a single model.
• Metadata fields are populated: The reported entry includes values for all displayed fields: modeling technique, modeling framework, framework version, and programming language.
• Implementation stack is explicitly identified: The model is documented as using the SKlearnModel technique, the sklearn framework, framework version 1.8.0, and Python as the programming language.

The result documents a complete metadata record for one model and provides a clear view of its implementation stack. Because the output includes only a single model entry, the test run functions as a metadata summary for that model rather than a cross-model comparison. The recorded metadata is internally structured and fully populated for the fields shown.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.8.0	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

The DatasetSplit test evaluates the distribution of records across the available model datasets. The result is presented as a table listing each dataset, its absolute size, and its proportion of the total sample. In this run, the table includes training and test datasets, along with the overall total, allowing the split composition to be viewed directly in both count and percentage terms.

Key insights:

• Training data is the majority split: train_dataset_final contains 2,585 records, representing 79.98% of the total 3,232 observations.
• Test data accounts for one-fifth: test_dataset_final contains 647 records, representing 20.02% of the total dataset.
• Two-way split is reflected: The reported results include training and test datasets only, with total records summing to 3,232 and proportions summing to 100%.

The results show that the dataset was partitioned into two observed components, with most records allocated to the training dataset and the remainder allocated to the test dataset. The split is fully reconciled by the reported totals, with 2,585 training records and 647 test records comprising the full population of 3,232 observations.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

The Population Stability Index (PSI) test evaluates the stability of model output distributions by comparing the bin-level distribution of observations between the initial and new datasets. The results are presented across 10 bins with counts, population shares, and bin-level PSI contributions for each dataset, along with a total PSI summary. The initial dataset contains 2,585 observations and the new dataset contains 647 observations. Across bins, the population ratios are closely aligned, and the total PSI reported for the comparison is 0.0112.

Key insights:

• Overall PSI is low: The total PSI is 0.0112, summarizing the aggregate distribution difference across all 10 bins between the initial and new datasets.

• Bin-level differences are small: Individual bin PSI contributions range from 0.0000 to 0.0073. Most bins contribute less than 0.001 to the total PSI, indicating that the aggregate result is not driven by broad-based divergence across bins.

• Largest shift occurs in the highest bin: Bin 9 shows the largest PSI contribution at 0.0073. Its population share increases from 4.3714% in the initial dataset to 6.3369% in the new dataset, representing the most pronounced bin-level change in the comparison.

• Mid-range bins remain closely aligned: Bins 0 through 8 show relatively similar population shares between datasets. For example, bin 1 is 8.8975% versus 8.9645%, and bin 5 is 12.2631% versus 12.5193%, with correspondingly negligible PSI contributions.

The PSI results show a closely matched distribution between the initial and new datasets at the aggregate level, with a total PSI of 0.0112. The observed distributional differences are concentrated primarily in bin 9, while the remaining bins exhibit only minor deviations and very small PSI contributions. Overall, the comparison reflects limited distributional change across the evaluated population bins.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	162	6.2669	42	6.4915	0.0001
1	230	8.8975	58	8.9645	0.0000
2	308	11.9149	71	10.9737	0.0008
3	373	14.4294	90	13.9104	0.0002
4	392	15.1644	92	14.2195	0.0006
5	317	12.2631	81	12.5193	0.0001
6	323	12.4952	75	11.5920	0.0007
7	227	8.7814	64	9.8918	0.0013
8	140	5.4159	33	5.1005	0.0002
9	113	4.3714	41	6.3369	0.0073
Total	2585	100.0000	647	100.0000	0.0112

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

The Confusion Matrix test evaluates classification performance by comparing predicted labels with observed labels and presenting the counts of true positives, true negatives, false positives, and false negatives. In this result, the matrix shows 211 true positives, 203 true negatives, 93 false positives, and 140 false negatives. The plot organizes these counts by true class on the vertical axis and predicted class on the horizontal axis, allowing direct comparison of correct and incorrect classifications across both classes.

Key insights:

• Correct classifications exceed errors: The model records 414 correct classifications in total, comprising 211 true positives and 203 true negatives, compared with 233 total misclassifications.
• False negatives exceed false positives: The number of false negatives is 140, while false positives total 93, indicating more missed positive cases than incorrect positive predictions.
• Positive class detection is mixed: Within the observed positive class, 211 cases are correctly identified and 140 are misclassified as negative, showing that positive cases are not captured uniformly.
• Negative class performance is comparatively stronger: Within the observed negative class, 203 cases are correctly classified and 93 are misclassified as positive, indicating fewer errors for the negative class than for the positive class.

The confusion matrix indicates that the model produces more correct than incorrect classifications overall, with true positives and true negatives both exceeding their corresponding error counts. The error distribution is asymmetric, with false negatives occurring more frequently than false positives. This pattern shows stronger classification performance for observed negative cases than for observed positive cases in the evaluated sample.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

The Classifier Performance test evaluates classification model performance using precision, recall, F1-score, accuracy, and ROC AUC. The results are reported for classes 0 and 1, along with weighted and macro averages across classes. Class-level precision values are 0.6327 for class 0 and 0.6347 for class 1, while recall values are 0.6735 and 0.5921 respectively. Overall accuracy is 0.6337 and ROC AUC is 0.6806.

Key insights:

• Precision is consistent across classes: Precision is closely aligned between class 0 (0.6327) and class 1 (0.6347), indicating similar positive predictive performance across the two classes.
• Recall differs by class: Recall is higher for class 0 at 0.6735 than for class 1 at 0.5921, showing stronger identification of class 0 observations than class 1 observations.
• F1-scores are moderate and close: F1-scores are 0.6525 for class 0 and 0.6127 for class 1, with weighted and macro average F1 values of 0.6330 and 0.6326, indicating similar aggregate performance across averaging methods.
• Overall summary metrics are aligned: Accuracy is 0.6337, matching the weighted average precision and recall at 0.6337, while ROC AUC is 0.6806.

The in-sample performance results show balanced precision across both classes, with a more visible difference in recall favoring class 0. Aggregate performance metrics are tightly grouped, with weighted and macro averages showing little separation. Overall, the reported metrics indicate moderate classification performance in sample, with ROC AUC exceeding the point estimates of accuracy and F1-based summary measures.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6327	0.6735	0.6525
1	0.6347	0.5921	0.6127
Weighted Average	0.6337	0.6337	0.6330
Macro Average	0.6337	0.6328	0.6326

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6337
ROC AUC	0.6806

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

The Classifier Performance test evaluates out-of-sample classification performance using precision, recall, F1-score, accuracy, and ROC AUC. The reported results show class-level metrics for the two classes, along with weighted and macro averages, and overall accuracy and ROC AUC values. Class 0 has precision 0.5918, recall 0.6858, and F1 0.6354, while Class 1 has precision 0.6941, recall 0.6011, and F1 0.6443. Aggregate results show weighted-average precision 0.6473, recall 0.6399, F1 0.6402, accuracy 0.6399, and ROC AUC 0.6821.

Key insights:

• Class-wise precision and recall are imbalanced: Class 0 shows higher recall than precision (0.6858 vs. 0.5918), while Class 1 shows higher precision than recall (0.6941 vs. 0.6011). This indicates different error trade-offs across the two classes.
• F1-scores are similar across classes: The F1-score is 0.6354 for Class 0 and 0.6443 for Class 1. This reflects broadly comparable combined precision-recall performance across the two classes despite the differing balance between those components.
• Aggregate performance is consistent across summary metrics: Weighted-average recall is 0.6399, matching the reported accuracy of 0.6399, and weighted-average F1 is 0.6402. Macro-average values are also close to weighted-average values, with macro precision 0.6430, recall 0.6435, and F1 0.6398.
• ROC AUC exceeds overall accuracy: The ROC AUC is 0.6821 compared with accuracy of 0.6399. This shows stronger ranking performance than the threshold-dependent classification metrics reported at the selected decision threshold.

Overall, the out-of-sample results indicate moderate classification performance, with aggregate precision, recall, F1, and accuracy all near 0.64 and ROC AUC at 0.6821. Performance is broadly similar across classes when measured by F1-score, while the underlying balance between precision and recall differs by class. The closeness of macro and weighted averages indicates limited divergence between class-level and aggregate summaries in this evaluation.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.5918	0.6858	0.6354
1	0.6941	0.6011	0.6443
Weighted Average	0.6473	0.6399	0.6402
Macro Average	0.6430	0.6435	0.6398

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6399
ROC AUC	0.6821

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

The PrecisionRecallCurve test evaluates the trade-off between precision and recall across classification thresholds for a binary classification model. The plotted curve shows precision on the y-axis and recall on the x-axis over the full threshold range. Precision is highest at very low recall, then moves through a broad middle range where values remain near the low-to-mid 0.70s, before declining as recall increases toward 1.0. The figure therefore summarizes how predictive purity changes as the model captures a larger share of positive cases.

Key insights:

• Highest precision occurs at minimal recall: Precision begins near 1.0 at the extreme low-recall end of the curve, indicating the most selective threshold region produces the highest positive prediction purity.
• Mid-range precision is relatively stable: Across approximately 0.15 to 0.50 recall, precision remains broadly clustered around 0.72 to 0.75, showing a comparatively flat trade-off region through the middle of the curve.
• Precision declines at high recall: After roughly 0.50 recall, precision trends downward steadily, falling below 0.70 around the 0.60 recall range and reaching the mid-0.50s as recall approaches 1.0.
• Early-threshold behavior is volatile: At very low recall levels, the curve shows sharp movement, with precision shifting quickly from below 0.60 to above 0.80 before stabilizing into a narrower band.

The precision-recall profile shows a clear threshold-dependent trade-off. The strongest precision is concentrated at very low recall, while the middle section of the curve maintains a relatively stable precision level in the low-to-mid 0.70s. As recall is pushed higher, precision deteriorates progressively, with the lowest precision observed near full recall. Overall, the result indicates a stable mid-range operating region followed by a pronounced loss of precision at higher recall levels.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

The ROCCurve test evaluates binary classification performance by plotting the relationship between true positive rate and false positive rate across classification thresholds and by calculating the area under the ROC curve (AUC). The result shows the ROC curve for log_reg_model_v1 on test_dataset_final, alongside the random-classification reference line. The plotted curve remains above the random baseline across most of the false positive rate range, and the chart reports an AUC value of 0.68 for the model compared with 0.50 for the random benchmark.

Key insights:

• AUC exceeds random baseline: The model records an AUC of 0.68, which is above the 0.50 reference line shown for random classification and indicates discrimination beyond chance level on the test dataset.
• ROC curve stays above baseline: The ROC curve lies above the diagonal random line for nearly the full range of thresholds displayed, showing higher true positive rates than the random benchmark at comparable false positive rates.
• Discrimination is moderate in shape: The curve rises progressively rather than sharply toward the upper-left corner, reflecting moderate separation between the two classes rather than near-perfect classification behavior.
• Higher TPR requires higher FPR: The plot shows true positive rates approaching the upper range only as false positive rate also increases substantially, indicating that stronger sensitivity is associated with increased false positive incidence.

Overall, the ROC result shows that log_reg_model_v1 demonstrates positive discriminatory power on test_dataset_final, with an AUC of 0.68 and a curve consistently above the random reference line. At the same time, the curve shape indicates that gains in true positive rate are gradual across thresholds rather than concentrated in a strongly convex upper-left profile. This pattern reflects measurable but limited class separation in the evaluated test sample.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

The TrainingTestDegradation test evaluates whether performance degradation between the training and test datasets remains within the predefined threshold across classification metrics. The results are reported separately for classes 1 and 0 using precision, recall, and F1-score, with train and test scores shown alongside the calculated degradation percentage and pass/fail outcome. In this run, all six metric evaluations are marked as Pass, and degradation values range from -9.3476% to 6.4645%.

Key insights:

• All metric checks passed: Each evaluated metric for both classes is marked Pass. No degradation value exceeds the 10% threshold defined for the test.

• Class 1 metrics improved on test data: For class 1, all three metrics show negative degradation, indicating higher scores on the test dataset than on the training dataset. Precision increases from 0.6347 to 0.6941, recall from 0.5921 to 0.6011, and F1-score from 0.6127 to 0.6443.

• Largest class 1 change is precision: The most pronounced shift for class 1 is precision, with degradation of -9.3476%. This corresponds to a test score increase of 0.0594 relative to training.

• Class 0 results are mixed but within threshold: For class 0, precision declines from 0.6327 to 0.5918 and F1-score declines from 0.6525 to 0.6354, while recall increases from 0.6735 to 0.6858. The positive degradation values for precision and F1-score are 6.4645% and 2.6222%, respectively, both remaining below the threshold.

The results indicate limited train-to-test performance variation across the evaluated classification metrics. Class 1 shows uniformly higher test performance than training performance, while class 0 exhibits modest declines in precision and F1-score offset by a small recall increase. Across both classes, all observed degradation values remain within the specified acceptance limit, and no individual metric fails the test.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6347	0.6941	-9.3476	Pass
1	Recall	0.5921	0.6011	-1.5276	Pass
1	F1-Score	0.6127	0.6443	-5.1570	Pass
0	Precision	0.6327	0.5918	6.4645	Pass
0	Recall	0.6735	0.6858	-1.8302	Pass
0	F1-Score	0.6525	0.6354	2.6222	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

The Minimum Accuracy test evaluates whether the model’s prediction accuracy meets or exceeds a predefined threshold on the evaluated dataset. The result table reports an accuracy score of 0.6399 against a threshold of 0.7, along with the overall test outcome. This output presents the direct comparison between the observed classification accuracy and the minimum required level defined for the test.

Key insights:

• Accuracy is below threshold: The observed accuracy score is 0.6399, which is lower than the specified minimum threshold of 0.7.
• Test result is fail: The test outcome is recorded as Fail based on the comparison of the reported score and threshold.
• Gap to threshold is 0.0601: The measured accuracy falls short of the threshold by 0.0601, quantifying the extent of the miss relative to the test criterion.

The result shows that the model did not satisfy the minimum accuracy criterion in this evaluation. The recorded score of 0.6399 remained below the threshold of 0.7, and the test therefore returned a Fail outcome. Collectively, the result and score-to-threshold comparison indicate that overall classification accuracy on the evaluated dataset was not sufficient to pass this test.

Tables

Score	Threshold	Pass/Fail
0.6399	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

The MinimumF1Score test evaluates whether the model’s F1 score on the validation dataset meets a predefined minimum threshold. The result table reports a validation F1 score of 0.6443 alongside a threshold of 0.5, with an overall test outcome of Pass. This result presents the model’s observed balance between precision and recall as measured against the configured minimum acceptance level.

Key insights:

• F1 score exceeds threshold: The observed validation F1 score is 0.6443, which is above the minimum threshold of 0.5 by 0.1443.
• Test outcome is pass: The reported Pass/Fail status is Pass, indicating the validation F1 score met the configured test criterion.
• Measured precision-recall balance is above minimum: Because the test is based on F1 score, the reported result shows that the model’s combined precision-recall performance on the validation set is above the established minimum level used in this assessment.

The test result shows that the model achieved a validation F1 score of 0.6443 against a threshold of 0.5, resulting in a passing outcome. Collectively, the result and supporting values indicate that the model satisfied the minimum criterion defined for this precision-recall balance measure on the validation dataset.

Tables

Score	Threshold	Pass/Fail
0.6443	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

The Minimum ROC AUC Score test evaluates whether the model’s ROC AUC on the validation dataset meets or exceeds a predefined minimum threshold. The reported result includes the computed ROC AUC score, the comparison threshold, and the resulting pass/fail outcome. In this test run, the model recorded a ROC AUC score of 0.6821 against a threshold of 0.5, and the test result is reported as Pass.

Key insights:

• Threshold exceeded: The observed ROC AUC score of 0.6821 is above the minimum threshold of 0.5 used in this test.
• Passing outcome recorded: The test outcome is reported as Pass, reflecting that the measured score satisfied the threshold condition.
• Positive margin over minimum: The score exceeds the threshold by 0.1821, indicating separation from the minimum acceptance level defined for this test.

The result shows that the model met the minimum ROC AUC criterion on the validation dataset under the specified test configuration. The measured score of 0.6821 was above the 0.5 threshold, and the recorded outcome was Pass. Collectively, these results indicate that the model satisfied the performance condition assessed by this test.

Tables

Score	Threshold	Pass/Fail
0.6821	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

The Permutation Feature Importance test evaluates the relative contribution of each input feature by measuring the change in model performance after that feature is randomly permuted. The result is presented as a ranked horizontal bar chart of permutation importances across the model inputs. The chart shows the largest importance values for IsActiveMember and Geography_Germany, followed by Gender_Male, with the remaining features contributing materially less. Several variables appear near zero importance, and one feature displays a slightly negative importance value.

Key insights:

• Importance is concentrated in three features: IsActiveMember and Geography_Germany have the highest permutation importances, both near 0.04 or above, and Gender_Male follows at approximately 0.035. These three variables are separated from the rest of the feature set by a visible gap in importance.

• Mid-tier features have limited contribution: Tenure, EstimatedSalary, and Geography_Spain show positive but much smaller importance values, each below 0.01. Their impact on model performance is materially lower than the top-ranked variables.

• Several variables are near zero importance: NumOfProducts, CreditScore, and HasCrCard have very small positive importance values clustered close to zero. This indicates minimal measured change in performance when these features are permuted in this test.

• Balance shows negative importance: Balance is the only feature with a slightly negative permutation importance. In this result, permuting Balance does not reduce performance and is associated with a marginal performance improvement.

The permutation importance profile shows that model performance is driven primarily by a small subset of features, led by IsActiveMember, Geography_Germany, and Gender_Male. A second group of variables contributes positively but at much lower magnitude, while several inputs have little measurable standalone impact in this test. The slightly negative importance for Balance indicates that its permutation did not degrade model performance in the evaluated sample.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

The SHAPGlobalImportance test evaluates global feature importance by ranking predictors according to their average absolute SHAP values and visualizing how feature values affect model output across observations. The results include a normalized feature importance bar chart and a SHAP summary plot. The bar chart orders features from highest to lowest contribution, while the summary plot shows the direction and spread of SHAP impacts for each feature across the sample. The displayed features are Geography_Germany, IsActiveMember, Gender_Male, Balance, CreditScore, Geography_Spain, NumOfProducts, HasCrCard, Tenure, and EstimatedSalary.

Key insights:

• Geography_Germany is the top driver: Geography_Germany has the highest normalized SHAP importance at 100, exceeding all other features in the ranking. In the summary plot, observations with high feature values are concentrated on the positive SHAP side, while low feature values appear on the negative side.

• IsActiveMember has strong global influence: IsActiveMember is the second most important feature, with normalized importance just below Geography_Germany. Its summary pattern shows a clear separation, with high feature values associated with negative SHAP values and low feature values associated with positive SHAP values.

• Gender_Male ranks among the leading features: Gender_Male is the third most important feature, with normalized importance materially above the remaining variables below it. In the summary plot, high feature values align with negative SHAP values and low feature values align with positive SHAP values.

• Importance drops after the top three features: After Geography_Germany, IsActiveMember, and Gender_Male, normalized importance declines to Balance at roughly one-third of the top feature, followed by smaller contributions from CreditScore and Geography_Spain. The remaining variables show comparatively limited global contribution.

• Balance and CreditScore show distributed effects: Balance displays SHAP values spread mostly from slightly below 0 to clearly positive values, with some negative values as well, indicating variation in impact across observations. CreditScore also shows a broad distribution around zero, with both negative and positive SHAP values present.

• EstimatedSalary and Tenure have minimal impact: EstimatedSalary has the lowest normalized SHAP importance, and Tenure is also near the bottom of the ranking. In the summary plot, both features are concentrated tightly around zero SHAP value, indicating limited contribution relative to the higher-ranked variables.

The SHAP results show that model behavior is primarily driven by Geography_Germany, IsActiveMember, and Gender_Male, with a noticeable reduction in importance for all remaining features. Balance and CreditScore contribute with more distributed case-level effects, while lower-ranked variables such as Tenure and EstimatedSalary remain centered close to zero and have comparatively small global influence. Overall, the importance structure is concentrated in a small set of features, with the summary plot providing directional separation for several of the highest-ranked predictors.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

The WeakspotsDiagnosis test evaluates model performance across feature-space slices by comparing accuracy, precision, recall, and F1 between training and test datasets. The results are reported for multiple binned regions of CreditScore, Balance, EstimatedSalary, Tenure, and selected binary or one-hot encoded features, with thresholds shown in the accompanying plots. Across these slices, the tables show the metric values and record counts for each bin, allowing direct comparison of localized performance levels and train-test differences. Several slices also have very small test sample sizes, which are visible in the record-count column and in the plotted bars.

Key insights:

• Low recall drives several weak regions: Multiple slices show recall as the lowest-performing metric. Examples include Balance in (-250.898, 25089.809] with test recall 0.2921 and F1 = 0.3939, Gender_Male = 1 with test recall 0.3576 and F1 = 0.4576, and IsActiveMember = 1 with test recall 0.3101 and F1 = 0.4188.

• Sparse Balance bins are unstable: Several Balance slices on the test dataset have very low record counts, including 4, 7, 3, and 0 observations. These bins also show highly variable metrics, such as precision/recall/F1 all equal to 0.0 in (25089.809, 50179.618] and accuracy 0.3333 in (200718.472, 225808.281].

• Mid-range Balance performs best: The strongest Balance results are concentrated in the middle bins. Test F1 reaches 0.7527 in (100359.236, 125449.045] and 0.7552 in (125449.045, 150538.854], with corresponding accuracies 0.6993 and 0.6875.

• CreditScore performance varies by band: Test performance for CreditScore is heterogeneous across bins. Test F1 ranges from 0.5532 in (800.0, 850.0] to 0.75 in (450.0, 500.0], while test accuracy ranges from 0.5532 to 0.7222 in the more populated bands and reaches 1.0 in (349.5, 400.0] with only 4 records.

• Tenure shows mixed but moderate variation: Tenure test F1 spans from 0.5625 in (2.0, 3.0] to 0.7632 in (3.0, 4.0]. Test accuracy is similarly variable, ranging from 0.5735 to 0.7429, with no single tenure bin showing uniformly dominant results across all metrics.

• Binary segments reveal asymmetric behavior: For Geography_Germany, the positive slice (0.9, 1.0] has materially higher test recall (0.8649) and F1 (0.7781) than the complement slice (-0.001, 0.1], where recall is 0.4089 and F1 is 0.5092. A similar asymmetry appears for IsActiveMember, where the 0 slice has test F1 = 0.7371 versus 0.4188 for the 1 slice.

• EstimatedSalary is comparatively even: EstimatedSalary slices are more balanced than several other features, with test F1 values between 0.5397 and 0.7647. Most bins remain within a relatively narrow range, although the (99960.45, 119950.224] slice is weaker with test accuracy 0.5672 and F1 = 0.5397.

• NumOfProducts separates strong and weak slices: For NumOfProducts, the test slice (2.8, 3.1] has strong precision 0.9524 and F1 = 0.7692, while (1.9, 2.2] has materially lower precision 0.4231 and F1 = 0.4714. The (3.7, 4.0] slice has only 9 test records and accuracy 0.4444.

Overall, the weak spot analysis shows that model performance is not uniform across the feature space and that the most material differences are concentrated in specific slices rather than being broad-based across all variables. The clearest weak regions are associated with low recall and low F1 in selected Balance, Gender_Male, and IsActiveMember slices, while mid-range Balance bins and some NumOfProducts and Geography_Germany slices perform materially better. The results also show that several of the most extreme metric values occur in bins with very small sample counts, which coincides with greater variability in the observed slice-level metrics.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-250.898, 25089.809]	210	Balance	0.6190	0.6047	0.2921	0.3939	test_dataset_final
(25089.809, 50179.618]	4	Balance	0.5000	0.0000	0.0000	0.0000	test_dataset_final
(50179.618, 75269.427]	28	Balance	0.5000	0.7000	0.3889	0.5000	test_dataset_final
(75269.427, 100359.236]	76	Balance	0.6447	0.7317	0.6522	0.6897	test_dataset_final
(100359.236, 125449.045]	153	Balance	0.6993	0.7692	0.7368	0.7527	test_dataset_final
(125449.045, 150538.854]	112	Balance	0.6875	0.7297	0.7826	0.7552	test_dataset_final
(150538.854, 175628.663]	54	Balance	0.5741	0.5714	0.7143	0.6349	test_dataset_final
(175628.663, 200718.472]	7	Balance	0.4286	0.5000	0.7500	0.6000	test_dataset_final
(200718.472, 225808.281]	3	Balance	0.3333	0.5000	0.5000	0.5000	test_dataset_final
(225808.281, 250898.09]	0	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(-250.898, 25089.809]	816	Balance	0.6532	0.5915	0.3099	0.4067	train_dataset_final
(25089.809, 50179.618]	27	Balance	0.5556	0.6250	0.3571	0.4545	train_dataset_final
(50179.618, 75269.427]	87	Balance	0.5287	0.5714	0.4348	0.4938	train_dataset_final
(75269.427, 100359.236]	296	Balance	0.6182	0.5733	0.6370	0.6035	train_dataset_final
(100359.236, 125449.045]	586	Balance	0.6980	0.7315	0.7717	0.7511	train_dataset_final
(125449.045, 150538.854]	517	Balance	0.5938	0.5969	0.7022	0.6453	train_dataset_final
(150538.854, 175628.663]	184	Balance	0.5815	0.5882	0.6316	0.6091	train_dataset_final
(175628.663, 200718.472]	53	Balance	0.5660	0.6000	0.5357	0.5660	train_dataset_final
(200718.472, 225808.281]	17	Balance	0.4118	0.7000	0.5000	0.5833	train_dataset_final
(225808.281, 250898.09]	2	Balance	0.5000	1.0000	0.5000	0.6667	train_dataset_final
(349.5, 400.0]	4	CreditScore	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(400.0, 450.0]	14	CreditScore	0.7143	0.8000	0.5714	0.6667	test_dataset_final
(450.0, 500.0]	36	CreditScore	0.7222	0.6818	0.8333	0.7500	test_dataset_final
(500.0, 550.0]	54	CreditScore	0.5556	0.6333	0.5938	0.6129	test_dataset_final
(550.0, 600.0]	96	CreditScore	0.6354	0.6316	0.7200	0.6729	test_dataset_final
(600.0, 650.0]	127	CreditScore	0.5748	0.6290	0.5571	0.5909	test_dataset_final
(650.0, 700.0]	123	CreditScore	0.7073	0.6923	0.6429	0.6667	test_dataset_final
(700.0, 750.0]	87	CreditScore	0.6667	0.8333	0.5102	0.6329	test_dataset_final
(750.0, 800.0]	59	CreditScore	0.6610	0.8000	0.5714	0.6667	test_dataset_final
(800.0, 850.0]	47	CreditScore	0.5532	0.7647	0.4333	0.5532	test_dataset_final
(349.5, 400.0]	10	CreditScore	0.5000	1.0000	0.5000	0.6667	train_dataset_final
(400.0, 450.0]	47	CreditScore	0.6596	0.7200	0.6667	0.6923	train_dataset_final
(450.0, 500.0]	113	CreditScore	0.5841	0.5672	0.6786	0.6179	train_dataset_final
(500.0, 550.0]	278	CreditScore	0.7014	0.7194	0.6944	0.7067	train_dataset_final
(550.0, 600.0]	355	CreditScore	0.6028	0.6154	0.6452	0.6299	train_dataset_final
(600.0, 650.0]	484	CreditScore	0.6322	0.6121	0.5796	0.5955	train_dataset_final
(650.0, 700.0]	498	CreditScore	0.6265	0.6316	0.5477	0.5867	train_dataset_final
(700.0, 750.0]	405	CreditScore	0.6148	0.6159	0.5206	0.5642	train_dataset_final
(750.0, 800.0]	246	CreditScore	0.6626	0.6436	0.5804	0.6103	train_dataset_final
(800.0, 850.0]	149	CreditScore	0.6510	0.6393	0.5652	0.6000	train_dataset_final
(-188.318, 20001.354]	60	EstimatedSalary	0.7333	0.8667	0.6842	0.7647	test_dataset_final
(20001.354, 39991.128]	66	EstimatedSalary	0.6364	0.6286	0.6667	0.6471	test_dataset_final
(39991.128, 59980.902]	65	EstimatedSalary	0.6462	0.7097	0.6111	0.6567	test_dataset_final
(59980.902, 79970.676]	72	EstimatedSalary	0.6389	0.7857	0.5238	0.6286	test_dataset_final
(79970.676, 99960.45]	69	EstimatedSalary	0.7391	0.7500	0.7059	0.7273	test_dataset_final
(99960.45, 119950.224]	67	EstimatedSalary	0.5672	0.5152	0.5667	0.5397	test_dataset_final
(119950.224, 139939.998]	65	EstimatedSalary	0.6154	0.6333	0.5758	0.6032	test_dataset_final
(139939.998, 159929.772]	67	EstimatedSalary	0.6567	0.6667	0.6061	0.6349	test_dataset_final
(159929.772, 179919.546]	67	EstimatedSalary	0.5522	0.7333	0.5000	0.5946	test_dataset_final
(179919.546, 199909.32]	49	EstimatedSalary	0.6122	0.6800	0.6071	0.6415	test_dataset_final
(-188.318, 20001.354]	247	EstimatedSalary	0.6559	0.6833	0.6357	0.6586	train_dataset_final
(20001.354, 39991.128]	245	EstimatedSalary	0.6163	0.6271	0.5968	0.6116	train_dataset_final
(39991.128, 59980.902]	259	EstimatedSalary	0.6486	0.5984	0.6348	0.6160	train_dataset_final
(59980.902, 79970.676]	261	EstimatedSalary	0.6437	0.6667	0.6015	0.6324	train_dataset_final
(79970.676, 99960.45]	252	EstimatedSalary	0.5794	0.5528	0.5714	0.5620	train_dataset_final
(99960.45, 119950.224]	250	EstimatedSalary	0.6000	0.6404	0.5530	0.5935	train_dataset_final
(119950.224, 139939.998]	272	EstimatedSalary	0.6287	0.6207	0.5581	0.5878	train_dataset_final
(139939.998, 159929.772]	257	EstimatedSalary	0.6809	0.6792	0.6000	0.6372	train_dataset_final
(159929.772, 179919.546]	276	EstimatedSalary	0.6594	0.6723	0.5926	0.6299	train_dataset_final
(179919.546, 199909.32]	266	EstimatedSalary	0.6203	0.6148	0.5814	0.5976	train_dataset_final
(-0.001, 0.1]	318	Gender_Male	0.6698	0.7169	0.7850	0.7494	test_dataset_final
(0.9, 1.0]	329	Gender_Male	0.6109	0.6353	0.3576	0.4576	test_dataset_final
(-0.001, 0.1]	1251	Gender_Male	0.6267	0.6388	0.7572	0.6930	train_dataset_final
(0.9, 1.0]	1334	Gender_Male	0.6402	0.6254	0.3902	0.4805	train_dataset_final
(-0.001, 0.1]	434	Geography_Germany	0.6313	0.6748	0.4089	0.5092	test_dataset_final
(0.9, 1.0]	213	Geography_Germany	0.6573	0.7072	0.8649	0.7781	test_dataset_final
(-0.001, 0.1]	1799	Geography_Germany	0.6187	0.5703	0.3900	0.4632	train_dataset_final
(0.9, 1.0]	786	Geography_Germany	0.6679	0.6853	0.8953	0.7763	train_dataset_final
(-0.001, 0.1]	512	Geography_Spain	0.6426	0.7047	0.6237	0.6617	test_dataset_final
(0.9, 1.0]	135	Geography_Spain	0.6296	0.6400	0.5000	0.5614	test_dataset_final
(-0.001, 0.1]	1997	Geography_Spain	0.6380	0.6426	0.6226	0.6324	train_dataset_final
(0.9, 1.0]	588	Geography_Spain	0.6190	0.5991	0.4774	0.5314	train_dataset_final
(-0.001, 0.1]	202	HasCrCard	0.6535	0.6765	0.6509	0.6635	test_dataset_final
(0.9, 1.0]	445	HasCrCard	0.6337	0.7030	0.5796	0.6353	test_dataset_final
(-0.001, 0.1]	773	HasCrCard	0.6132	0.6193	0.5954	0.6071	train_dataset_final
(0.9, 1.0]	1812	HasCrCard	0.6424	0.6419	0.5906	0.6152	train_dataset_final
(-0.001, 0.1]	357	IsActiveMember	0.6583	0.7066	0.7703	0.7371	test_dataset_final
(0.9, 1.0]	290	IsActiveMember	0.6172	0.6452	0.3101	0.4188	test_dataset_final
(-0.001, 0.1]	1385	IsActiveMember	0.6166	0.6406	0.7604	0.6954	train_dataset_final
(0.9, 1.0]	1200	IsActiveMember	0.6533	0.6111	0.3056	0.4074	train_dataset_final
(0.997, 1.3]	381	NumOfProducts	0.6273	0.7662	0.6185	0.6844	test_dataset_final
(1.9, 2.2]	221	NumOfProducts	0.6652	0.4231	0.5323	0.4714	test_dataset_final
(2.8, 3.1]	36	NumOfProducts	0.6667	0.9524	0.6452	0.7692	test_dataset_final
(3.7, 4.0]	9	NumOfProducts	0.4444	1.0000	0.4444	0.6154	test_dataset_final
(0.997, 1.3]	1473	NumOfProducts	0.6191	0.7022	0.6193	0.6581	train_dataset_final
(1.9, 2.2]	921	NumOfProducts	0.6851	0.3730	0.5694	0.4508	train_dataset_final
(2.8, 3.1]	156	NumOfProducts	0.5192	0.9744	0.5101	0.6696	train_dataset_final
(3.7, 4.0]	35	NumOfProducts	0.4000	1.0000	0.4000	0.5714	train_dataset_final
(-0.01, 1.0]	90	Tenure	0.6444	0.7838	0.5472	0.6444	test_dataset_final
(1.0, 2.0]	68	Tenure	0.5735	0.5882	0.5714	0.5797	test_dataset_final
(2.0, 3.0]	66	Tenure	0.5758	0.6429	0.5000	0.5625	test_dataset_final
(3.0, 4.0]	70	Tenure	0.7429	0.8529	0.6905	0.7632	test_dataset_final
(4.0, 5.0]	66	Tenure	0.6667	0.6667	0.7059	0.6857	test_dataset_final
(5.0, 6.0]	58	Tenure	0.6207	0.7273	0.5000	0.5926	test_dataset_final
(6.0, 7.0]	61	Tenure	0.6393	0.6552	0.6129	0.6333	test_dataset_final
(7.0, 8.0]	69	Tenure	0.6377	0.5758	0.6333	0.6032	test_dataset_final
(8.0, 9.0]	64	Tenure	0.6094	0.7000	0.5676	0.6269	test_dataset_final
(9.0, 10.0]	35	Tenure	0.7143	0.7619	0.7619	0.7619	test_dataset_final
(-0.01, 1.0]	394	Tenure	0.6421	0.6408	0.6633	0.6519	train_dataset_final
(1.0, 2.0]	265	Tenure	0.6000	0.5691	0.5691	0.5691	train_dataset_final
(2.0, 3.0]	256	Tenure	0.6523	0.7025	0.6159	0.6564	train_dataset_final
(3.0, 4.0]	272	Tenure	0.6176	0.6147	0.5194	0.5630	train_dataset_final
(4.0, 5.0]	250	Tenure	0.6520	0.6585	0.6429	0.6506	train_dataset_final
(5.0, 6.0]	247	Tenure	0.6316	0.6372	0.5902	0.6128	train_dataset_final
(6.0, 7.0]	258	Tenure	0.6240	0.5429	0.5377	0.5403	train_dataset_final
(7.0, 8.0]	260	Tenure	0.6462	0.6759	0.5615	0.6134	train_dataset_final
(8.0, 9.0]	246	Tenure	0.6585	0.6870	0.6220	0.6529	train_dataset_final
(9.0, 10.0]	137	Tenure	0.5912	0.5789	0.5077	0.5410	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

The Overfit Diagnosis test evaluates differences between training and test-set model performance across feature slices to identify regions with elevated train-test AUC gaps. The results table and accompanying plots report slices where the observed gap exceeds the configured threshold of 0.04. Overfit regions were identified across CreditScore, Tenure, Balance, NumOfProducts, EstimatedSalary, and Geography_Germany, with the largest gaps concentrated in a small number of slices. Reported training and test record counts for each slice provide the sample context for these observed differences.

Key insights:

• Largest gap occurs in Balance: The Balance slice (25089.809, 50179.618] shows the largest reported gap at 0.6319, with training AUC 0.6319 and test AUC 0.0. This slice has 27 training records and 4 test records.

• Additional Balance slices exceed threshold: Three other Balance slices are flagged: (175628.663, 200718.472] with gap 0.1676, (200718.472, 225808.281] with gap 0.1429, and (100359.236, 125449.045] with gap 0.0439. These results indicate multiple overfit regions within the Balance feature.

• CreditScore shows several elevated gaps: CreditScore contains three flagged slices, with the largest in (500.0, 550.0] at gap 0.1761 (training AUC 0.7428, test AUC 0.5668). Additional flagged slices are (600.0, 650.0] at 0.0616 and (800.0, 850.0] at 0.0639.

• EstimatedSalary contains two material gaps: EstimatedSalary slice (159929.772, 179919.546] has a gap of 0.1422, and (119950.224, 139939.998] has a gap of 0.0763. A third slice, (39991.128, 59980.902], is also flagged at 0.0471.

• Threshold exceedances are limited in some features: Tenure has two flagged slices with gaps of 0.0482 and 0.0442, NumOfProducts has one flagged slice at 0.0528, and Geography_Germany has one flagged slice at 0.0588. No table entries are reported for HasCrCard, IsActiveMember, Geography_Spain, or Gender_Male.

The test identifies a set of localized train-test AUC differences above the 0.04 threshold rather than a uniform pattern across all analyzed features. The most pronounced gaps are concentrated in selected Balance, CreditScore, and EstimatedSalary slices, while Tenure, NumOfProducts, and Geography_Germany show smaller but still above-threshold differences. Overall, the result indicates that elevated train-test performance divergence is present in specific feature segments, with the largest observed values occurring in a small subset of slices.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(500.0, 550.0]	278	54	0.7428	0.5668	0.1761
CreditScore	(600.0, 650.0]	484	127	0.6717	0.6100	0.0616
CreditScore	(800.0, 850.0]	149	47	0.7149	0.6510	0.0639
Tenure	(2.0, 3.0]	256	66	0.6834	0.6352	0.0482
Tenure	(8.0, 9.0]	246	64	0.6969	0.6527	0.0442
Balance	(25089.809, 50179.618]	27	4	0.6319	0.0000	0.6319
Balance	(100359.236, 125449.045]	586	153	0.7441	0.7002	0.0439
Balance	(175628.663, 200718.472]	53	7	0.5843	0.4167	0.1676
Balance	(200718.472, 225808.281]	17	3	0.1429	0.0000	0.1429
NumOfProducts	(1.9, 2.2]	921	221	0.7070	0.6542	0.0528
EstimatedSalary	(39991.128, 59980.902]	259	65	0.6841	0.6370	0.0471
EstimatedSalary	(119950.224, 139939.998]	272	65	0.7013	0.6250	0.0763
EstimatedSalary	(159929.772, 179919.546]	276	67	0.7144	0.5721	0.1422
Geography_Germany	(0.9, 1.0]	786	213	0.6375	0.5787	0.0588

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

The Robustness Diagnosis test evaluates model robustness by measuring AUC decay after adding Gaussian noise to numerical input features at increasing perturbation sizes. The results report baseline and perturbed AUC values for both train_dataset_final and test_dataset_final, along with performance decay and pass status at each perturbation level from 0.0 to 0.5. Baseline AUC is 0.6806 for the training dataset and 0.6821 for the test dataset, and subsequent rows show how these values change as noise intensity increases. The accompanying plot visualizes the same trajectory and includes the perturbed feature list and threshold reference lines.

Key insights:

• Baseline train and test AUC are aligned: Baseline AUC values are 0.6806 for train_dataset_final and 0.6821 for test_dataset_final, indicating very similar starting performance before perturbation is introduced.

• Performance declines as noise increases: Relative to baseline, AUC generally decreases as perturbation size increases across both datasets. By perturbation size 0.4, AUC declines to 0.6658 on the training dataset and 0.6613 on the test dataset, corresponding to performance decay values of 0.0149 and 0.0208.

• Largest observed decay occurs on test at 0.4: The maximum reported performance decay is 0.0208 for test_dataset_final at perturbation size 0.4, where AUC falls from 0.6821 to 0.6613.

• Train deterioration is strongest at 0.5: On train_dataset_final, the lowest AUC observed is 0.6546 at perturbation size 0.5, with a corresponding performance decay of 0.0260.

• Test performance is not strictly monotonic: On test_dataset_final, AUC decreases through perturbation size 0.4 to 0.6613, then rises to 0.6704 at perturbation size 0.5, reducing the reported decay from 0.0208 to 0.0117.

• All perturbation levels passed: Every evaluated row is marked Passed = true, including baseline and all perturbation sizes for both training and test datasets.

Across the evaluated perturbation range, the model’s AUC remains close to baseline levels while showing moderate degradation under higher noise levels. The strongest deterioration is observed at perturbation size 0.4 on the test dataset and at 0.5 on the training dataset, while the test trajectory shows partial recovery at the highest perturbation level. All evaluated scenarios are recorded as passed, and the train and test series remain broadly similar in magnitude throughout the robustness assessment.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6806	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6821	0.0000	True
0.1	train_dataset_final	2585	0.6807	-0.0000	True
0.1	test_dataset_final	647	0.6812	0.0008	True
0.2	train_dataset_final	2585	0.6737	0.0069	True
0.2	test_dataset_final	647	0.6754	0.0067	True
0.3	train_dataset_final	2585	0.6748	0.0058	True
0.3	test_dataset_final	647	0.6742	0.0079	True
0.4	train_dataset_final	2585	0.6658	0.0149	True
0.4	test_dataset_final	647	0.6613	0.0208	True
0.5	train_dataset_final	2585	0.6546	0.0260	True
0.5	test_dataset_final	647	0.6704	0.0117	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests

---

Copyright © 2023-2026 ValidMind Inc. All rights reserved.
Refer to LICENSE for details.
SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial