How to run tests and test suites

Published

June 2, 2026

​ValidMind provides many built-in tests and test suites, which help you produce documentation during stages of your risk management lifecycle, where you need to ensure that your work satisfies regulatory and risk management requirements.

ImportantQuick reference

Test ID
Each test has a unique identifier like validmind.data_validation.ClassImbalance. Use vm.tests.list_tests() to browse available tests.
Result ID
When running the same test multiple times with different parameters, add a :suffix to distinguish results, such as validmind.data_validation.ClassImbalance:custom_threshold.
Inputs
The datasets and records (models) your test analyzes. Pass these via inputs={}:
  • dataset — A VM dataset object
  • model — A VM model object, which can be any type of record
  • datasets / models — For tests comparing multiple datasets or models
Parameters
Configuration that customizes how the test runs. Pass these via params={}:
  • Thresholds, such as min_percent_threshold, disparity_tolerance
  • Metrics to compute, such as metrics=["fnr", "fpr", "tpr"]
  • Any setting that isn't a dataset or model1
Outputs
Tests return results like tables and plots. Tables can be lists of dictionaries or pandas DataFrames; plots can be matplotlib or plotly figures.
Logging results
Call .log() on a test result to send it to the ValidMind Platform.

Test suite ID
Test suites are collections of tests that run together. Each has an identifier like classifier_full_suite. Use vm.tests.list_test_suites() to browse available test suites.
Inputs
The datasets and records (models) the test suite analyzes. Pass these via inputs={}:
  • dataset — A VM dataset object
  • model — A VM model object, which can be any type of record
  • datasets / models — For suites comparing multiple datasets or models
Config
A dictionary of parameters to pass to the tests in the suite. Use vm.get_test_suite().get_default_config() to see the default configuration for all tests.
Outputs
Test suites return results from all tests — tables and plots. Results are automatically logged to the platform unless send=False.
Logging results
By default, run_test_suite() logs all results automatically. Set send=False to run without logging.

1 Use vm.tests.describe_test("test_id") for programmatic parameter details, or browse the ​ValidMind test sandbox.

Explore tests

Start by exploring the ValidMind Library's available tests and test suites:

When do I use tests and test suites?

While you have the flexibility to decide when to use which ​ValidMind tests, we have identified a few typical scenarios with their own characteristics and needs:

To document and validate your dataset:

To document and validate your record:

  • For binary classification models: use the classifier test suite.
  • For time series models: use the timeseries test suite.

To document a binary classification model and the relevant dataset end-to-end:

Use the classifier_full_suite test suite.

Run tests

Next, learn how to use the ValidMind Library to run tests on datasets and records (models):

Configure judge LLM and judge embeddings
This notebook shows how to configure and validate the default judge LLM and judge embeddings used by the ValidMind Library for LLM-focused tests.
Customize test result descriptions
When you run ValidMind tests, test descriptions are automatically generated with LLM using the test results, the test name, and the static test definitions provided in the test's docstring. While this metadata offers valuable high-level overviews of tests, insights produced by the LLM-based descriptions may not always align with your specific use…
Enable PII detection in tests
Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform.
Dataset Column Filters when Running Tests
To run a test on a dataset but only include certain columns from that dataset, you can use the new columns option. This is done by passing a dictionary for the dataset input for the test instead of the dataset object or dataset input ID directly. This dictionary should have the following keys:
Run tests with multiple datasets
To support running tests that require more than one dataset, ValidMind provides a mechanim that allows you to pass multiple datasets as inputs.
Understand and utilize RawData in ValidMind tests
Test functions in ValidMind can return a special object called RawData, which holds intermediate or unprocessed data produced somewhere in the test logic but not returned as part of the test's visible output, such as in tables or figures.
No matching items

Can I use my own tests?

Absolutely! ​ValidMind supports custom tests that you develop yourself or that are provided by third-party test libraries, also referred to as test providers.

We provide instructions with code examples that you can adapt:

What's next

Learn more about using the other features of the ValidMind Library and browse our code samples by use case: