validmind.tests

List all tests in the tests directory.

Arguments:

• filter (str, optional): Find tests where the ID, tasks or tags match the filter string. Defaults to None.
• task (str, optional): Find tests that match the task. Can be used to narrow down matches from the filter string. Defaults to None.
• tags (list, optional): Find tests that match list of tags. Can be used to narrow down matches from the filter string. Defaults to None.
• pretty (bool, optional): If True, returns a pandas DataFrame with a formatted table. Defaults to True.
• truncate (bool, optional): If True, truncates the test description to the first line. Defaults to True. (only used if pretty=True)

Returns:

list or pandas.DataFrame: A list of all tests or a formatted table.

Load a test by test ID

Test IDs are in the format namespace.path_to_module.TestClassOrFuncName[:result_id]. The result ID is optional and is used to distinguish between multiple results from the running the same test.

Arguments:

• test_id (str): The test ID in the format namespace.path_to_module.TestName[:result_id]
• reload (bool, optional): Whether to reload the test module. Defaults to False.

Get or show details about the test

This function can be used to see test details including the test name, description, required inputs and default params. It can also be used to get a dictionary of the above information for programmatic use.

Arguments:

• test_id (str, optional): The test ID. Defaults to None.
• raw (bool, optional): If True, returns a dictionary with the test details. Defaults to False.

Run a test by test ID

Arguments:

• test_id (TestID, optional): The test ID to run. Not required if unit_metrics is provided.
• params (dict, optional): A dictionary of parameters to pass into the test. Params are used to customize the test behavior and are specific to each test. See the test details for more information on the available parameters. Defaults to None.
• inputs (Dict[str, Any], optional): A dictionary of test inputs to pass into the test. Inputs are either models or datasets that have been initialized using vm.init_model() or vm.init_dataset(). Defaults to None.
• input_grid (Union[Dict[str, List[Any]], List[Dict[str, Any]]], optional): To run a comparison test, provide either a dictionary of inputs where the keys are the input names and the values are lists of different inputs, or a list of dictionaries where each dictionary is a set of inputs to run the test with. This will run the test multiple times with different sets of inputs and then combine the results into a single output. When passing a dictionary, the grid will be created by taking the Cartesian product of the input lists. Its simply a more convenient way of forming the input grid as opposed to passing a list of all possible combinations. Defaults to None.
• name (str, optional): The name of the test (used to create a composite metric out of multiple unit metrics) - required when running multiple unit metrics
• unit_metrics (list, optional): A list of unit metric IDs to run as a composite metric - required when running multiple unit metrics
• output_template (str, optional): A jinja2 html template to customize the output of the test. Defaults to None.
• show (bool, optional): Whether to display the results. Defaults to True.
• **kwargs: Keyword inputs to pass into the test (same as inputs but as keyword args instead of a dictionary):
  • dataset: A validmind Dataset object or a Pandas DataFrame
  • model: A model to use for the test
  • models: A list of models to use for the test
  • dataset: A validmind Dataset object or a Pandas DataFrame

Register an external test provider

Arguments:

• namespace (str): The namespace of the test provider
• test_provider (TestProvider): The test provider

Exception raised when an error occurs while loading a test

Test providers in ValidMind are responsible for loading tests from different sources, such as local files, databases, or remote services. The LocalTestProvider specifically loads tests from the local file system.

To use the LocalTestProvider, you need to provide the root_folder, which is the root directory for local tests. The test_id is a combination of the namespace (set when registering the test provider) and the path to the test class module, where slashes are replaced by dots and the .py extension is left out.

Example usage:

# Create an instance of LocalTestProvider with the root folder
test_provider = LocalTestProvider("/path/to/tests/folder")

# Register the test provider with a namespace
register_test_provider("my_namespace", test_provider)

# Load a test using the test_id (namespace + path to test class module)
test = test_provider.load_test("my_namespace.my_test_class")
# full path to the test class module is /path/to/tests/folder/my_test_class.py

Attributes:

• root_folder (str): The root directory for local tests.

Protocol for user-defined test providers

List unique tags from all test classes.

List unique tasks from all test classes.

List all task types and their associated tags, with one row per task type and all tags for a task type in one row.

Returns:

pandas.DataFrame: A DataFrame with 'Task Type' and concatenated 'Tags'.

Decorator for creating and registering metrics with the ValidMind framework.

Creates a metric object and registers it with ValidMind under the provided ID. If no ID is provided, the function name will be used as to build one. So if the function name is my_metric, the metric will be registered under the ID validmind.custom_metrics.my_metric.

This decorator works by creating a new Metric class will be created whose run method calls the decorated function. This function should take as arguments the inputs it requires (dataset, datasets, model, models) followed by any parameters. It can return any number of the following types:

• Table: Either a list of dictionaries or a pandas DataFrame
• Plot: Either a matplotlib figure or a plotly figure
• Scalar: A single number or string

The function may also include a docstring. This docstring will be used and logged as the metric's description.

Arguments:

• func: The function to decorate
• test_id: The identifier for the metric. If not provided, the function name is used.

Returns:

The decorated function.

DEPRECATED, use @vm.test instead

Decorator for specifying tags for a metric.

Arguments:

• *tags: The tags to apply to the metric.

Decorator for specifying the task types that a metric is designed for.

Arguments:

• *tasks: The task types that the metric is designed for.