validmind.vm_models

Models entrypoint

class VMInput(abc.ABC):

Base class for ValidMind Input types

def with_options(self, **kwargs) -> VMInput:

Allows for setting options on the input object that are passed by the user when using the input to run a test or set of tests

To allow options, just override this method in the subclass (see VMDataset) and ensure that it returns a new instance of the input with the specified options set.

Arguments:
  • **kwargs: Arbitrary keyword arguments that will be passed to the input object
Returns:

VMInput: A new instance of the input with the specified options set

class VMDataset(validmind.vm_models.VMInput):

Base class for VM datasets

Child classes should be used to support new dataset types (tensor, polars etc) by converting the user's dataset into a numpy array collecting metadata like column names and then call this (parent) class __init__ method.

This way we can support multiple dataset types but under the hood we only need to work with numpy arrays and pandas dataframes in this class.

Attributes:
  • raw_dataset (np.ndarray): The raw dataset as a NumPy array.
  • input_id (str): Identifier for the dataset.
  • index (np.ndarray): The raw dataset index as a NumPy array.
  • columns (Set[str]): The column names of the dataset.
  • target_column (str): The target column name of the dataset.
  • feature_columns (List[str]): The feature column names of the dataset.
  • feature_columns_numeric (List[str]): The numeric feature column names of the dataset.
  • feature_columns_categorical (List[str]): The categorical feature column names of the dataset.
  • text_column (str): The text column name of the dataset for NLP tasks.
  • target_class_labels (Dict): The class labels for the target columns.
  • df (pd.DataFrame): The dataset as a pandas DataFrame.
  • extra_columns (Dict): Extra columns to include in the dataset.
VMDataset( raw_dataset: numpy.ndarray, input_id: str = None, model: VMModel = None, index: numpy.ndarray = None, index_name: str = None, date_time_index: bool = False, columns: list = None, target_column: str = None, feature_columns: list = None, text_column: str = None, extra_columns: dict = None, target_class_labels: dict = None)

Initializes a VMDataset instance.

Arguments:
  • raw_dataset (np.ndarray): The raw dataset as a NumPy array.
  • input_id (str): Identifier for the dataset.
  • model (VMModel): Model associated with the dataset.
  • index (np.ndarray): The raw dataset index as a NumPy array.
  • index_name (str): The raw dataset index name as a NumPy array.
  • date_time_index (bool): Whether the index is a datetime index.
  • columns (List[str], optional): The column names of the dataset. Defaults to None.
  • target_column (str, optional): The target column name of the dataset. Defaults to None.
  • feature_columns (str, optional): The feature column names of the dataset. Defaults to None.
  • text_column (str, optional): The text column name of the dataset for nlp tasks. Defaults to None.
  • target_class_labels (Dict, optional): The class labels for the target columns. Defaults to None.
def with_options(self, **kwargs) -> VMDataset:

Support options provided when passing an input to run_test or run_test_suite

Example:

# to only use a certain subset of columns in the dataset:
run_test(
    "validmind.SomeTestID",
    inputs={
        "dataset": {
            "input_id": "my_dataset_id",
            "columns": ["col1", "col2"],
        }
    }
)

# behind the scenes, this retrieves the dataset object (VMDataset) from the registry
# and then calls the `with_options()` method and passes `{"columns": ...}`
Arguments:
  • **kwargs: Options:
    • columns: Filter columns in the dataset
Returns:

VMDataset: A new instance of the dataset with only the specified columns

def assign_predictions( self, model: VMModel, prediction_column: str = None, prediction_values: list = None, probability_column: str = None, probability_values: list = None, prediction_probabilities: list = None, **kwargs):

Assign predictions and probabilities to the dataset.

Arguments:
  • model (VMModel): The model used to generate the predictions.
  • prediction_column (str, optional): The name of the column containing the predictions. Defaults to None.
  • prediction_values (list, optional): The values of the predictions. Defaults to None.
  • probability_column (str, optional): The name of the column containing the probabilities. Defaults to None.
  • probability_values (list, optional): The values of the probabilities. Defaults to None.
  • prediction_probabilities (list, optional): DEPRECATED: The values of the probabilities. Defaults to None.
  • kwargs: Additional keyword arguments that will get passed through to the model's predict method.
def prediction_column( self, model: VMModel, column_name: str = None) -> str:

Get or set the prediction column for a model.

def probability_column( self, model: VMModel, column_name: str = None) -> str:

Get or set the probability column for a model.

def add_extra_column(self, column_name, column_values=None):

Adds an extra column to the dataset without modifying the dataset features and target columns.

Arguments:
  • column_name (str): The name of the extra column.
  • column_values (np.ndarray, optional): The values of the extra column.
df: pandas.core.frame.DataFrame

Returns the dataset as a pandas DataFrame.

Returns:

pd.DataFrame: The dataset as a pandas DataFrame.

x: numpy.ndarray

Returns the input features (X) of the dataset.

Returns:

np.ndarray: The input features.

y: numpy.ndarray

Returns the target variables (y) of the dataset.

Returns:

np.ndarray: The target variables.

def y_pred(self, model) -> numpy.ndarray:

Returns the predictions for a given model.

Attempts to stack complex prediction types (e.g., embeddings) into a single, multi-dimensional array.

Arguments:
  • model (VMModel): The model whose predictions are sought.
Returns:

np.ndarray: The predictions for the model

def y_prob(self, model) -> numpy.ndarray:

Returns the probabilities for a given model.

Arguments:
  • model (str): The ID of the model whose predictions are sought.
Returns:

np.ndarray: The probability variables.

def x_df(self):

Returns a dataframe containing only the feature columns

def y_df(self) -> pandas.core.frame.DataFrame:

Returns a dataframe containing the target column

def y_pred_df(self, model) -> pandas.core.frame.DataFrame:

Returns a dataframe containing the predictions for a given model

def y_prob_df(self, model) -> pandas.core.frame.DataFrame:

Returns a dataframe containing the probabilities for a given model

def target_classes(self):

Returns the target class labels or unique values of the target column.

class VMModel(validmind.vm_models.VMInput):

An base class that wraps a trained model instance and its associated data.

Attributes:
  • model (object, optional): The trained model instance. Defaults to None.
  • input_id (str, optional): The input ID for the model. Defaults to None.
  • attributes (ModelAttributes, optional): The attributes of the model. Defaults to None.
  • name (str, optional): The name of the model. Defaults to the class name.
def serialize(self):

Serializes the model to a dictionary so it can be sent to the API

def predict_proba(self, *args, **kwargs):

Predict probabilties - must be implemented by subclass if needed

@abstractmethod
def predict(self, *args, **kwargs):

Predict method for the model. This is a wrapper around the model's

Inherited Members
VMInput
with_options
@dataclass
class Figure:

Figure objects track the schema supported by the ValidMind API

Figure( key: str, figure: object, metadata: Optional[dict] = None, for_object: Optional[object] = None, extras: Optional[dict] = None, _type: str = 'plot')
def to_widget(self):

Returns the ipywidget compatible representation of the figure. Ideally we would render images as-is, but Plotly FigureWidgets don't work well on Google Colab when they are combined with ipywidgets.

def serialize(self):

Serializes the Figure to a dictionary so it can be sent to the API

def serialize_files(self):

Creates a requests-compatible files object to be sent to the API

@dataclass
class ModelAttributes:

Model attributes definition

ModelAttributes( architecture: str = None, framework: str = None, framework_version: str = None, language: str = None, task: validmind.vm_models.model.ModelTask = None)
@classmethod
def from_dict(cls, data):

Creates a ModelAttributes instance from a dictionary

R_MODEL_TYPES = ['LogisticRegression', 'LinearRegression', 'XGBClassifier', 'XGBRegressor']
@dataclass()
class ResultSummary:

A dataclass that holds the summary of a metric or threshold test results

ResultSummary(results: List[ResultTable])
def add_result(self, result: ResultTable):

Adds a result to the list of results

def serialize(self, as_df=False):

Serializes the ResultSummary to a list of results

@dataclass
class ResultTable:

A dataclass that holds the table summary of result

ResultTable( data: Union[List[Any], pandas.core.frame.DataFrame], type: str = 'table', metadata: ResultTableMetadata = None)
def serialize(self, as_df=False):

Serializes the Figure to a dictionary so it can be sent to the API.

This method accepts as_df parameter to return the data as a DataFrame if we're returning the data to R.

@dataclass
class ResultTableMetadata:

A dataclass that holds the metadata of a table summary

ResultTableMetadata(title: str)
@dataclass
class Test(validmind.vm_models.test_context.TestUtils):
Test( context: Optional[TestContext] = None, inputs: Optional[TestInput] = None, tasks: List[str] = None, tags: List[str] = None, _ref_id: str = None, _section_id: str = None, test_id: str = None, result: validmind.vm_models.test.result_wrapper.ResultWrapper = None, params: dict = None, output_template: str = None, generate_description: bool = True)
def description(self):

Return the test description. May be overridden by subclasses. Defaults to returning the class' docstring

@abstractmethod
def summary(self, *args, **kwargs):

Return the summary. Should be overridden by subclasses.

@abstractmethod
def run(self, *args, **kwargs):

Run the calculation and cache its results

@abstractmethod
def cache_results(self, *args, **kwargs):

Cache the results of the calculation

def log(self):

Log the test results to ValidMind

Inherited Members
validmind.vm_models.test_context.TestUtils
get_accessed_inputs
dataset
model
models
validate_inputs
@dataclass
class Metric(validmind.vm_models.Test):

Metric objects track the schema supported by the ValidMind API

Metric( context: Optional[TestContext] = None, inputs: Optional[TestInput] = None, tasks: List[str] = None, tags: List[str] = None, _ref_id: str = None, _section_id: str = None, test_id: str = None, result: validmind.vm_models.test.result_wrapper.MetricResultWrapper = None, params: dict = None, output_template: str = None, generate_description: bool = True)
@abstractmethod
def summary( self, metric_value: Union[dict, list, pandas.core.frame.DataFrame, NoneType] = None):

Return the metric summary. Should be overridden by subclasses. Defaults to None. The metric summary allows renderers (e.g. Word and ValidMind UI) to display a short summary of the metric results.

We return None here because the metric summary is optional.

def cache_results( self, metric_value: Union[dict, list, pandas.core.frame.DataFrame, NoneType] = None, figures: Optional[List[Figure]] = None):

Cache the results of the metric calculation and do any post-processing if needed

Arguments:
  • metric_value (Union[dict, list, pd.DataFrame]): The value of the metric
  • figures (Optional[object]): Any figures to attach to the test suite result
Returns:

TestSuiteResult: The test suite result object

Inherited Members
Test
description
run
log
validmind.vm_models.test_context.TestUtils
get_accessed_inputs
dataset
model
models
validate_inputs
@dataclass
class MetricResult:

MetricResult class definition. A MetricResult is returned by any internal method that extracts metrics from a dataset or model, and returns 1) Metric and Figure objects that can be sent to the API and 2) and plots and metadata for display purposes

MetricResult( key: dict, ref_id: str, value: Union[dict, list, pandas.core.frame.DataFrame], summary: Optional[ResultSummary] = None, value_formatter: Optional[str] = None)
def serialize(self):

Serializes the Metric to a dictionary so it can be sent to the API

@dataclass
class ThresholdTest(validmind.vm_models.Test):

A threshold test is a combination of a metric/plot we track and a corresponding set of parameters and thresholds values that allow us to determine whether the metric/plot passes or fails.

ThresholdTest( context: Optional[TestContext] = None, inputs: Optional[TestInput] = None, tasks: List[str] = None, tags: List[str] = None, _ref_id: str = None, _section_id: str = None, test_id: str = None, result: ThresholdTestResults = None, params: dict = None, output_template: str = None, generate_description: bool = True)
def summary( self, results: Optional[List[ThresholdTestResult]], all_passed: bool):

Return the threshold test summary. May be overridden by subclasses. Defaults to showing a table with test_name(optional), column and passed.

The test summary allows renderers (e.g. Word and ValidMind UI) to display a short summary of the test results.

def cache_results( self, test_results_list: List[ThresholdTestResult], passed: bool, figures: Optional[List[Figure]] = None):

Cache the individual results of the threshold test as a list of ThresholdTestResult objects

Arguments:
  • result (List[ThresholdTestResult]): The results of the threshold test
  • passed (bool): Whether the threshold test passed or failed
Returns:

TestSuiteResult: The test suite result object

Inherited Members
Test
description
run
log
validmind.vm_models.test_context.TestUtils
get_accessed_inputs
dataset
model
models
validate_inputs
@dataclass
class ThresholdTestResult:

ThresholdTestResult model

ThresholdTestResult( values: dict, test_name: Optional[str] = None, column: Optional[str] = None, passed: Optional[bool] = None)
def serialize(self):

Serializes the ThresholdTestResult to a dictionary so it can be sent to the API

@dataclass
class ThresholdTestResults:

ThresholdTestResults model

ThresholdTestResults( test_name: str, ref_id: str, params: dict, passed: bool, results: List[ThresholdTestResult], summary: Optional[ResultSummary])
def serialize(self):

Serializes the ThresholdTestResults to a dictionary so it can be sent to the API

@dataclass
class TestContext:

Holds context that can be used by tests to run. Allows us to store data that needs to be reused across different tests/metrics such as shared dataset metrics, etc.

TestContext( context_data: Optional[dict] = None, dataset: VMDataset = None, model: VMModel = None, models: List[VMModel] = None)
def set_context_data(self, key, value):
def get_context_data(self, key):
class TestInput:

Holds models, datasets and other arbitrary inputs for test(s)

Attributes:
  • dataset (VMDataset): Single dataset for dataset-only tests
  • model (VMModel): Single model for model-related tests
  • models (List[VMModel]): Multiple models for model comparison tests
  • ... (any): Any other arbitrary inputs that can be used by tests
TestInput(inputs)

Initialize with either a dictionary of inputs

@dataclass
class TestSuite:

Base class for test suites. Test suites are used to define a grouping of tests that can be run as a suite against datasets and models. Test Suites can be defined by inheriting from this base class and defining the list of tests as a class variable.

Tests can be a flat list of strings or may be nested into sections by using a dict

TestSuite( sections: List[validmind.vm_models.test_suite.test_suite.TestSuiteSection] = None)
def get_tests(self) -> List[str]:

Get all test IDs from all sections

def num_tests(self) -> int:

Returns the total number of tests in the test suite

def get_default_config(self) -> dict:

Returns the default configuration for the test suite

Each test in a test suite can accept parameters and those parameters can have default values. Both the parameters and their defaults are set in the test class and a config object can be passed to the test suite's run method to override the defaults. This function returns a dictionary containing the parameters and their default values for every test to allow users to view and set values

Returns:

dict: A dictionary of test names and their default parameters

class TestSuiteRunner:

Runs a test suite

TestSuiteRunner( suite: TestSuite, input: TestInput, config: dict = None)
async def log_results(self):

Logs the results of the test suite to ValidMind

This method will be called after the test suite has been run and all results have been collected. This method will log the results to ValidMind.

def summarize(self, show_link: bool = True):
def run(self, send: bool = True, fail_fast: bool = False):

Runs the test suite, renders the summary and sends the results to ValidMind

Arguments:
  • send (bool, optional): Whether to send the results to ValidMind. Defaults to True.
  • fail_fast (bool, optional): Whether to stop running tests after the first failure. Defaults to False.