validmind.scorer

scorer

defscorer(func_or_id:Union[Callable[..., Any], str, None]=None) → Callable[[validmind.vm_models.F], validmind.vm_models.F]:

Decorator for creating and registering custom scorers

This decorator registers the function it wraps as a scorer function within ValidMind under the provided ID. Once decorated, the function can be run using the validmind.scorer.run_scorer function.

The scorer ID can be provided in three ways:

  1. Explicit ID: @scorer("validmind.scorer.classification.BrierScore")
  2. Auto-generated from path: @scorer() - automatically generates ID from file path
  3. Function name only: @scorer - uses function name with validmind.scorer prefix

The function can take two different types of arguments:

  • Inputs: ValidMind model or dataset (or list of models/datasets). These arguments must use the following names: model, models, dataset, datasets.
  • Parameters: Any additional keyword arguments of any type (must have a default value) that can have any name.

The function should return one of the following types:

  • Table: Either a list of dictionaries or a pandas DataFrame
  • Plot: Either a matplotlib figure or a plotly figure
  • Scalar: A single number (int or float)
  • Boolean: A single boolean value indicating whether the test passed or failed
  • List: A list of values (for row-level metrics) or a list of dictionaries with consistent keys
  • Any other type: The output will be stored as raw data for use by calling code

When returning a list of dictionaries:

  • All dictionaries must have the same keys
  • The list length must match the number of rows in the dataset
  • Each dictionary key will become a separate column when using assign_scores
  • Column naming follows the pattern: {model_id}{metric_name}{dict_key}

Note: Scorer outputs are not logged to the backend and are intended for use by other parts of the system (e.g., assign_scores method).

The function may also include a docstring. This docstring will be used and logged as the scorer's description.

Arguments

  • func_or_id (Union[Callable[..., Any], str, None], optional): Either the function to decorate or the scorer ID. If None or empty string, the ID is auto-generated from the file path. Defaults to None.

Returns

  • The decorated function.

list_scorers

deflist_scorers(**kwargs):

List all scorers

describe_scorer

defdescribe_scorer(scorer_id:str,**kwargs):

Describe a scorer

run_scorer

defrun_scorer(scorer_id:str,**kwargs):

Run a scorer