Enable PII detection in tests

Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform.

About ValidMind

ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.

You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

Before you begin

This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

New to ValidMind?

If you haven't already seen our documentation on the ValidMind Library, we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.

For access to all features available in this notebook, you'll need access to a ValidMind account.

Register with ValidMind

Key concepts

Model documentation: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.

Documentation template: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.

Tests: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.

Metrics: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

Custom metrics: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

Inputs: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

model: A single model that has been initialized in ValidMind with vm.init_model().
dataset: Single dataset that has been initialized in ValidMind with vm.init_dataset().
models: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.
datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: Run tests with multiple datasets)

Parameters: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.

Outputs: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Test suites: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.

Example: the classifier_full_suite test suite runs tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use-cases.

Setting up

Install the ValidMind Library with PII detection

Recommended Python versions

Python 3.8 <= x <= 3.11

To use PII detection powered by Microsoft Presidio, install the library with the explicit [pii-detection] extra specifier:

%pip install -q "validmind[pii-detection]"

Initialize the ValidMind Library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet

In a browser, log in to ValidMind.
In the left sidebar, navigate to Inventory and click + Register Model.
Enter the model details and click Continue. (Need more help?)
Go to Getting Started and click Copy snippet to clipboard.

Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
)

Using PII detection

Create a custom test that outputs PII

To demonstrate the feature, we'll need a test that outputs PII. First we'll create a custom test that returns:

A description string containing PII (name, email, phone)
A small table containing PII in columns

This output mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths. However, if structured detection is unavailable, the library falls back to token-level text scans when possible.

import pandas as pd

from validmind import test

@test("pii_demo.PIIDetection")
def pii_custom_test():
    """A custom test that returns demo PII.
    This default test description will display when PII is not sent to the LLM to generate test descriptions based on test result data."""
    return pd.DataFrame(
        {
            "name": ["Jane Smith", "John Doe", "Alice Johnson"],
            "email": [
                "jane.smith@bank.example",
                "john.doe@company.example",
                "alice.johnson@service.example",
            ],
            "phone": ["(212) 555-9876", "(415) 555-1234", "(646) 555-5678"],
        }
    )

Want to learn more about custom tests?

Check out our extended introduction to custom tests — Implement custom tests

Run test under different PII detection modes

Next, let's import the run_test function provided by the validmind.tests module to run our custom test via a function called run_pii_test() that catches exceptions to observe blocking behavior when PII is present:

import os
from validmind.tests import run_test

# Run test and tag result with unique `result_id`
def run_pii_test(result_id=""):
    try:
        test_name = f"pii_demo.PIIDetection:{result_id}"
        result = run_test(test_name)

        # Check if the test description was generated by LLM
        if not result._was_description_generated:
            print("PII detected: LLM-generated test description skipped")
        else:
            print("No PII detected or detection disabled: Test description generated by LLM")

        # Try logging test results to the ValidMind Platform
        result.log()
        print("No PII detected or detection disabled: Test results logged to the ValidMind Platform")
    except Exception as e:
        print("PII detected: Test results not logged to the ValidMind Platform")

We'll then switch the VALIDMIND_PII_DETECTION environment variable across modes in the below examples.

Note that since we are running a custom test that does not exist in your model's default documentation template, we'll receive output indicating that a test-driven block doesn't currently exist in your model's documentation for that particular test ID.

That's expected, as when we run custom tests the results logged need to be manually added to your documentation within the ValidMind Platform or added to your documentation template.

disabled

When detection is set to disabled, tests run and generate test descriptions. Logging tests with .log() will also send test descriptions and test results to the ValidMind Platform as usual:

print("\n=== Mode: disabled ===")
os.environ["VALIDMIND_PII_DETECTION"] = "disabled"

# Run test and tag result with unique ID `disabled`
run_pii_test("disabled")

test_results

When detection is set for test_results, tests run and generate test descriptions for review in your environment, but logging tests will not send descriptions or test results to the ValidMind Platform:

print("\n=== Mode: test_results ===")
os.environ["VALIDMIND_PII_DETECTION"] = "test_results"

# Run test and tag result with unique ID `results_blocked`
run_pii_test("results_blocked")

test_descriptions

When detection is set for test_descriptions, tests run but will not generate test descriptions, and logging tests will not send descriptions but will send test results to the ValidMind Platform:

print("\n=== Mode: test_descriptions ===")
os.environ["VALIDMIND_PII_DETECTION"] = "test_descriptions"

# Run test and tag result with unique ID `desc_blocked`
run_pii_test("desc_blocked")

all

When detection is set to all, tests run will not generate test descriptions or log test results to the ValidMind Platform.

print("\n=== Mode: all ===")
os.environ["VALIDMIND_PII_DETECTION"] = "all"

# Run test and tag result with unique ID `all_blocked`
run_pii_test("all_blocked")

Override detection

You can override blocking by passing unsafe=True to result.log(unsafe=True), but this is not recommended outside controlled workflows.

To demonstrate, let's rerun our custom test with some override scenarios.

Override test result logging

First, let's rerun our custom test with detection set to all, which will send the test results but not the test descriptions to the ValidMind Platform:

print("\n=== Mode: all & unsafe=True ===")
os.environ["VALIDMIND_PII_DETECTION"] = "all"

# Run test and tag result with unique ID `override_results`
try:
    result = run_test("pii_demo.PIIDetection:override_results")

    # Check if the test description was generated by LLM
    if not result._was_description_generated:
        print("PII detected: LLM-generated test description skipped")
    else:
        print("No PII detected or detection disabled: Test description generated by LLM")

    # Try logging test results to the ValidMind Platform
    result.log(unsafe=True)
    print("No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform")
except Exception as e:
    print("PII detected: Test results not logged to the ValidMind Platform")

Override test descriptions and test result logging

To send both the test descriptions and test results via override, set the VALIDMIND_PII_DETECTION environment variable to test_results while including the override flag:

print("\n=== Mode: test_results & unsafe=True ===")
os.environ["VALIDMIND_PII_DETECTION"] = "test_results"

# Run test and tag result with unique ID `override_both`
try:
    result = run_test("pii_demo.PIIDetection:override_both")

    # Check if the test description was generated by LLM
    if not result._was_description_generated:
        print("PII detected: LLM-generated test description skipped")
    else:
        print("No PII detected, detection disabled, or override set: Test description generated by LLM")

    # Try logging test results to the ValidMind Platform
    result.log(unsafe=True)
    print("No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform")
except Exception as e:
    print("PII detected: Test results not logged to the ValidMind Platform")

Review logged test results

Now let's take a look at the results that were logged to the ValidMind Platform:

From the Inventory in the ValidMind Platform, go to the model you registered earlier.
In the left sidebar that appears for your model, click Documentation under Documents.
Click on any section heading to expand that section to add a new test-driven block (Need more help?).
Under TEST-DRIVEN in the sidebar, click Custom.
Confirm that you're able to insert the following logged results:
- pii_demo.PIIDetection:disabled
- pii_demo.PIIDetection:desc_blocked
- pii_demo.PIIDetection:override_results
- pii_demo.PIIDetection:override_both

Troubleshooting

If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: validmind[pii-detection].
Ensure your environment is restarted after installing new packages if imports fail.

Learn more

We offer many interactive notebooks to help you document models:

Or, visit our documentation to learn more about ValidMind.