Enable PII detection in tests

Learn how to enable and configure Personally Identifiable Information (PII) detection when running tests with the ValidMind Library. Choose whether or not to include PII in test descriptions generated, or whether or not to include PII in test results logged to the ValidMind Platform.

About ValidMind

ValidMind is a suite of tools for managing risk, including risk associated with AI and statistical models.

You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on documentation. Together, these products simplify risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and validators.

Before you begin

This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

New to ValidMind?

If you haven't already seen our documentation on the ValidMind Library, we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting records such as models and running tests, as well as find code samples and our Python Library API reference.

For access to all features available in this notebook, you'll need access to a ValidMind account.

Register with ValidMind

Key concepts

record: A tool tracked in the ValidMind inventory, such as a model. Records include traditional statistical models, legacy systems, artificial intelligence/machine learning models, large language models (LLMs), agentic AI systems, and other documentable items that benefit from oversight, testing, and lifecycle management.

model: SR 26-2 (which supersedes SR 11-7) defines a model as a "complex quantitative method, system, or approach that applies statistical, economic, or financial theories to process input data into quantitative estimates." Simple arithmetic, deterministic rule-based processes, or software without statistical, economic, or financial theories underpinning their design or use are generally outside SR 26-2’s definition of a model. Within ValidMind, a model is a type of record tracked in the inventory.

documentation, model documentation: A structured and detailed document pertaining to a record, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. Within the realm of risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the record's application.

document template: Lays out the structure of documents, segmented into various sections and sub-sections, and functions as a test suite specifying the tests that should be run, and how the results should be displayed. Document templates help automate your development, validation, monitoring, and other risk management processes. Document templates are available for default ValidMind document types as well as custom document types.

documentation template: A default ValidMind document type that serves as a standardized framework for developing and documenting records, including sections designated for record details, data descriptions, test results, and performance metrics. By outlining required documentation and recommended analyses, document templates ensure consistency and completeness across documentation and help guide developers through a systematic development process while promoting comparability and traceability of development outcomes.

test: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or record. Test results are logged to the ValidMind Platform, where they are attached to documents. Tests are the building blocks of ValidMind, used to evaluate and document records and datasets, and can be run individually or as part of a suite defined by your templates.

test suite: A collection of tests designed to run together to automate and generate documentation end-to-end for specific use cases. (Learn more: test_suites)

metric: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.

custom test: Functions that you define to evaluate your record or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.

inputs: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:

model: A single record that has been initialized in ValidMind with init_model(). Despite the naming convention, model objects can be any type of record you want to test, document, validate, or monitor with ValidMind.
dataset: A single dataset that has been initialized in ValidMind with init_dataset().
models: A list of ValidMind records - usually this is used when you want to compare multiple records in your custom tests.
datasets: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom tests. (Learn more: Run tests with multiple datasets)

parameters: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.

outputs: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.

Setting up

Install the ValidMind Library with PII detection

Recommended Python versions

Python 3.8 <= x <= 3.14

To use PII detection powered by Microsoft Presidio, install the library with the explicit [pii-detection] extra specifier:

%pip install -q "validmind[pii-detection]"

Initialize the ValidMind Library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet

Initialize the ValidMind Library with the code snippet unique to each record per document, ensuring your test results are uploaded to the correct record and automatically populated in the right document in the ValidMind Platform when you run the Library.

On the left sidebar that appears for your model, select Getting Started and select Development from the DOCUMENT drop-down menu.
Click Copy snippet to clipboard.
Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
    document="documentation",
)

Create a custom test that outputs PII

To demonstrate the feature, we'll need a test that outputs PII. First we'll create a custom test that returns:

A description string containing PII (name, email, phone)
A small table containing PII in columns

This output mirrors the structure used in other custom test notebooks and will exercise both table and description PII detection paths. However, if structured detection is unavailable, the library falls back to token-level text scans when possible.

import pandas as pd

from validmind import test

@test("pii_demo.PIIDetection")
def pii_custom_test():
    """A custom test that returns demo PII.
    This default test description will display when PII is not sent to the LLM to generate test descriptions based on test result data."""
    return pd.DataFrame(
        {
            "name": ["Jane Smith", "John Doe", "Alice Johnson"],
            "email": [
                "jane.smith@bank.example",
                "john.doe@company.example",
                "alice.johnson@service.example",
            ],
            "phone": ["(212) 555-9876", "(415) 555-1234", "(646) 555-5678"],
        }
    )

Want to learn more about custom tests?

Check out our extended introduction to custom tests — Implement custom tests

Running tests under different PII detection modes

Next, let's import the run_test function provided by the validmind.tests module to run our custom test via a function called run_pii_test() that catches exceptions to observe blocking behavior when PII is present:

import os
from validmind.tests import run_test

# Run test and tag result with unique `result_id`
def run_pii_test(result_id=""):
    try:
        test_name = f"pii_demo.PIIDetection:{result_id}"
        result = run_test(test_name)

        # Check if the test description was generated by LLM
        if not result._was_description_generated:
            print("PII detected: LLM-generated test description skipped")
        else:
            print("No PII detected or detection disabled: Test description generated by LLM")

        # Try logging test results to the ValidMind Platform
        result.log()
        print("No PII detected or detection disabled: Test results logged to the ValidMind Platform")
    except Exception as e:
        print("PII detected: Test results not logged to the ValidMind Platform")

We'll then switch the VALIDMIND_PII_DETECTION environment variable across modes in the below examples.

Note that since we are running a custom test that does not exist in your model's default documentation template, we'll receive output indicating that a test-driven block doesn't currently exist in your model's documentation for that particular test ID.

That's expected, as when we run custom tests the results logged need to be manually added to your documentation within the ValidMind Platform or added to your documentation template.

disabled

When detection is set to disabled, tests run and generate test descriptions. Logging tests with .log() will also send test descriptions and test results to the ValidMind Platform as usual:

print("\n=== Mode: disabled ===")
os.environ["VALIDMIND_PII_DETECTION"] = "disabled"

# Run test and tag result with unique ID `disabled`
run_pii_test("disabled")

test_results

When detection is set for test_results, tests run and generate test descriptions for review in your environment, but logging tests will not send descriptions or test results to the ValidMind Platform:

print("\n=== Mode: test_results ===")
os.environ["VALIDMIND_PII_DETECTION"] = "test_results"

# Run test and tag result with unique ID `results_blocked`
run_pii_test("results_blocked")

test_descriptions

When detection is set for test_descriptions, tests run but will not generate test descriptions, and logging tests will not send descriptions but will send test results to the ValidMind Platform:

print("\n=== Mode: test_descriptions ===")
os.environ["VALIDMIND_PII_DETECTION"] = "test_descriptions"

# Run test and tag result with unique ID `desc_blocked`
run_pii_test("desc_blocked")

all

When detection is set to all, tests run will not generate test descriptions or log test results to the ValidMind Platform.

print("\n=== Mode: all ===")
os.environ["VALIDMIND_PII_DETECTION"] = "all"

# Run test and tag result with unique ID `all_blocked`
run_pii_test("all_blocked")

Overriding detection

You can override blocking by passing unsafe=True to result.log(unsafe=True), but this is not recommended outside controlled workflows.

To demonstrate, let's rerun our custom test with some override scenarios.

Override test result logging

First, let's rerun our custom test with detection set to all, which will send the test results but not the test descriptions to the ValidMind Platform:

print("\n=== Mode: all & unsafe=True ===")
os.environ["VALIDMIND_PII_DETECTION"] = "all"

# Run test and tag result with unique ID `override_results`
try:
    result = run_test("pii_demo.PIIDetection:override_results")

    # Check if the test description was generated by LLM
    if not result._was_description_generated:
        print("PII detected: LLM-generated test description skipped")
    else:
        print("No PII detected or detection disabled: Test description generated by LLM")

    # Try logging test results to the ValidMind Platform
    result.log(unsafe=True)
    print("No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform")
except Exception as e:
    print("PII detected: Test results not logged to the ValidMind Platform")

Override test descriptions and test result logging

To send both the test descriptions and test results via override, set the VALIDMIND_PII_DETECTION environment variable to test_results while including the override flag:

print("\n=== Mode: test_results & unsafe=True ===")
os.environ["VALIDMIND_PII_DETECTION"] = "test_results"

# Run test and tag result with unique ID `override_both`
try:
    result = run_test("pii_demo.PIIDetection:override_both")

    # Check if the test description was generated by LLM
    if not result._was_description_generated:
        print("PII detected: LLM-generated test description skipped")
    else:
        print("No PII detected, detection disabled, or override set: Test description generated by LLM")

    # Try logging test results to the ValidMind Platform
    result.log(unsafe=True)
    print("No PII detected, detection disabled, or override set: Test results logged to the ValidMind Platform")
except Exception as e:
    print("PII detected: Test results not logged to the ValidMind Platform")

Review logged test results

Now let's take a look at the results that were logged to the ValidMind Platform:

From the Inventory in the ValidMind Platform, go to the model you registered earlier.
In the left sidebar that appears for your model, click Development under Documents.
Click on any section heading to expand that section to add a new test-driven block. (Learn more: Work with test results)
Under TEST-DRIVEN in the sidebar, click Custom.
Confirm that you're able to insert the following logged results:
- pii_demo.PIIDetection:disabled
- pii_demo.PIIDetection:desc_blocked
- pii_demo.PIIDetection:override_results
- pii_demo.PIIDetection:override_both

Troubleshooting

If you see warnings that Presidio or Presidio analyzer is unavailable, ensure you installed extras: validmind[pii-detection].
Ensure your environment is restarted after installing new packages if imports fail.

Learn more

We also offer many interactive notebooks to help you use the ValidMind Library to streamline your work:

Or, visit our documentation to learn more about ValidMind.

Upgrade ValidMind

After installing ValidMind, you'll want to periodically make sure you are on the latest version to access any new features and other enhancements.

Retrieve the information for the currently installed version of ValidMind:

%pip show validmind

If the version returned is lower than the version indicated in our production open-source code, restart your notebook and run:

%pip install --upgrade validmind

You may need to restart your kernel after running the upgrade package for changes to be applied.