Benchmarking Databricks ai_mask on clinical de-identification: 0.71 PHI F1

`ai_mask()` is a Databricks SQL function, in Public Preview and HIPAA compliant, that masks entity types named in a SQL array literal. Run against an expert-annotated clinical corpus with the same evaluation code as our published comparisons, it reaches 0.71 PHI F1, against 0.96 for John Snow Labs Healthcare NLP. A second finding matters as much as the score: asking ai_mask() for more identifier types makes it find fewer of them.

What ai_mask is

You give ai_mask() a column of text and a list of entity types written in plain English, and it returns the same text with matching spans replaced by the literal string [MASKED]. Its appeal is that there is nothing to deploy: one line of SQL, no endpoint, no model to manage, and it composes with Unity Catalog column masks and dynamic views. It runs on a chat model behind the Foundation Model APIs, is tuned for English, and requires serverless compute on Databricks Runtime 18.2 or above.

How the benchmark was run

The ground truth is a public expert-annotated set of 1,479 PHI chunks across 48 clinical documents (45,608 tokens), covering six classes: DATE 582, NAME 380, LOCATION 236, IDNUM 185, CONTACT 49, AGE 47. The scoring code is the same code behind our published de-identification comparisons, unmodified.

One extra step was needed. ai_mask() returns masked text and no spans, so each [MASKED] placeholder was replaced with a sentinel character and the output diffed against the original at token level to recover offsets. Everything downstream of that step is the original benchmark code. The run used the Public Preview version of ai_mask() in August 2026, as a single Spark job across 16 partitions.

Two scoring levels are reported, because they answer different questions. Chunk-level scoring, ignoring the label, answers whether the identifier was found at all, which is the compliance question. Token-level classification gives precision, recall, and F1 per entity type. Recall is the number that carries regulatory risk, because anything missed stays in the text; precision measures how much non-PHI got redacted along the way.

Results: recall from 0.35 to 0.85 by entity

Precision exceeds recall in every class, which makes this a coverage gap rather than a false-positive problem. What ai_mask() finds, it labels correctly: the entity type is right on 422 of 423 matched dates and 236 of 239 matched names. The failure is that 28% of annotated identifiers are never touched and another 21% are only partially covered, and a partial match on a name leaves surname or initials in the text. Age is the exception, missing almost nothing but aligning boundaries poorly, at 68% partial overlaps. Contact is the sharpest gap, at 0.35 recall on phone, fax, and email.

For context, the same corpus and code put Gemini 3.1 Pro at 0.86 PHI F1, GPT-5.5 at 0.89, Claude Opus 4.8 at 0.91, and Healthcare NLP at 0.96. Throughput for ai_mask() was 64.2 seconds for all 48 documents, or 1.34 seconds per document, billed as serverless model serving at per-second granularity.

Naming more identifier types drops F1 from 0.71 to 0.61

ai_mask() takes free-form natural-language labels rather than a fixed schema, so the label list is itself an experimental variable:

Going from six broad labels to the eighteen specific ones a HIPAA-aligned taxonomy needs, including medical record number, hospital name, and the patient-versus-doctor distinction, drops PHI F1 from 0.71 to 0.61 at the same call cost. The configuration closest to the regulation is the one that performs worst. Since nothing in the function checks the label list against the regulation, the completeness of the de-identification equals the completeness of a list somebody typed into a SQL array literal.

The accuracy gap is the smaller problem

Suppose ai_mask() scored 0.96 tomorrow. It would still not carry a de-identification program, because detection accuracy is the first requirement of several.

ai_mask() masks. That is the entire feature set. It does not obfuscate, so it cannot put a readable surrogate in place of a name. It does not maintain consistency, so the same patient masked in two notes leaves no way to tell it was the same patient. It does not return spans, so there is no record of what was detected, no confidence to threshold on, and no way to sample detections for human review, and those artifacts are what an Expert Determination file is built from. It has no fixed entity taxonomy to validate against a regulation, and no way to customize behavior for your document types, your local identifier formats, or your institution’s rules.

The last row decides whether the result can go in a compliance file at all. If a regulator asks which model version de-identified a 2026 cohort, “whichever one the platform was serving that quarter” is not an answer that survives review.

Healthcare NLP is purpose-built for this task: 0.98 micro F1 on the official 2014 i2b2 test set and 0.98 recall on a 381,959-token clinical corpus, published in peer-reviewed benchmarks, with all 18 Safe Harbor categories covered, and detection, obfuscation, tokenization, and validation in one pipeline that runs inside your environment, including on the Databricks Marketplace, so the two are not either-or on that platform.

Reproduce it on your own data

The ground truth and scoring functions are public at the link above, and the span-recovery step is a token-level diff any engineer can rewrite in an afternoon. To try the healthcare pipeline on your own notes, use the live demos and notebooks page or the clinical de-identification Colab, and measure recall per entity type rather than the aggregate, because a 0.71 headline hides a 0.35 on contact identifiers.

Frequently asked questions

ai_mask is HIPAA compliant. Isn’t that enough?

The two claims describe different things. “HIPAA compliant” describes the service that processes your data, meaning the platform will sign a BAA and handle PHI appropriately in transit and at rest. It says nothing about whether the output is de-identified. That depends on the recall of the detection, on whether the entity list covers all 18 Safe Harbor categories, and on whether you can document what was detected. On this benchmark, ai_mask() reaches 0.71 PHI F1, returns no spans, and detects only the categories the caller lists.

Can I close the gap by listing all 18 Safe Harbor categories as labels?

The measured effect goes the other way. The full 18-label HIPAA-aligned configuration scored 0.61 PHI F1, while six broad labels scored 0.71 at the same call cost. Splitting the labels across six separate calls recovered some spans at 0.69, at six times the inference cost.

Why does returning no spans matter?

Spans, offsets, and confidence scores are the record of what was detected. They are what lets a compliance team sample detections for review, threshold on confidence, and document the process for Expert Determination. A function that returns only masked text produces an output without an audit trail.

Is 0.71 F1 good enough for anything?

For clinical de-identification, no: the literature treats 95% as the level equivalent to careful manual de-identification, and regulatory status is binary. For lower-stakes uses, such as reducing incidental PII in logs or internal analytics on non-clinical text, general-purpose masking at this level has legitimate uses. The line is whether the output needs de-identified status under HIPAA or GDPR.

Can Healthcare NLP run inside Databricks?

Yes. The de-identification pipelines are available on the Databricks Marketplace and run on your own cluster, so the text never leaves your environment. The comparison here is between tools, and both run on the same platform.