Clinical de-identification benchmarks 2026: John Snow Labs against OpenAI, Databricks, Presidio, and LLM APIs

Clinical de-identification benchmarks in 2026 put John Snow Labs Healthcare NLP at 0.96 PHI F1 on expert-annotated clinical notes, against 0.91 for Claude Opus 4.8, 0.89 for GPT-5.5, 0.86 for Gemini 3.1 Pro, 0.71 for Databricks ai_mask(), 0.60 to 0.85 for Microsoft Presidio in independent peer-reviewed studies, and 0.55 for OpenAI’s Privacy Filter. On corpora where recall is reported, the pipeline holds 0.98 recall, and it reaches 0.98 micro F1 on the official 2014 i2b2 test set. This page consolidates every comparison in one place, with methodology and reproduction links for each.

The systems and the headline numbers

Three ground-truth corpora are behind these numbers. A public expert-annotated set of 1,479 PHI chunks across 48 clinical documents drives the frontier-model and ai_mask() comparisons, with the scoring code published alongside it. A private 381,959-token corpus of mixed note types at 3.25% PHI density drives the Privacy Filter comparison. And 100 notes sampled from the public 2014 i2b2 corpus and annotated by two physicians drive the peer-reviewed cloud API comparison. The Presidio rows come from independent studies whose corpora we had no hand in, which is why they are included: they show what third parties measure on clinical text we did not choose.

OpenAI Privacy Filter: 0.55 F1

`openai/privacy-filter` is an Apache 2.0 token classifier for high-throughput PII sanitization across eight categories. On 381,959 tokens of clinical text, the healthcare pipeline detects 54% more PHI at 5.8 times the CPU throughput.

Addresses and dates are both Safe Harbor categories, and OpenAI’s own model card states the model is not an anonymization or compliance tool and that accuracy degrades on domain-specific text including medical. Full methodology and per-entity tables.

Presidio: 0.60 to 0.85 F1 in independent studies

Presidio is the most widely deployed open-source PII framework. Both evaluations below are peer-reviewed and vendor-independent.

The higher result came with OCR preprocessing and relaxed scoring that counts partial redactions as correct; the lower one used raw notes, where roughly half of what Presidio redacted was not PHI. Our full comparison covers the entity-coverage differences behind these numbers.

Databricks ai_mask: 0.71 F1 and lower with more labels

`ai_mask()` masks entity types named in a SQL array literal, returning [MASKED] with no spans. On the expert-annotated set, run with the published scoring code:

Precision exceeds recall in every entity class, contact identifiers are found 35% of the time, and requesting the eighteen categories a HIPAA-aligned taxonomy needs drops PHI F1 from 0.71 to 0.61. The dedicated benchmark post has the per-entity tables, the span-recovery method, and the capability comparison.

Frontier LLM APIs: below the bar at 0.86 to 0.91

Prompted frontier models, same expert-annotated corpus, same evaluation code, per-entity F1:

GPT-4o scored 0.79 PHI F1 in our Text2Story 2025 evaluation; the current generation reaches 0.86 to 0.91. Both sit below the 95% level the literature treats as equivalent to manual de-identification, so neither output qualifies as de-identified, and the errors concentrate in the categories specific to clinical documents: contact runs 0.49 to 0.69 against 0.95 because phone and fax numbers sit in headers, footers, and referral blocks rather than prose, and age runs 0.60 to 0.93 against 0.97, on a Safe Harbor category with its own aggregation rule above 89. These are enumerated identifier categories, and where general training distributions have the least to say.

Cloud healthcare APIs: 90%, 67%, and 42% of patient names

On 100 physician-annotated i2b2 notes, peer-reviewed at ML4H 2023, micro-averaged F1 was 0.97 for Healthcare NLP, 0.96 for AWS Comprehend Medical, 0.72 for Azure Text Analytics for Health, and 0.64 for the GCP Healthcare API. AWS is the close one on the aggregate, at 0.959 against 0.969 before rounding. Recall on names is what separates the field: patient names were found 93% of the time by Healthcare NLP, 90% by AWS, 67% by Azure, and 42% by GCP, and Azure found 55% of physician names. On the official 2014 i2b2 test set, the same NER models score 0.98 micro F1 with 13 labels and 0.96 with 7. Full comparison, including prompts and per-entity tables.

The human baseline: 0.81 alone and 0.94 in consensus

The regulatory bar is binary, and the practical bar sits above the average individual expert. Aggregate scores also hide the failures that decide compliance, so measure recall per entity type on your own notes: a 0.71 headline can hide a 0.35 on contact identifiers, and a 0.64 average can hide a 0.42 on patient names.

Reproduce any of these results

The expert-annotated ground truth and scoring code are public, the i2b2 corpus is available to any researcher through n2c2, and the clinical de-identification Colab runs the pipeline in one click. The live demos page accepts pasted text, and the pipelines are on the AWS, Azure, Snowflake, and Databricks marketplaces so they can run next to data you cannot move. Detection scores are the entry ticket rather than the whole job: the de-identification solution page covers Safe Harbor entity coverage, consistent obfuscation, deterministic tokenization, and the Providence deployment, where 2 billion notes were de-identified with zero re-identifications in adversarial testing on 790 patients.

Frequently asked questions

Which de-identification tool is most accurate on clinical text?

Across every 2026 comparison on expert-annotated clinical notes, John Snow Labs Healthcare NLP scores highest at 0.96 PHI F1, alongside 0.98 micro F1 on the official i2b2 2014 test set and 0.98 recall at 381,959-token scale, followed by prompted frontier models at 0.86 to 0.91, Databricks ai_mask() at 0.71, Presidio at 0.60 to 0.85 in independent studies, and OpenAI’s Privacy Filter at 0.55.

Are these vendor benchmarks or independent ones?

Both, and the page labels which is which. The Presidio numbers come from two peer-reviewed studies with no vendor involvement. The cloud API comparison is peer-reviewed at ML4H 2023 on notes from the public i2b2 corpus. The Privacy Filter, ai_mask(), and frontier-model comparisons are ours, with the ground truth and scoring code published so they can be rerun.

Why do general LLMs score lower than a purpose-built pipeline?

The gap concentrates in identifiers specific to clinical documents. Contact details sit in templates, headers, and signature blocks rather than prose, ages carry a regulatory aggregation rule above 89, and identifiers appear as MRN, MR#, and local conventions the models rarely saw in training. On identifiers that resemble general PII, such as names, the frontier models are near parity, and the aggregate still falls below the 95% regulatory-grade threshold.

Is a high F1 score sufficient for HIPAA de-identification?

No. HIPAA Safe Harbor requires removing all 18 identifier categories, several of which general tools carry no label for, and Expert Determination requires documented, statistically validated re-identification risk. A production program also needs consistent obfuscation, tokenization, and span-level audit records, which are separate capabilities from detection.

How can I run these benchmarks on my own notes?

Download the public ground truth and scoring code, or annotate a sample of your own notes and score each tool per entity type at both token and chunk level. Report recall per entity and the share of documents with zero missed identifiers, and count partial matches as misses, because a partially redacted name leaves PHI in the text.