De-identification removes protected health information (PHI) from clinical notes, and the team that releases the output still has to decide, note by note, whether anything was missed. PHI Guard makes that decision for every note. You pass it a de-identified note, and it returns the probability that PHI is still readable, so you release the notes that pass and send the rest to a reviewer. The family has seven John Snow Labs models. All of them run on a CPU in about 1 GB of RAM, and they ship in Healthcare NLP 7.0.1 through the DecisionModel annotator.
Three notes and their release decisions
Here are three synthetic notes after masking, and the leak probability the v0.7 guard returns for each:
Note 1 Name: <PATIENT>, MRN <MEDICALRECORD>, seen on <DATE> by Dr. <DOCTOR> at <HOSPITAL>. Phone <PHONE>. Chest pain resolved, discharged home on aspirin 81 mg daily. Guard leak 0.0009: release Note 2 Discharged home. Follow up with [NAME] at 555-0142 in two weeks. Continue metformin 500 mg twice daily. Guard leak 0.9999: hold, a phone number is left Note 3 A 93-year-old patient was admitted on [DATE] with a hip fracture after a fall at home. Surgery planned for tomorrow. Guard leak 0.9986: hold, an age over 89 is left
Note 1 is fully masked and passes. Note 2 still shows a phone number. In note 3 the date is masked and the age is left in, and HIPAA Safe Harbor lists all ages over 89 among the identifiers to remove, or to group into a single category of 90 or older (45 CFR § 164.514(b)(2)(i)). The guard returns one score for the whole note and does not change the text; the notes it flags go to review.
Why check the output of de-identification
Without a check on the output, the release decision rests on a workaround:
- A person reviews a sample of notes by hand, and the notes outside the sample go out unchecked.
- A second extraction pass returns entity spans, and someone has to read them and decide what each one means for release.
- Each note is sent to a general LLM with a prompt, and its free-text reply is read.
PHI Guard adds the check as one more stage of the Spark pipeline. It reads the masked output itself, the text you are about to release, and writes one decision per document with the probability in the metadata. The stage is saved and loaded with the pipeline like any other, and it runs on a CPU.
Four questions about every note
Each PHI Guard model answers one or more of four questions. For each one: what it measures, why you need it, and what you would do without it.
Leak
The leak output is the probability that PHI is still readable in a de-identified note. It is the release gate: notes under your cut-off go out, and the rest go to review. Without it, finding leftover PHI means reviewing samples by hand or running a second extraction pass.
Harmful leak
The harmful output is the probability that a leak is harmful. The v0.7 models treat these leaks as harmful: a name, an ID, a contact detail, a date of birth, a street address, an age over 89, or three or more harmless kinds of detail together. A clinician name, a city or state, a hospital, an organization, a date other than the birth date, or an age up to 89 on its own is a harmless leak. The harmful score is never higher than the leak score.
It cuts the number of notes with only harmless leaks that go to review. With only a leak score, phi_guard_3m_v0.6 flags 70% of the notes in our main test set whose leaks are all harmless; the harmful output of phi_guard_3m_v0.7 flags 38%. Without it, reviewers sort harmless leaks from harmful ones by hand.
Raw
The raw output is the probability that a raw note contains any PHI, before de-identification. A note without PHI can skip the de-identification step. Without it, every note goes through de-identification, including the ones that hold no PHI.
Richness
The richness output grades how much PHI a note exposes on seven levels: none, trace, low, moderate, high, very high, and saturated (0 to 6). Tracked over batches of de-identified output, the level shows how much PHI is left and whether a change in the source data or in the pipeline moved it.

Two synthetic raw notes scored by the v0.7 guard.
Seven models for different speed and scope

The seven PHI Guard models. Source: Medical LLM page.
The v0.6 models answer one question each. The three v0.6 leak models read de-identified text and return one score; phi_guard_3m_v0.6_masked scores almost the same as phi_guard_3m_v0.6. phi_guard_3m_v0.6_raw reads raw text, and phi_guard_3m_v0.6_richness reads raw or de-identified text and returns six outputs, the probability that the exposure level is at least 1, 2, up to 6. The two v0.7 models answer all four questions in nine outputs: leak and harmful leak for de-identified text, the six richness outputs for either, and raw for raw text. phi_guard_870k_v0.7_fast returns the same nine about five times faster and is a little less exact than phi_guard_3m_v0.7.
All seven run on a CPU with ONNX and need about 1 GB of RAM. A guard reads a long note in overlapping windows and returns one decision for the whole note; the Healthcare NLP library scores documents up to 2,000,000 UTF-8 bytes.
Send the guard masked notes
PHI Guard reads masked or raw text. Send it the masked output of de-identification, where PHI is replaced by tags such as <PATIENT> or [NAME]. An obfuscated copy replaces PHI with realistic fake names, dates, and numbers, and to the guard those look like real PHI and raise false flags.
Note 1 above is a synthetic raw note after masking with the John Snow Labs de-identification pipeline in Healthcare NLP. Masking, checking, and the decision model all come from John Snow Labs. For pipeline choices and speed and accuracy figures across infrastructures, see Clinical de-identification at scale.
All patient data shown is synthetically generated for illustration.
Results on internal hard cases
We scored the models on internal test sets of hard cases: a main set of de-identified clinical notes, held-out notes, and long notes, plus the mean of four public datasets for leak detection. The internal sets are not published. The DecisionModel notebook shows how to load the seven models and run them on your own notes, and the full tables are on the Medical LLM page. We report these measures, with the cut-off at 0.5 wherever we count flagged notes:
- Leak detection: PR-AUC with a leak as the positive class, and the number of clean notes wrongly flagged on the main set.
- Harmful leaks: PR-AUC for the harmful class, and the share of notes with only harmless leaks that a model still flags. The v0.6 models have no harm output, so their leak score stands in.
- Raw notes: PR-AUC on a test split and on held-out notes, plus false positives and false negatives on the test split.
- Richness: mean absolute error (MAE) of the predicted level, lower is better, and rank correlation (rho), higher is better.
- Speed: one CPU thread with a 1 KB note for the guards, and batched on a 24 GB GPU for Medical DM Small.
Leak detection

Leak detection PR-AUC by test set, and clean notes wrongly flagged at the 0.5 cut-off on the main set. Public sets are the mean of four public datasets. Source: Medical LLM page.
Leak detection PR-AUC by test set, and clean notes wrongly flagged at the 0.5 cut-off on the main set. Public sets are the mean of four public datasets. Source: Medical LLM page.
phi_guard_3m_v0.7 reaches a PR-AUC of 0.986 on the main set and matches or slightly beats phi_guard_3m_v0.6 on every set, so folding the leak, raw, and richness models into one model lost no leak accuracy. Between the two fast models, phi_guard_870k_v0.7_fast flags 113 clean notes on the main set against 384 for phi_guard_850k_v0.6_fast, and scores lower on the public sets (0.679 against 0.733).
Medical DM Small, our GPU decision model, has the best scores on held-out and long notes and flags the fewest clean notes, 39 against 91 for phi_guard_3m_v0.7. It takes 40 ms per note on a GPU, against 6.4 ms on one CPU thread for the guard.
Every model scores between 0.679 and 0.753 on the public sets, against 0.960 to 0.986 on the main set. When your release process has to catch nearly every leak, set a lower cut-off on the leak probability, chosen on a sample of your own de-identified notes.
Harmful leaks

Share of notes with only harmless leaks that each model flags at the 0.5 cut-off on the main set; lower is better. The v0.6 model has no harm output, so its leak score stands in.
Harmful PR-AUC is higher with the harmful output on every set. On the main set, the phi_guard_3m_v0.6 leak score reaches 0.962, against 0.980 for phi_guard_3m_v0.7 and 0.977 for phi_guard_870k_v0.7_fast. On held-out notes the three reach 0.779, 0.883, and 0.885, and on long notes 0.904, 0.928, and 0.936.
Raw and richness

Raw-note PHI detection: PR-AUC on the test split and on held-out notes, and errors at the 0.5 cut-off on the test split.

Richness: level error (MAE, lower is better) and rank correlation (rho, higher is better) on masked notes; the raw columns are held-out raw notes.
Public benchmark
We also scored the leak output of phi_guard_3m_v0.6 and phi_guard_850k_v0.6_fast on the test split of the public MedDeID English synthetic benchmark (CC BY 4.0). Its 300 source notes give 600 examples: in 300, every PHI span is masked; in the other 300, exactly one PHI span is left in. One leaked value in an otherwise masked note is the hard case for a gate.
A gate has to catch nearly every leak, so we report precision at a fixed recall. At the cut-off that catches 98% of the leaks, 294 of 300, phi_guard_3m_v0.6 flags 159 of the 300 clean notes for review, a precision of 0.649. phi_guard_850k_v0.6_fast reaches the same recall with 244 false alarms and a precision of 0.546. At 95% recall, precision is 0.743 and 0.604.
The masking in this benchmark is idealized: the gold PHI spans were masked directly, without a de-identification pipeline, so results on the output of a production pipeline will differ. The cut-offs were chosen on the test set after scoring. They show the trade-off between missed leaks and review load, and are not recommended thresholds.
Benchmark: Hellemans, S., Stroobants, T., Scheurwegs, E., Meysman, P., Jorens, P., and Laukens, K. (2026). MedDeID English synthetic clinical corpus, benchmark and annotation guideline (v3). Zenodo. doi.org/10.5281/zenodo.22689857. Paper: Hellemans, S., et al. (2026). MedDeID enables locally governed clinical-text de-identification from real or synthetic training data. arXiv:2609.10049. Licensed under CC BY 4.0; we built the leak examples by masking the gold PHI spans.
Speed

Time per 1 KB note and per million notes, log scale, measured on one CPU thread with ONNX for the guards and batched on a 24 GB GPU for Medical DM Small.
Time per 1 KB note and per million notes, log scale, measured on one CPU thread with ONNX for the guards and batched on a 24 GB GPU for Medical DM Small.
Per million notes, the two fast models take 20 and 21.7 minutes, the other four v0.6 models 1.5 hours each, phi_guard_3m_v0.7 1.8 hours, and Medical DM Small about 10 to 12 hours. phi_guard_3m_v0.7 returns nine outputs in 6.4 ms. The single-purpose v0.6 models for leak, raw, and richness take 5.3, 5.4, and 5.4 ms, or 16.1 ms if you run all three on the same note. In a Spark pipeline, DecisionModel scores the partitions of a DataFrame in parallel, so a cluster scores many notes at the same time.
A long note is read in overlapping windows; in the Docker image they are 1,024 bytes. In a synthetic 3,028-character note with a patient name and record number added at the very end, the v0.7 guard read four windows and returned a leak score of 1.0.
The ensemble sends disagreements to a reviewer
The ensemble checks each note with the v0.7 guard and with Medical DM Small, combines the two scores (by default, the mean of their logits), and routes the note: leak when both models flag it, review when only one does, and clean when neither does.

Eight synthetic notes screened by the ensemble of the v0.7 guard and Medical DM Small.
In a screen of eight synthetic notes, both models catch four leaks, three clean notes pass, and one note goes to review: note 3 from the start of this post. The guard flags it at 0.9986, Medical DM Small scores it 0.0090, and a person makes the release call. Very confident scores round to 1.0. To rank the flagged notes for review, sort on the logit fields in the response.
Run PHI Guard in Healthcare NLP or Docker
The seven PHI Guard models are in the Healthcare NLP library and on Models Hub, from Healthcare NLP 7.0.1. Load a model with DecisionModel. The v0.6 models answer their one question with no extra setting. The two v0.7 models answer one question per pipeline stage, chosen with setMode: raw, leak, harmful, or richness; for a second mode, add a copy of the stage with its own output column.
from pyspark.ml import Pipeline
from sparknlp.base import DocumentAssembler
import sparknlp_jsl
from sparknlp_jsl.annotator import DecisionModel
spark = sparknlp_jsl.start()
notes = spark.createDataFrame([
("masked", "[NAME] visited [HOSPITAL] on [DATE]."),
("leaked", "Pt Maria Delgado, DOB 03/14/1956."),
], ["case_id", "text"])
doc = DocumentAssembler().setInputCol("text").setOutputCol("document")
guard = DecisionModel.pretrained("phi_guard_3m_v0.7", "en", "clinical/models") \
.setMode("leak") \
.setInputCols("document") \
.setOutputCol("phi_leak")
Pipeline(stages=[doc, guard]).fit(notes).transform(notes).selectExpr(
"case_id", "phi_leak[0].result AS label", "phi_leak[0].metadata.status AS status"
).show(truncate=False)
In leak mode, a scored note gets PHI_LEAK or NO_PHI_LEAK, and the metadata holds the probability: phi_leak_prob in leak mode, harmful_leak_prob in harmful mode, phi_present_prob in raw mode, and level with score_1 to score_6 in richness mode. Every result also has a status: scored for a model decision, empty for an empty document, which gets a negative label without running the model (UNKNOWN in richness mode), and overlength for a document above the size limit, which gets REVIEW_REQUIRED and no probability. Filter on status equal to scored before you release documents on a negative label. A NO_PHI_LEAK label does not certify that a note is safe to release.
To gate a de-identification pipeline, add the leak stage after it and point it at the masked output, then keep the scored documents labeled NO_PHI_LEAK and send the rest to review. Here raw_notes is a DataFrame with the original notes in a text column:
from pyspark.ml import PipelineModel
from pyspark.sql import functions as F
from sparknlp.pretrained import PretrainedPipeline
from sparknlp_jsl.annotator import DocumentFiltererByClassifier
deid_pipeline = PretrainedPipeline(
"clinical_deidentification_docwise_benchmark_optimized_v2", "en", "clinical/models"
)
phi_leak = DecisionModel.pretrained("phi_guard_3m_v0.7", "en", "clinical/models") \
.setMode("leak") \
.setInputCols(["masked"]) \
.setOutputCol("phi_leak")
deid_with_guard = PipelineModel(stages=list(deid_pipeline.model.stages) + [phi_leak])
result = deid_with_guard.transform(raw_notes)
scored = result.filter(F.col("phi_leak")[0]["metadata"]["status"] == "scored")
released = DocumentFiltererByClassifier() \
.setInputCols(["masked", "phi_leak"]) \
.setOutputCol("released_documents") \
.setWhiteList(["NO_PHI_LEAK"]) \
.setCaseSensitive(False) \
.transform(scored)
For on-premises deployment, the two v0.7 PHI Guard models also run in the johnsnowlabs/jsl-llms Docker image, version 0.33.0 and later, next to Medical DM Small:
docker run -d \ --gpus all \ --env "SPARK_NLP_LICENSE=your_license_key" \ -p 8080:8080 \ --ipc=host \ johnsnowlabs/jsl-llms:0.33.0 \ --model Medical-DM-Small \ --guards all \ --port 8080
POST /v1/guards takes up to 16,384 documents and 50 MiB per request, each with an id and a text. POST /v1/guards/ensemble takes up to 64 documents per request. The endpoints and request formats are on the on-premise deployment page.
Check every de-identified note before release
PHI Guard adds a release check after de-identification: every note gets a leak probability, a harmful-leak probability, and an exposure level, and raw notes without PHI can skip masking. It runs on a CPU inside the same Spark pipeline as the John Snow Labs de-identification pipeline, at 6.4 ms per note on one CPU thread for phi_guard_3m_v0.7. With Medical DM Small in the ensemble, each note gets a route: leak, clean, or review by a person. Run the DecisionModel notebook, or book a call to test PHI Guard on a sample of your own masked notes.
Frequently asked questions
What does PHI Guard do? PHI Guard is a family of seven John Snow Labs models that score clinical notes for protected health information. They tell you whether a de-identified note still shows PHI, whether that leak is harmful, how much PHI a note exposes on a scale from 0 to 6, and whether a raw note contains PHI at all. All seven run on a CPU in about 1 GB of RAM.
Why not review a sample of de-identified notes by hand? A sample leaves the other notes unchecked. PHI Guard scores every note, so reviewers see only the notes it flags.
How accurate is PHI Guard at finding PHI left after de-identification? On our internal hard cases, phi_guard_3m_v0.7 reaches a PR-AUC of 0.986 on the main de-identified test set, 0.982 on held-out notes, and 0.933 on long notes. On the mean of four public datasets it scores 0.753, so measure it on a sample of your own notes before you set a threshold.
Which PHI Guard model should I use? phi_guard_3m_v0.7 answers all four questions in one call at 6.4 ms per note. phi_guard_870k_v0.7_fast returns the same outputs at 1.3 ms per note with slightly lower accuracy, and flags 113 clean notes on the main set against 91. The single-purpose v0.6 models answer one question each.
What is a harmful PHI leak? The v0.7 models treat a name, an ID, a contact detail, a date of birth, a street address, an age over 89, or three or more harmless details together as harmful. A clinician name, a city or state, a hospital, or an age up to 89 on its own is harmless. On the main set, phi_guard_3m_v0.7 flags 38% of harmless-only leaks, against 70% for the v0.6 leak score.
Can I send obfuscated notes to PHI Guard? No. Send masked notes, where PHI is replaced by tags such as <PATIENT> or [NAME], or raw notes. Obfuscated notes replace PHI with realistic fake values, which look like real PHI to the guard and raise false flags.
How fast is PHI Guard on a large corpus? phi_guard_3m_v0.7 takes 6.4 ms per 1 KB note on one CPU thread, about 1.8 hours per million notes. phi_guard_870k_v0.7_fast takes 1.3 ms per note, about 21.7 minutes per million notes on one thread.
What does the ensemble with Medical DM Small add? The ensemble scores each note with the v0.7 guard and with Medical DM Small and routes it as leak, review, or clean. Notes where the two models disagree go to a person. Medical DM Small flags 39 clean notes on the main set against 91 for phi_guard_3m_v0.7, at 40 ms per note on a GPU.





























