Skip to main content

Clinical de-identification: PHI removal across every modality

Patient Journey Intelligence maintains two synchronized versions of your clinical data from the moment ingestion begins: an identified OMOP CDM dataset and a continuously updated, de-identified OMOP CDM dataset. Research pipelines, AI agents, cohort queries, and registry workflows route to the de-identified dataset by default, with no separate de-identification step, no manual export, and no lag between ingestion and research availability.

De-identification UI showing original and de-identified versions of DICOM, PDF, and plain-text clinical documents side by side, with associated PHI detection statistics.

Why a live de-identified dataset matters for researchers​

In most healthcare organizations de-identification is a project-level task. A researcher requests data, someone runs a de-identification job, review takes days, and a frozen extract is delivered. That extract is stale on arrival: a new encounter, a new lab result, or a new note that changes the patient's clinical picture is not in it.

Here, de-identification runs as an integrated step of the ingestion pipeline, the same pipeline that builds your OMOP CDM. The de-identified dataset is therefore a live mirror of the identified one, updated on every ingestion cycle, and research agents and AI pipelines querying it work with current data.

The practical consequences are specific. A clinical trial matching agent needs to know whether a patient meets eligibility criteria today, not when someone last ran an export. A pharmacovigilance pipeline monitoring post-market safety signals works from cohorts that move weekly. A quality improvement workflow running monthly on frozen extracts misses the patients who deteriorated between runs.

A live de-identified dataset also simplifies access control. Secondary-use queries go to the de-identified dataset by default, and access to the identified dataset requires explicit elevated permission with a documented purpose. The alternative is governing which researchers may touch PHI-containing data and auditing every query against a record of PHI exposure.

Performance at production scale​

Clinical de-identification is harder than it looks. A general-purpose model that finds names and dates in news text performs poorly on clinical notes, where PHI appears in non-standard forms: pager numbers embedded in free-text instructions, age references above the HIPAA 89-year threshold buried in a social history, facility names abbreviated in one note and spelled out in the next.

A peer-reviewed evaluation published at the Text2Story Workshop (ECIR 2025) measured PHI detection accuracy across healthcare-specific and general-purpose models. Healthcare-specific small language models reached 96% F1 on PHI detection. GPT-4o reached 79%, AWS Comprehend Medical 83%, and Azure 91%. The healthcare-specific models also ran at over 80% lower cost per document, which is the margin that makes processing at population scale economically viable.

The production validation behind these numbers is concrete. Providence, one of the largest health systems in the United States, de-identified 2 billion patient notes using John Snow Labs' Healthcare NLP, with zero re-identifications confirmed in a peer-reviewed evaluation. That scale and that outcome define what regulatory-grade de-identification means.

De-identification architecture: how the pipeline works​

The de-identification service processes every document type through a purpose-built path. Routing happens automatically on content type: you configure a profile and the pipeline handles the rest.

Profile-based clinical de-identification workflow across text, DICOM headers and pixels, PDFs, and whole-slide images

Profile-based de-identification workflow. The service validates incoming requests, routes workloads by profile and payload type, and processes unstructured text, DICOM imaging, PDFs, and whole-slide images through the appropriate de-identification path before returning inline JSON results or storing versioned outputs for downstream use.

Unstructured text, meaning clinical notes, discharge summaries, radiology reports, and pathology narratives, goes through healthcare-specific NLP models that detect all 18 HIPAA Safe Harbor identifier categories. PHI spans are replaced, suppressed, or pseudonymized according to the active profile.

PDFs receive combined text-layer and OCR coverage. The pipeline handles digital PDFs with embedded text and scanned PDFs where text was never selectable, including the multi-column layouts and degraded scan quality common in legacy document archives.

DICOM studies require two separate passes. Header-level de-identification removes or replaces PHI in DICOM metadata tags: patient name, date of birth, MRN, referring physician, institution, acquisition timestamps. Pixel-level de-identification uses a vision-language model to detect identifiers burned into image pixels, such as patient demographics and facility names overlaid on the image by the acquisition device, which header-only de-identification cannot reach.

Whole-slide images (histopathology, cytology) receive the same pixel-level treatment, with spatial detection tuned for the label regions and annotation overlays that typically carry identifying information.

Across all modalities, consistent pseudonymization links a patient's records across documents and time. The pseudonymized identifier is stable within a dataset, so a patient's notes, labs, imaging, and claims all resolve to the same pseudonym and longitudinal analytics on the de-identified dataset stay coherent. Date shifting applies a patient-specific offset that preserves intra-patient temporal relationships, so the gap between a diagnosis and the first treatment visit stays accurate, while preventing date-correlation attacks across patients.

Key capabilities​

PHI detection in clinical text

All 18 HIPAA Safe Harbor identifier categories detected in free-text notes, reports, and narratives using healthcare-specific NLP models. 99% accuracy, validated at 2 billion notes.

PHI detection in PDFs

Combined text-layer and OCR coverage for digital and scanned PDFs. Handles multi-column layouts, degraded scan quality, and historical document formats.

DICOM header de-identification

Tag-level PHI removal across all standard DICOM SOP classes, including burned-in annotation text in structured report objects.

DICOM pixel de-identification

Vision-language model detection and removal of identifiers burned into image pixels, the modality where header-only de-identification fails.

Whole-slide image de-identification

Pixel-level PHI detection in histopathology and cytology slides, tuned for label regions and annotation overlays.

Cross-document pseudonymization

Patient pseudonyms consistent across all documents and modalities, preserving longitudinal linkage through the de-identified dataset.

Patient-specific date shifting

Dates shifted by a patient-specific offset preserving intra-patient temporal relationships while preventing date-correlation re-identification.

Configurable de-identification profiles

HIPAA Safe Harbor, HIPAA Expert Determination, and GDPR pseudonymization. Profile selection is configurable per project without pipeline changes.

The de-identification dashboard​

De-identification runs automatically during ingestion to maintain the live de-identified OMOP dataset. You can also run custom de-identification jobs on specific datasets or documents, on demand and in parallel to the ingestion pipeline. The dashboard gives you real-time visibility into all de-identification activity across the platform, automated and manually triggered alike.

Patient Journey Intelligence de-identification dashboard showing job status indicators, PHI detection metrics, and the job table

De-Identification Dashboard showing system-wide job status indicators, PHI detection metrics, and the full job table with profile, workflow type, and per-job actions.

Six system-wide indicators track status at a glance:

  • Total: Cumulative jobs executed, broken down by content type (unstructured, imaging, mixed, PDF)
  • Completed: Successfully finalized workflows
  • Ready to Review: Jobs where processing is complete and results are awaiting manual validation before promotion
  • In Progress: Active anonymization processes currently running
  • Pending: Jobs queued but not yet started
  • Failed: Jobs with processing or rule application errors requiring attention

The job table below the indicators shows every active and historical job with the profile applied, workflow type (cohort-based or source-based), PHI detection metrics, processing timestamps, and direct links to view details, export results, or edit the profile for reuse.

Running a de-identification job​

To start a new job, click New De-Identification and follow three steps.

Choose your data source. Scope the job to a defined cohort, meaning a patient-level collection already built in the platform, or to a source such as an external PACS or cloud imaging store. Cohort-based jobs suit a specific patient population. Source-based jobs suit incoming data streams processed continuously.

Select an anonymization profile. The profile determines which PHI categories are removed, how each is handled (drop, replace, hash, or shift), and which regulatory standard the output satisfies. Standard profiles cover HIPAA Minimal Safe Harbor, GDPR-compliant pseudonymization, and clinical trial redaction policies. Custom profiles can be created and cloned for organization-specific or study-specific requirements.

Confirm and run. The pipeline scans every document in scope for PHI in both metadata and content, applies the profile rules, and generates a per-document outcome record. For DICOM studies, every slice is scanned independently.

Running de-identification on Databricks​

For large document volumes, the de-identification step of an ingestion job runs on a Databricks cluster instead of local compute. The choice is made per job at configuration time.

When a job is configured for Databricks de-identification, the platform uses the Databricks credential configured in Administration. Documents are sent to the cluster for processing and the de-identified results are written back. No documents are permanently retained on the Databricks cluster, which holds only the active job.

The same profiles and profile rules apply on either compute. The output, the audit record, and the de-identified OMOP dataset update identically, so a 90,000-document backfill on a cluster and a 400-document weekly job run locally produce records an auditor reads the same way.

Reviewing results: the job detail interface​

Selecting a completed job opens a multi-panel interface for review, validation, and export.

Patient panel​

A filterable sidebar lists every patient in the job with age, sex, total documents, and number of imaging studies. Selecting a patient updates the main viewing area with that patient's documents and de-identification statistics.

Split view: side-by-side quality assurance​

The dual-pane viewer is the primary tool for verifying de-identification quality across unstructured text, PDFs, DICOM studies, and whole-slide images. The original document appears on the left and the de-identified version on the right. For DICOM studies, both panes support slice navigation, zoom and pan, window and level controls, and synchronized scroll. PHI regions detected in the original are highlighted, and metadata tag overlays are available for imaging modalities.

The side-by-side layout is how you confirm that names, dates, and identifiers present in the original were removed or replaced in the output, in every format the pipeline processes.

DICOM metadata inspector​

The metadata inspector displays the original and de-identified DICOM headers side by side, with a change summary showing which tags were modified, masked, or removed. The categories reviewed are patient identifiers, study and series descriptors, institutional metadata, and timestamps and acquisition details.

De-identification statistics panel​

Every job produces a structured summary of anonymization performance per study:

General metrics: total slices processed, slices with detected PHI, and overall detection rate.

Pixel-based PHI breakdown: detected instances, cleared (anonymized), rejected, and pending manual review.

Entity breakdown: PHI detected by category, including name, MRN, institution, and dates, which shows where your data carries the most PHI and whether any category has an unexpectedly low detection rate.

Metadata PHI: field-level enumeration of PHI tags with anonymization status for each DICOM tag.

Outcome summary: successful (all PHI addressed per profile), failed, or requires review.

Pipeline metadata: the profile used, execution time, processor ID where applicable, and final status, which are the fields reproducibility and audit documentation require.

Adding an entity the pipeline missed​

A reviewer who spots a PHI span in the original document that the pipeline did not flag, such as an unusual name format, an institution abbreviation, or a locally specific identifier, can add it from the review interface.

Select the text in the document viewer, assign it an entity type (Name, Date, ID, Institution, or another type in the active profile), and confirm. The entity joins the job's annotation set and is treated like a detected entity: de-identified according to the active profile rules and included in the per-job audit record.

Manual additions appear in the audit trail beside model-detected entities, so the job's outcome documentation covers every PHI decision made on the job.

Example: a local ward abbreviation​

A reviewer working through a batch of transfer notes sees disch to 7WN left in the de-identified output, where 7WN is a ward name specific to the institution. They select it, tag it as Institution, and confirm. The span is de-identified under the active profile and the addition is recorded, and the pattern is a candidate for the profile before the next production run.

Reviewing failed documents​

When de-identification fails for a document, because the file could not be read, no text was found, or the pipeline produced no result, the review interface shows the failed document with its error instead of an empty slot.

A red banner marks the document as not de-identified and confirms that it still contains personal information. Selecting View error shows the reason the run recorded, and the error message can be copied for support or audit purposes.

For a failed imaging study, the viewer shows the original images in the standard viewer and lists the specific instances that failed. For a document that partially succeeded, where some pages processed and others did not, the review shows the results that were produced alongside the same banner, so you can see how far the run progressed.

Detecting de-identification drift​

De-identification drift occurs when the PHI patterns in a document corpus change over time, through new identifier formats, new institution names, or new document templates, in ways the active profile does not cover. A profile that was accurate when first applied becomes incomplete as the corpus evolves, and without detection it does so silently.

Drift detection monitors the entity detection rate across successive de-identification runs against the same source. When detection rates fall outside expected bounds, or new PHI patterns appear that the active profile does not cover, the platform raises a drift alert on the job dashboard.

Treat a drift alert as a signal to review whether the active profile still covers the current document set. Open the flagged job, inspect which entities the pipeline is and is not detecting on the new documents, and extend or clone the profile to cover the new patterns before the next production run.

Example: a new referring-facility format​

A health system's outside-records feed starts carrying documents from a newly acquired clinic whose header writes the facility as SVMC-NORTH rather than a spelled-out name. The institution detection rate on that source drops from 4.1 entities per document to 2.6, and the dashboard raises a drift alert. The profile is extended to cover the new pattern before the next monthly run of 50,000 documents.

Export and data persistence​

Once results are reviewed and the quality is confirmed, anonymized data can be:

  • Exported locally for research or regulatory submission
  • Saved to internal Patient Journey Intelligence storage for downstream platform use
  • Linked to OMOP CDM to enable imaging analytics alongside the structured clinical dataset

A confirmation banner summarizes the number of exported studies, series, and slices.

Managing de-identification profiles​

Profiles are the governance layer that converts your regulatory requirements into pipeline behavior. Each profile defines tag-level rules (drop, replace, or hash per field), pixel-level detection thresholds, pattern recognition settings, and the regulatory compliance target.

Administrators can create custom profiles, clone existing ones for study-specific variations, preview rule effects on sample data before applying them at scale, and temporarily disable profiles without deleting them. The standard profile set covers:

Best practices for production de-identification​

The practices below separate a de-identification workflow that holds up in an audit from one that generates re-review requests, compliance gaps, and research delays.

Validate profiles on pilot datasets before production-scale use. Before applying a profile to millions of documents, run it against a representative sample that includes the edge cases: scanned PDFs with degraded quality, DICOM studies with burned-in annotations, notes with non-standard abbreviations. A pilot run surfaces detection gaps while the correction costs an afternoon, rather than after a full patient cohort has been processed and needs a rerun.

Use the split view to verify pixel-level anonymization as well as metadata. The metadata inspector confirms that DICOM header tags were modified. The split viewer confirms that text burned into image pixels was removed from the image itself. These are two different checks and both matter. For imaging modalities, make pixel verification a required step before results are promoted, because it is the validation the metadata inspector cannot provide.

Review DICOM headers for residual identifiers after each new source is onboarded. Different imaging devices and PACS systems write PHI into different tag sets, and a profile tuned for one manufacturer's scanner may miss proprietary tags written by another. The first time you ingest from a new source, open the metadata inspector and scan the change summary for tags the active profile did not address. Update the profile before scaling ingestion from that source.

Align de-identification rules with study-specific requirements before a project starts. HIPAA Safe Harbor, Expert Determination, GDPR Article 89, and clinical trial redaction policies each impose different constraints, and the right choice depends on the downstream use. A retrospective cohort study may need only Safe Harbor. A clinical trial regulated under ICH GCP needs sponsor-specific redaction rules. Choosing the profile after data has been de-identified means reprocessing. Choosing it upfront is a configuration step.

Document the profile used for each project and keep that record with the data. When a researcher submits findings to a journal, or a sponsor asks about de-identification methodology during an audit, the answer has to be specific: which profile, which version, which pipeline run, and what the outcome statistics were. The per-job audit record captures all of this automatically, as long as you preserve the association between the job and the dataset or publication it produced.

Patient Journey Intelligence's De-Identification module removes Protected Health Information from clinical notes, PDFs, DICOM studies, and whole-slide images, then maintains a continuously synchronized de-identified OMOP CDM dataset alongside the identified one. Research pipelines and AI agents query the de-identified dataset by default, and it is updated on every ingestion cycle rather than on demand.

Healthcare-specific NLP models in Patient Journey Intelligence reach 96% F1 on PHI detection in clinical text, against 79% for GPT-4o, 83% for AWS Comprehend Medical, and 91% for Azure, in a peer-reviewed evaluation published at the Text2Story Workshop (ECIR 2025). At Providence, the platform de-identified 2 billion patient notes with zero re-identifications, confirmed in a peer-reviewed study.

The platform de-identifies structured data (csv, json, FHIR), unstructured clinical text (notes, reports, summaries), scanned and digital PDFs, DICOM imaging studies at both header and pixel level, and whole-slide images. Each modality has a purpose-built processing path. DICOM pixel de-identification, for example, uses a vision-language model to detect identifiers burned into image pixels, which header-only processing cannot reach.

Each patient is assigned a stable pseudonym at the start of ingestion, and that pseudonym is used consistently across all documents, modalities, and time periods in the de-identified dataset. Notes, labs, imaging studies, and claims all link to the same pseudonym, so longitudinal analytics stay coherent without exposing the identified patient record.

Standard profiles include HIPAA Minimal Safe Harbor (all 18 categories removed), HIPAA Expert Determination (more granular data retained with statistical certification), GDPR pseudonymization for Article 89 research, and clinical trial redaction for ICH GCP compliance. Custom profiles can be created, cloned, and previewed on sample data before production deployment.

Dates are shifted by a patient-specific offset, and the same offset applies to every date for a given patient. Temporal relationships inside a patient's record are preserved, so the gap between a diagnosis date and a first treatment date stays accurate. Because each patient's offset differs, date-correlation re-identification across patients is prevented.

Yes. Select the text in the document viewer, assign an entity type such as Name, Date, ID, or Institution, and confirm. The span joins the job's annotation set, is de-identified under the active profile rules, and is recorded in the per-job audit trail alongside model-detected entities.

Drift is a change in the PHI patterns of a corpus over time, such as new identifier formats, institution names, or document templates that the active profile does not cover. Drift detection monitors the entity detection rate across successive runs against the same source and raises an alert on the job dashboard when rates fall outside expected bounds or new patterns appear. Review the flagged job and extend or clone the profile before the next production run.

Yes. The de-identification step of an ingestion job runs on a Databricks cluster instead of local compute, selected per job at configuration time, using the Databricks credential configured in Administration. The same profiles and rules apply on either compute, no documents are permanently retained on the cluster, and the output, audit record, and de-identified OMOP dataset update identically.

The review interface shows the document with a red banner marking it as not de-identified and still containing personal information. View error shows the reason the run recorded, and the message can be copied for support or audit. A failed imaging study shows the original images with the specific failed instances listed, and a partially processed document shows the results that were produced alongside the banner.

De-identification runs as an integrated step of the standard ingestion pipeline rather than as a separate export. Every ingestion cycle updates the identified and de-identified datasets in parallel, so research pipelines and agents querying the de-identified dataset see current data.

Each job generates a per-document outcome record: the profile used, PHI entities detected by category, pixel-level results per slice, execution time, and final status, with manually added entities recorded alongside model detections. This documentation supports HIPAA compliance documentation, IRB reporting, and regulatory submissions that require evidence of de-identification methodology and outcome.