Skip to main content

Patient Journey Intelligence 2.4.0: AI case finding, agentic de-identification, and scalable ingestion

Release 2.4.0 is a major platform milestone. It unifies specialized clinical registry workflows with generic, domain-agnostic registries, strengthens enterprise de-identification accuracy, fixes critical workflow defects, and hardens ingestion and database scalability for large patient cohorts.

New features and enhancements​

Patient Registry and automated case finding​

Two-pass AI case finding framework​

  • Screening pass: Rapidly evaluates cohorts of up to 40,000 patients. It assigns preliminary eligibility scores (0.0–1.0) and classification bands (Eligible, Needs Review, Not Eligible).

Case finding screening pass

  • Evidence pass (under 10 seconds per case): Automatically links extracted eligibility criteria to the exact supporting and contradictory passages in clinical notes, with document dates.

See Patient Registries and Case Finder for the full workflow.

Enterprise and Agentic De-Identification (De-ID)​

Agentic De-Identification console​

  • Connect PJI directly to the pipeline adaptation console to train and evaluate recall-first de-identification models on your own clinical datasets.

Agentic De-Identification console

  • Follow live adaptation runs cycle by cycle, inspect per-PHI-type metrics and error examples (with full audit logging), compare up to four runs side by side, and deploy adapted pipelines straight into PJI profiles via Use in De-ID.
  • Optional second-opinion checks using the KEV model and PHI guard help audit training labels, missed spans, and batch drift.

Multi-frame DICOM pixel masking and built-in standards​

DICOM de-identification split screen

  • Detects and masks burned-in PHI across every frame of multi-frame studies (ultrasound cine, breast tomosynthesis) with per-frame bounding boxes and configurable OCR resolution.
  • New imaging profiles default to the 198-tag Built-in MIDI-B / TCIA rules, delete private vendor tags when enabled, and stamp Patient Identity Removed = YES into output headers.

Spark DataFrame batching and PDF memory optimization​

PDF de-identification

  • Replaced serialized file-by-file processing with batch DataFrame transformations for C-CDA and tabular data, reducing runtime from about 7–13 seconds per file to 2 seconds or less.
  • Fixed PDF obfuscation so entities beyond the first 500 characters per page are masked, added an optional direct text-layer reader (PDF_DEID_USE_TEXT_LAYER=true) that cuts digital PDF runtime in half, and updated the PDF service to unload unused pipelines to prevent OOM restarts.

Clinical context guardrails and Safe Harbor accuracy​

  • Prevents lab values, units (mg/dL, 10^3/uL), reference ranges, test names (HbA1c, CA 15-3), section headers, and medical eponyms (Cushing syndrome, NKDA, LAD) from being falsely redacted as driver's licenses, cities, or hospitals.
  • Upgraded the built-in HIPAA Text Minimal profile to the medium v2 model with per-patient consistent date shifts, unified per-profile random seeds, state and 3-digit ZIP preservation, and full replacement of accession numbers, apartment/suite suffixes, SSN last-four phrases, and base64-encoded FHIR attachments.
  • Synchronizes de-identified patient demographics (tpj.person_personal_data) with their de-identified documents and allows runs to discard original PHI (keep_original_data=false).

Pre-run validation, archiving, drift, and Expert Determination​

  • Pre-flight checks catch missing files, expired cloud credentials, and unmapped table columns before a run starts, with plain-language explanations and optional skipping of non-PHI documents.
  • Test profiles directly against a single file, DICOM instance, or table preview from an external data source without ingesting it first.
  • Archive completed runs without breaking downstream datasets or drift baselines, measure Data Drift across all modalities (text, PDF, DICOM, FHIR, C-CDA, and tables), and generate unified Expert Determination reports covering mixed-modality runs.

See de-identification and de-identification best practices for how projects, profiles, and review work.

Clinical NLP, OCR, and Information Extraction (IE)​

HunyuanOCR and embedded GLM readers​

  • Integrated HunyuanOCR (via vLLM) to extract structured text and table crops from images, charts, ECG printouts, and radiology figures. Markdown table header parsing is fixed, so top lab rows (for example, Sodium) are preserved.
  • Embedded GLM directly inside CPU document processing containers (removing the separate OCR service dependency) and retired the unused Granite-Docling reader.

Fine-tuned GLiNER2 LoRA adapter for measurements​

  • Added a custom LoRA adapter to GLiNER2, raising measurement extraction F1 scores from 0.69 to 0.89 on validation cohorts. KEV sentence-splitting issues are resolved, so note vitals and labs are reliably captured.

LOINC concept resolution and unit normalization​

  • Resolves extracted clinical note vitals and labs against LOINC as the primary vocabulary (with SNOMED fallback), normalizes units (for example, lbs to kg, in to cm), splits blood pressure pairs (138/84) into systolic and diastolic measurements, and preserves multiple same-day readings of the same test.

Occurrence-aware Decision Gate and fact accounting​

  • Filters out non-occurring, hypothetical, or planned events (such as "scheduled for surgery tomorrow") from being recorded as active OMOP conditions, while preserving family history mentions (4210989).
  • Step 5 now tracks every extracted fact. It shows exact counts of saved records versus items skipped (and why) in the ingestion job's Batches view, and enables note and table measurement extraction by default.

Information Extraction and Terminology Server parity​

  • Standalone Information Extraction runs now match ingestion's Information Extraction logic, display ICD-10-CM and RxNorm codes with hover descriptions, and support optional Decision Model verification.
  • Upgraded the Terminology Server resolver to a streamlined 31 GB database and added Decision Model context scoring for term resolution and document diagnosis ranking.

See Information Extraction and medical terminology for details.

Ingestion, connectors, patient insights, and platform controls​

Production connectors and multi-stage ingestion​

  • Connectors for Epic EHR (with interactive patient search by name and date of birth, or by ID) and Azure Blob Storage.
  • Ingest the same files independently across multiple environment stages without cross-stage duplicate collisions, copy S3 and AWS HealthImaging studies locally into PJI storage so they remain accessible after external credentials expire, and route XML and JSON payloads automatically based on content rather than file extension.

Streamlined ingestion wizard, FHIR export, and job cleanup​

  • Raw store jobs automatically create a dataset upon completion and no longer create unintended patient records from raw HL7 files.
  • FHIR exports now support NDJSON output, direct upload to cloud buckets, and upsert pushes to external FHIR servers.
  • Rewrote job deletion to purge unreferenced patient demographics, eras, and staging schemas cleanly while preserving data shared with other active jobs.

See data ingestion and source registry for configuration details.

Patient Timeline, Problem List, and Clinical Measures​

  • A Patient Timeline tab on Patient Details lets you explore dated clinical events grouped by source format, event type, or data origin.

Patient Timeline tab

  • Upgraded the Diagnosis tab to a ranked Problem List with High, Medium, and Low salience filters, and updated Clinical Measures to display full historical timelines and cohort result counts.

Ranked Problem List

Grounded Assistant and cohort discovery​

  • Added SQL validation and claim-verification guards across the Assistant's SQL generator, Explain panel, multi-source summaries, and Thought Process panel to prevent unsupported numbers or cross-cohort leakage.
  • Added a Diagnosis Group cohort filter to match broad conditions (for example, all malignant neoplasms) in one selection.

Modular licensing, granular roles, and centralized LLM routing​

  • Enforced signed license scopes (pji:registry, pji:deid, pji:curation, pji:insights), added granular role capability switches, enabled personal OAuth sign-in for Claude and MCP clients (pji-claude), and unified LLM provider configuration inside Service Configuration.

Infrastructure, scaling, and reliability hardening​

PostgreSQL connection pooling and memory tuning​

  • Introduced PgBouncer connection pooling for the OMOP database, reused per-schema merge connections, and sized pools dynamically to container memory limits to eliminate connection exhaustion outages.

Databricks unified execution​

  • Consolidated Databricks De-ID and Information Extraction into the standard New Run and New Extraction wizards and the unified Results UI, with GPU cluster support and write-permission connection checks.

See Databricks deployment for configuration details.

Airflow, OpenSearch, and OMOP resilience​

  • Eliminated automatic Airflow scheduler pod restarts on immediate job submissions, switched OpenSearch clients to refreshable AWS IAM V4 credentials to prevent HTTP 403 errors on long pipelines, and added paginated checkpoint recovery for OMOP Data Modeling.

Unified model cache and deployment docs​

  • Consolidated all model downloads under a single host root, added Kubernetes installer flags for the Patient Registry service, and published deployment guides for k3s, EKS, and AKS.

Bug fixes and resolved issues​

De-identification and review workflow​

  • Review correction damage: Fixed a bug where saving manual review corrections appended the last character of the real value to the de-identified string (for example, Greenn).
  • Scrambled accepted values: Accepting a detected entity during review no longer gives it a scrambled replacement value.
  • Deleted run drift baselines: Deleting a run now purges its dependent drift measurements and Expert Determination reports, so deleted runs no longer act as active drift baselines.
  • Empty run exports: Fixed an issue where exporting de-identified document run results produced empty ZIP archives and blank result logs.
  • HL7 and table reviews: Restored the ability to accept or reject individual fields in HL7 reviews, view plain-text HL7 files in the text review tool, and save imported demo run reviews.
  • Databricks token failures: Fixed an issue where re-entering a Databricks authentication token continued to fail due to stale connection caching.

Data ingestion and Assistant​

  • Pipeline catalog recovery: Restored the empty pipeline catalog list seen after upgrading to Spark NLP 7.0 by pulling directly from the John Snow Labs Models Hub on build.
  • Source rename disappearance: Fixed a bug where renaming a data source caused its ingestion jobs to vanish from the job list. Job tracking no longer depends on display names.
  • Coding pipeline code loss: Jobs using ICD-10-CM and RxNorm coding pipelines no longer drop extracted classification codes.
  • Assistant query guard failures: Fixed edge cases where Assistant SQL queries returned empty results or drifted outside designated patient cohorts, by enforcing strict context boundary guards.

Patient Journey Intelligence 2.4.0 adds a two-pass AI case finding pipeline and domain-agnostic registries, an Agentic De-ID adaptation console, multi-frame DICOM pixel masking, HunyuanOCR and embedded GLM readers, a fine-tuned GLiNER2 LoRA adapter, LOINC-first lab resolution, Epic EHR and Azure Blob Storage connectors, a Patient Timeline tab, and PgBouncer connection pooling.

The screening pass evaluates cohorts of up to 40,000 patients and assigns each an eligibility score from 0.0 to 1.0 with an Eligible, Needs Review, or Not Eligible band. The evidence pass then links criteria to supporting and contradictory note passages in under 10 seconds per case.

It connects PJI to a pipeline adaptation console where you train and evaluate recall-first de-identification models on your own clinical data. You can follow runs cycle by cycle, compare up to four runs, and deploy an adapted pipeline into a PJI profile with Use in De-ID.

Yes. Burned-in PHI is detected and masked across every frame of multi-frame studies such as ultrasound cine and breast tomosynthesis. New imaging profiles default to the 198-tag built-in MIDI-B / TCIA rules.

Release 2.4.0 adds production connectors for Epic EHR, with patient search by name and date of birth or by ID, and for Azure Blob Storage.

PgBouncer now pools connections to the OMOP database, merge connections are reused per schema, and pool sizes follow container memory limits. This removes the connection exhaustion outages seen on large runs.