Demos
When Ohio State University built their Medical LLM infrastructure to process over 200 million clinical notes, the technical challenges extended far beyond preventing hallucinations. Their system required unified data ingestion consolidating structured electronic health record tables with unstructured clinical notes, HIPAA-approved de-identification with auditability, information extraction and coding handling inconsistent terminologies and missing fields, human-in-the-loop validation using Generative AI Lab, and portable configuration executing unchanged on both Azure Databricks and on-premise environments with consistent logging for compliance [Ohio State University Case Study, 2025].
This demonstrates why focusing exclusively on hallucinations misleads organizations about clinical artificial intelligence deployment requirements. Even perfectly factual large language model outputs fail when data quality is poor, clinical context is incomplete, workflow integration is inadequate, or governance frameworks are absent. The implementations processing millions of clinical documents at production scale reveal that hallucinations represent only one failure mode among many – and often not the most consequential.
Why data quality breaks clinical AI before hallucinations do
Providence St. Joseph Health’s de-identification of 2 billion patient notes demonstrates the data engineering foundation clinical artificial intelligence requires. Their system, built on proprietary medical language models and a modified version of Spark NLP, achieves 99% Protected Health Information (PHI) obfuscation and 100% masking or shifting of target data fields, satisfying the Expert Determination de-identification criteria under HIPAA Privacy Rules with less than 5% PHI prevalence both in aggregate and per record – a level of accuracy that surpasses triple manual review by three human annotators. An independent audit, including adversarial testing on 790 randomly selected patients by a dedicated red team working for three months, was unable to re-identifyany patient in the dataset [Kocaman et al., 2025a].
The data engineering challenge was not preventing hallucinations but handling real-world data heterogeneity: clinical notes with inconsistent formatting across departments, scanned documents requiring optical character recognition, legacy data with non-standard terminologies, missing fields where critical information was documented elsewhere, and temporal ambiguity about whether conditions are current versus historical. Without systematic data normalization, even factually accurate large language model outputs operate on incomplete or misleading inputs.
MiBA’s oncology data curation shows scale requirements. Processing 1.4 million physician notes and approximately 1 million PDF reports and scans, their system had to extract entities with 93% F1-score while handling diverse documentation patterns across primary care, oncology, pathology, and radiology sources. Electronic health records are a treasure trove for oncology data, but structured data lacks context while unstructured data is rich with insights yet infeasible to review manually at scale [MiBA Case Study, 2025].
The systematic assessment showing Healthcare NLP achieving 96% F1-score compared to GPT-4o’s 79% with GPT-4o missing 14.6% of entities versus Healthcare NLP’s 0.9% miss rate reflects both hallucination prevention and handling of real clinical data variability [Kocaman et al., 2025b]. General-purpose models trained on clean text struggle with messy clinical documentation that specialized systems handle systematically.
Why limited clinical context undermines even factually correct outputs
Roche’s oncology patient timeline construction demonstrates longitudinal context requirements. Their system applies healthcare-specific large language models to construct detailed patient timelines from diverse clinical notes: primary care providers, oncologists, discharge summaries, emergency departments, pathology reports, and radiology reports. This enables matching patients with National Comprehensive Cancer Network guidelines by analyzing genetic, epigenetic, and phenotypic information [Roche Case Study, 2024].
Clinical decision-making requires temporal understanding that isolated note analysis cannot provide. A chest pain mention requires knowing whether this is new presentation, follow-up after myocardial infarction, or chronic stable angina. Laboratory trends matter more than single values. Medication changes require understanding prior treatments and adverse reactions. Large language models operating on limited context windows miss critical signals even when they generate factually accurate text about the specific input they see.
TriNetX’s smoking status extraction demonstrates context complexity. Their pipeline scales from one site to many by fine-tuning models to overcome site-level variation in documentation patterns, creating structured, harmonized labels that bring consistency across networks [TriNetX Case Study, 2025]. Smoking status is rarely a simple current fact – it requires understanding whether the patient currently smokes, previously smoked, never smoked, or status is unknown. This contextual nuance determines appropriate screening recommendations, yet general-purpose models often miss these distinctions.
Why explainability and trust matter more than technical accuracy
Care-Connect and Spryfox’s framework demonstrates why transparency enables clinical deployment. Reaching regulatory-grade accuracy requires augmenting automated natural language processing with human-in-the-loop workflows using clinician-developed logic to filter and rank entities for manual review, reducing workload while maintaining high safety levels [Care-Connect & Spryfox Case Study, 2024].
The requirement is correctness and explainability: clinicians must be able to verify reasoning. Guideline Central content integrated into the Patient Journey Intelligence platform shows this pattern: every answer is grounded in guideline source material and cites the specific guideline passages behind it, letting clinicians trace how the response was produced and confirm it is correct – transparency that black-box outputs cannot provide [Guidelines Central Case Study, 2025]. When clinicians cannot trace how conclusions were reached or what evidence was considered, even correct suggestions may be rejected.
The CLEVER study showing medical doctors preferred specialized healthcare natural language processing 45-92% more often than GPT-4o on factuality, clinical relevance, and conciseness [Kocaman et al., 2025c] reflects trust based on consistent, explainable performance rather than occasional impressive outputs. Production clinical systems require reliability that enables trust over technical capability that impresses in demonstrations.
Why workflow integration determines adoption regardless of model quality
vCare Companion’s implementation demonstrates workflow integration requirements. Their robot helps hospital staff reduce administrative burden by over 3 hours per shift through ambient listening to patient-staff conversations for charting, automated reporting, vitals collection, and integration with electronic health records at point of care. The system autonomously follows care staff with necessary tools, listens to conversations, automatically fills forms, and integrates with electronic medical records – freeing hands and time to focus on patient care delivery [vCare Companion Case Study, 2025].
The design principle is workflow-first: technology serves operational needs rather than seeking adoption for its own sake. Even technically perfect artificial intelligence tools fail when they create additional documentation burden, generate alerts requiring manual triage, produce outputs in formats incompatible with existing systems, or require clinicians to learn new interfaces disrupting established patterns.
Intermountain Health’s medical text summarization reducing review time from 10 minutes to 3 minutes per document, achieving 70% efficiency gain, demonstrates successful integration: automating existing workflow steps rather than adding new tasks [Intermountain Health Case Study, 2024]. The 70% reduction comes from fitting directly into document review processes clinicians already perform.
Why bias and fairness auditing prevent systemic harm at scale
The systematic assessment on 48 medical expert-annotated clinical documents showing Healthcare NLP missing only 0.9% of entities versus GPT-4o missing 14.6% has fairness implications beyond aggregate accuracy [Kocaman et al., 2025b]. If missed entities cluster in documentation patterns associated with specific patient populations – non-English speakers, certain socioeconomic groups, or particular clinical settings – model performance appears adequate overall while systematically failing for vulnerable populations.
TriNetX’s site-level validation demonstrates fairness requirements: their pipeline fine-tunes models to overcome institutional documentation variation, creating harmonized labels bringing consistency across networks [TriNetX Case Study, 2025]. Without this validation across diverse sites, models optimized for well-resourced academic medical centers may perform poorly in community hospitals or rural settings, exacerbating rather than reducing healthcare disparities.
Organizations must validate model performance across demographic groups, documentation practices, and clinical settings – ensuring artificial intelligence does not amplify existing inequities. This requires end-to-end testing that many pilots skip in favor of demonstrating impressive results on convenient datasets.
Why governance and compliance infrastructure enables sustainable deployment
Dandelion Health’s de-identification process demonstrates governance requirements beyond technical capability. Their approach requires HIPAA compliance through expert determination, breaking down different note types by risk level and adapting processesaccordingly, assessing subtypes to carve out exceptions, determining appropriate balance of recall and precision, and applying pre-processing enhancements – all while maintaining full provenance for regulatory review [Dandelion Health Case Study, 2024].
Martlet.ai’s RADV audit preparation shows audit trail requirements [Martlet.ai Case Study, 2025]. Their platform maintains full provenance for every assertion, links each HCC to verifiable, date-bounded, provider-appropriate documentation, enforces evidencesufficiency, and produces CMS-ready export packets. This level of auditability enables retrospective review when outcomes are questioned or regulatory audits occur.
Ohio State’s infrastructure includes human-in-the-loop validation using Generative AI Lab, HIPAA-approved de-identification, and consistent logging for monitoring and compliance [Ohio State University Case Study, 2025]. This governance framework makes production deployment sustainable rather than creating liability exposure that eventually forces system retirement.
Why infrastructure and maintenance determine long-term viability
Providence’s de-identification of 2 billion patient notes demonstrates computational infrastructure requirements at a scale few organizations attempt [Kocaman et al., 2025a]. The system is built on Apache Spark and a modified version of Spark NLP – a distributed computing framework engineered for efficient execution across large clusters of commodity hardware – letting the pipeline batch-process the entire historical archive cost-efficiently while also keeping pace with roughly 200,000 new notes added daily. Because the default Spark architecture lost efficiency and hardware utilization as cluster size grew during batch inference, the team built a custom node and workload manager that takes over master-node duties and orchestrates smaller batches across independent machines, achieving linear scaling to a theoretically unlimited number of worker nodes while keeping every machine at 95-100% utilization throughout each run. This combination of distributed architecture and purpose-built scaling infrastructure, not model accuracy alone, is what makes billion-note throughput, automated data pipelines, real-time monitoring, error detection, and systematic quality assurance possible – the opposite of a demonstration prototype running on sample data.
Healthcare NLP reduces processing costs by over 80% compared to cloud-based large language model APIs through fixed-cost local deployment [Kocaman et al., 2025b]. For organizations processing millions of documents, per-request API pricing is economically unsustainable. Providence’s on-premise infrastructure with predictable costs enables production scale that cloud API dependency cannot support.
MiBA processing 1.4 million physician notes and approximately 1 million PDF reports and scans required deployed infrastructure supporting continuous operation, not research computing resources allocated periodically [MiBA Case Study, 2025]. Organizations must plan for ongoing maintenance including model updates as medical knowledge evolves, terminology mappings as code systems change, performance monitoring detecting drift, and error correction as edge cases emerge.
The seven-layer checklist for clinical AI deployment beyond hallucination prevention
Layer 1: Data engineering and quality infrastructure Ohio State’s portable configuration demonstrates requirements: unified data ingestion, terminology normalization, consistency validation, and longitudinal integrity ensuring complete patient context [Ohio State University Case Study, 2025].
Layer 2: Population-level validation TriNetX’s site-level validation shows testing beyond aggregate metrics: validating performance across demographic groups, institutional documentation patterns, and clinical settings ensuring equitable accuracy [TriNetX Case Study, 2025].
Layer 3: Explainability and human-in-the-loop workflows Care-Connect and Spryfox’s framework demonstrates governance: clinician-developed logic filtering actions for review, maintaining transparency enabling verification, and implementing override mechanisms preserving human authority [Care-Connect & Spryfox Case Study, 2024].
Layer 4: Workflow integration and change management vCare Companion’s 3+ hour per shift reduction comes from workflow-first design: automating existing tasks rather than adding new burden, integrating with familiar tools, and minimizing disruption to established patterns [vCare Companion Case Study, 2025].
Layer 5: Bias auditing and fairness validation Organizations must validate across diverse populations, monitor for systematic performance gaps, and ensure artificial intelligence does not amplify existing healthcare disparities.
Layer 6: Governance, compliance, and audit trails Martlet.ai’s full provenance tracking [Martlet.ai Case Study, 2025] and Dandelion Health’s HIPAA expert determination [Dandelion Health Case Study, 2024] demonstrate requirements: full audit logging, regulatory compliance frameworks, and audit capabilities supporting retrospective review.
Layer 7: Sustainable infrastructure and maintenance Providence’s 2-billion-note de-identification pipeline shows requirements: distributed processing supporting massive throughput, consistent obfuscation and tokenization for longitudinal linkage, predictable costs enabling long-term operation, and systematic maintenance handling model updates and drift detection [Kocaman et al., 2025a].
Organizations can explore production-grade clinical artificial intelligence infrastructure through Healthcare NLP for entity extraction, Medical LLM for clinical reasoning, and Generative AI Lab for human-in-the-loop governance, and de-identification for privacy-preserving processing. The customer implementations including Ohio State, Providence, MiBA, Dandelion Health, and vCare Companion demonstrate architectures addressing deployment challenges beyond hallucination prevention.
FAQs
Why do data quality issues matter more than hallucinations for clinical AI deployment? Providence’s de-identification of 2 billion patient notes [Kocaman et al., 2025a] and Ohio State processing 200+ million notes [Ohio State University Case Study, 2025] demonstrate that real clinical data is fragmented, inconsistent, and heterogeneous – challenges that break artificial intelligence systems regardless of hallucination prevention. Clinical notes use inconsistent terminology across departments, scanned documents require optical character recognition with variable quality, legacy data uses non-standard codes, missing fields contain information documented elsewhere, and temporal ambiguity obscures whether conditions are current versus historical. Even factually accurate large language models operating on incomplete or misleading data produce clinically invalid outputs. MiBA’s 93% entity extraction across 1.4 million oncology notes [MiBA Case Study, 2025] required handling diverse documentation patterns that general-purpose models cannot systematically process. Organizations must invest in data engineering, terminology normalization, and quality validation before artificial intelligence deployment – infrastructure that hallucination prevention alone does not address.
How do organizations validate that clinical AI works across diverse patient populations? TriNetX’s implementation demonstrates validation requirements: fine-tuning models to overcome site-level documentation variation and creating harmonized labels bringing consistency across networks [TriNetX Case Study, 2025]. Organizations must test model performance across: (1) demographic groups including age, race, ethnicity, gender, and socioeconomic status, (2) clinical settings including academic medical centers, community hospitals, rural facilities, and specialty practices, (3) documentation patterns reflecting institutional and individual clinician variation, (4) disease severity and complexity ranges, and (5) data completeness levels acknowledging that underserved populations often have sparser records. Aggregate accuracy metrics mask systematic failures for specific subpopulations. Organizations should measure performance stratified by patient characteristics, compare results across sites, monitor fordisparities indicating bias, and implement remediation when subgroup performance gaps emerge. The systematic assessment showing Healthcare NLP missing 0.9% of entities versus GPT-4o missing 14.6% [Kocaman et al., 2025b] matters more when those missed entities cluster in documentation associated with vulnerable populations.
What makes clinical AI explainable enough for production deployment? Guidelines Central’s platform demonstrates explainability requirements: answering questions about clinical guidelines with deep links to supporting sections, explaining recommendations with reasoning chains, and providing evidence sources clinicians can verify [Guidelines Central Case Study, 2025]. Care-Connect and Spryfox show validation patterns: using clinician-developed logic to filter entities for review, maintainingtransparency enabling verification, and implementing override mechanisms [Care-Connect & Spryfox Case Study, 2024]. Explainable systems must provide: (1) reasoning chains showing how conclusions were reached from input data, (2) confidence scores indicating uncertainty levels, (3) source attribution linking outputs to specific clinical documentation, (4) feature importance identifying which information most influenced decisions, and (5) natural language explanations clinicians can understand without technical expertise. Black-box outputs – even when correct – undermine trust and prevent clinical adoption. The CLEVER study showing physicians preferring specialized models 45-92% more often [Kocaman et al., 2025c] reflects trust built through consistent, explainable performance rather than occasional impressive but opaque outputs.
Why do technically excellent AI tools fail due to workflow integration issues? vCare Companion reducing administrative burden by 3+ hours per shift demonstrates successful integration: automating existing documentation workflows without requiring new tasks, integrating with electronic medical records clinicians already use, and operating through ambient listening requiring no interface changes [vCare Companion Case Study, 2025]. Failed integrations create additional burden: generating outputs in formats requiring manual transfer to electronic health records, producing alerts requiring clinician triage without actionable recommendations, requiring new software interfaces disrupting established patterns, or duplicating documentation already captured elsewhere. Intermountain Health’s 70% time reduction through automated summarization succeeded because it accelerated existing document review workflows rather than adding new processes [Intermountain Health Case Study, 2024]. Organizations should co-design tools with end users, pilot incrementally to identify friction points, measure actual workflow impact rather than technical capability alone, adapt tools to existing practices rather than forcing practice changes, and train users ensuring they understand when and how to use new capabilities. Technical excellence without workflow fit yields unused technology, not improved care.
What infrastructure do organizations need for sustainable clinical AI beyond pilot demonstrations? Providence’s de-identification of 2 billion patient notes demonstrates production requirements: distributed processing across large clusters of commodity hardware, automated data pipelines with consistent obfuscation and tokenization to link records longitudinally, real-time monitoring detecting errors and performance degradation, systematic quality assurance validating outputs (including equity analysis across gender, age, ethnicity, and state), and full audit logging [Kocaman et al., 2025a]. Ohio State’s portable configuration shows deployment flexibility: executing unchanged on both Azure Databricks and on-premise environments with consistent logging [Ohio State University Case Study, 2025]. Organizations need: (1) computational infrastructure supporting required throughput at production scale, (2) data pipelines consolidating electronic health records, laboratory, imaging, and other clinical systems, (3) Healthcare NLP for entity extraction handling clinical data heterogeneity, (4) Generative AI Lab for human-in-the-loop governance and validation, (5) monitoring detecting drift, errors, and performance changes, (6) maintenance processes updating models as medical knowledge evolves, and (7) fixed-cost deployment avoiding per-request cloud API pricing that becomes prohibitively expensive at scale. Healthcare NLP reduces processing costs by over 80% compared to cloud APIs through on-premise deployment [Kocaman et al., 2025b] – economic sustainability enabling long-term operation.
How do organizations ensure clinical AI compliance with healthcare regulations? Dandelion Health’s de-identification requiring HIPAA expert determination demonstrates compliance requirements: breaking down data types by risk level, adapting processes accordingly, determining recall versus precision balance, applying pre-processing enhancements, and maintaining full provenance for regulatory review [Dandelion Health Case Study, 2024]. Martlet.ai shows audit trail requirements: linking every assertion to source documentation, enforcing evidence sufficiency, and producing compliance-ready export [Martlet.ai Case Study, 2025]. Organizations must implement: (1) data governance frameworks defining who accesses what data under what conditions, (2) de-identification meeting Safe Harbor or expert determination standards, (3) full audit trails documenting inputs, processing, outputs, and human reviews, (4) consent management tracking patient authorization for data use, (5) security controls preventing unauthorized access, (6) incident response procedures addressing breaches or errors, and (7) regular compliance audits validating adherence to HIPAA, GDPR, or other applicable regulations. Software as medical device classification may apply when artificial intelligence makes clinical recommendations, triggering FDA oversight requiring premarket validation, safety testing, and post-market surveillance.
What is the realistic timeline and cost for moving clinical AI from pilot to production? Ohio State processing 200+ million notes [Ohio State University Case Study, 2025] and Providence’s de-identification of 2 billion patient notes [Kocaman et al., 2025a]represent multi-year infrastructure development, not rapid pilot-to-production transitions. Realistic timelines include: (1) data infrastructure development consolidating clinical sources (6-12 months), (2) model deployment and institutional validation (3-6 months), (3) workflow integration and change management (6-12 months), (4) governance framework implementation (3-6 months), (5) end-to-end testing across normal operations and failure modes (3-6 months), and (6) iterative refinement based on production experience (ongoing). Total timeline from planning to stable production typically ranges from 18-36 months for full production systems. Costs include Healthcare NLP and Medical LLM licensing, computational infrastructure for processing at scale, data engineering and integration development, governance and compliance framework implementation, staff training and change management, and ongoing maintenance and model updates. However, operational benefits justify investment: Intermountain’s 70% time reduction [Intermountain Health Case Study, 2024] and vCare Companion’s 3+ hour per shift reduction [vCare Companion Case Study, 2025] represent substantial labor reallocation from administrative tasks to patient care. Organizations should plan for multi-year investment with incremental value delivery rather than expecting rapid pilot-to-production transitions.
What solutions does John Snow Labs offer for addressing deployment challenges beyond hallucination prevention? John Snow Labs provides purpose-built infrastructure addressing all seven deployment challenge layers. Healthcare NLP includes pre-trained models achieving 96% F1-score handling real clinical data heterogeneity that breaks general-purpose models [Kocaman et al., 2025b] – addressing data quality and context challenges demonstrated by Providence, Ohio State, and MiBA implementations [Kocaman et al., 2025a]; [Ohio State University Case Study, 2025]; [MiBA Case Study, 2025]. Medical LLM provides specialized clinical reasoning with explainability capabilities that Guidelines Central case-study demonstrates [Guidelines Central Case Study, 2025]. Generative AI Lab enables human-in-the-loop governance and validation workflows that Care-Connect and Spryfox require for regulatory-grade accuracy [Care-Connect & Spryfox Case Study, 2024]. De-identification provides HIPAA-compliant processing that Dandelion Health uses for expert determination [Dandelion Health Case Study, 2024]. Visual NLP processes scanned documents and imaging data addressing multimodal requirements. These integrate with Databricks, AWS, Azure, and on-premise environments supporting deployment flexibility. Fixed-cost local deployment reduces processing costs by over 80% versus cloud APIs [Kocaman et al., 2025b] – economic sustainability Providence demonstrates at the scale of 2 billion de-identifiedpatient notes [Kocaman et al., 2025a]. JSL offers live demonstrations of production-grade clinical AI infrastructure—contact the team to review architecture patterns relevant to your deployment environment.
References
Care-Connect & Spryfox Case Study. (2024). How Care-Connect and Spryfox use NLP in Making Patient-level Decisions. NLP Summit / John Snow Labs. https://www.nlpsummit.org/how-care-connect-and-spryfox-use-nlp-in-making-patient-level-decisions/
Dandelion Health Case Study. (2024). Deidentifying Free-Text Patient Notes: No Need for Tradeoffs. NLP Summit / John Snow Labs. https://www.nlpsummit.org/deidentifying-free-text-patient-notes-no-need-for-tradeoffs/
Guidelines Central Case Study. (2025). Matching Patients and Answers to the Largest Clinical Guidelines Library in the World. NLP Summit / John Snow Labs. https://www.nlpsummit.org/matching-patients-and-answers-to-the-largest-clinical-guidelines-library-in-the-world/
Intermountain Health Case Study. (2024). Empowering Healthcare through NLP: Harnessing Clinical Document Insights at Intermountain Health. NLP Summit / John Snow Labs. https://www.nlpsummit.org/empowering-healthcare-through-nlp-harnessing-clinical-document-insights-at-intermountain-health/
Kocaman, V., Mico, L., Kaya, M. A., Taiyab, N., Talby, D., Surh, T., Guo, Y., Tomer, V., & Kramer, R. (2025a). Automated De-Identification, Consistent Obfuscation, and Regulatory Grade Validation of 2 Billion Patient Notes. Research Square (preprint, in collaboration with Providence Health). https://www.researchsquare.com/article/rs-6867162/v1
Kocaman, V., Santas, M., Gul, Y., Butgul, M., & Talby, D. (2025b). Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text De-Identification? Text2Story Workshop, ECIR 2025. https://arxiv.org/abs/2503.20794
Kocaman, V., Kaya, M., Ferrer, A., & Talby, D. (2025c). CLEVER: Clinical Large Language Model Evaluation by Expert Review. JMIR AI. https://ai.jmir.org/2025/1/e72153
Martlet.ai Case Study. (2025). Preparing for RADV Audits with AI: Automating HCC Evidence Packets for Medicare Advantage. Applied AI Summit / John Snow Labs. https://appliedaisummit.org/preparing-for-radv-audits-with-ai-automating-hcc-evidence-packets-for-medicare-advantage/
