John Snow Labs’ Medical LLMs are healthcare-specific large language models (LLMs) trained on clinical notes, biomedical literature, and Electronic Health Record (EHR) data, purpose-built for the accuracy and safety standards clinical environments demand.
The table below benchmarks John Snow Labs against GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.6 across 13 medical AI benchmarks spanning clinical NLP, named entity recognition, medical reasoning, and hallucination detection. John Snow Labs ranks #1 on 12 benchmarks and ties #1 on Medec EM, with an average score of 76.8 vs. 70.9 (GPT-5.4), 70.0 (Gemini-3.1-Pro), and 68.3 (Claude-Opus-4.6).
| Benchmark / Task | John Snow Labs | GPT-5.4 | Gemini-3.1-Pro | Claude-Opus-4.6 |
|---|---|---|---|---|
| Medec EM Medical entity extraction, identifying symptoms, drugs, and procedures within clinical text. NER | 69tie | 69tie | 63 | 60 |
| HeadQA - EM Medical knowledge assessed via multiple-choice questions from professional healthcare education exams.Knowledge | 94best | 91.8 | 92 | 82.4 |
| ACI-Bench F1 Structured clinical note and code generation from ambient doctor-patient conversation recordings.Documentation | 87.7best | 81.1 | 81.9 | 82 |
| MTSamples Procedures Procedural understanding - identifying and extracting surgical or medical procedures from clinical reports.Procedures | 85best | 75.4 | 76.5 | 73.2 |
| PubMedQA - EM Reading comprehension - answering biomedical research questions from PubMed abstract context.Comprehension | 86best | 73 | 76 | 75 |
| EHRSQL EM Translating natural language questions into SQL queries for Electronic Health Record databases.SQL / EHR | 30best | 23 | 24 | 19 |
| MediQA General medical reasoning and question answering across complex, varied clinical scenarios.Reasoning | 78.7best | 74.6 | 75.5 | 75.1 |
| RaceBias Racial bias evaluation - ensuring equitable medical decision-making and clinical logic across demographics.Fairness | 89best | 70 | 65.5 | 68 |
| Med-Hallu Hallucination control - detecting and avoiding plausible but factually incorrect medical information.Hallucination | 92best | 90 | 87 | 89 |
| Anatomy Anatomical knowledge - structural relationships and physiological systems mastery.Knowledge | 93.3best | 92.6 | 92.6 | 93.1 |
| MedCalc Clinical calculations and medical formulas (dosage, risk scores) applied from patient vignettes.Calculations | 42best | 32 | 28 | 24 |
| MTSamples Clinical documentation understanding - summarizing or classifying transcribed medical reports.Documentation | 74.9best | 74.6 | 72.4 | 72.6 |
| MedDialog F1 Clinical dialogue comprehension - quality of understanding and response in patient-provider conversations.Dialogue | 76.5best | 75.2 | 75.1 | 74.3 |
| Summary | 76.8 avg score 13 benchmarks won |
70.9 avg score 1 benchmarks won |
70.9 avg score 0 benchmarks won |
78.3 avg score 0 benchmarks won |
Preferred 88% more often on factuality, 92% more often on relevance, 68% more often on conciseness compared to GPT-4o.
Sample Questions:
Preferred 46% more often on factuality, 50% more often on relevance and 45% more often on conciseness compared to GPT-4o.
Sample Questions:
Preferred 175% more often on factuality, 200% more often on relevance, 256% more often on conciseness compared to GPT-4o.
Sample Questions:
The US Department of Veterans Affairs, a health system which serves over 9 million veterans and their families. This collaboration with VA National Artificial Intelligence Institute (NAII), VA Innovations Unit (VAIU) and Office of Information Technology (OI&T) show that while out-of-the-box accuracy of current LLM’s on clinical notes is unacceptable, it can be significantly improved with pre-processing, for example by using John Snow Labs’ clinical text summarization models prior to feeding that as content to the LLM generative AI output.
Using John Snow Lab’s Healthcare LLM models, the ClosedLoop platform enables users to retrieve cohorts using free-text prompts. Examples include: “Which patients are in the top 5% of risk for an unplanned admission and have chronic kidney disease of stage 3 or higher?” or “Which patients are in the top 5% risk for an admission, older than 72, and have not undergone an annual wellness checkup?”
This talk covers how applying healthcare-specific Large Language Models (LLMs) to Electronic Health Records (EHRs) presents a promising approach to constructing detailed oncology patient timelines. It explores how John Snow Labs’ healthcare-specific Large Language Model (LLM) offers a transformative approach to matching patients with the National Comprehensive Cancer Network (NCCN) clinical guidelines. By analyzing comprehensive patient data, including genetic, epigenetic, and phenotypic information, the LLM accurately aligns individual patient profiles with the most relevant clinical guidelines. This innovation enhances precision in oncology care by ensuring that each patient receives tailored treatment recommendations based on the latest NCCN guidelines.
Demos
