The table below benchmarks John Snow Labs against GPT-5.5, Gemini-3.5-Flesh, and Claude-Opus-4.8 across 15 medical AI benchmarks spanning clinical reasoning, answering patient and clinician questions, clinical NLP, and hallucination detection.
John Snow Labs ranks #1 on all 15 benchmarks.
| Benchmark / Task | John Snow Labs | OpenAI GPT-5.5 | Claude-Opus-4.8 | Gemini-3.5-Flesh |
|---|---|---|---|---|
| Medec EM Medical entity extraction, identifying symptoms, drugs, and procedures within clinical text. NER | 85.0best | 68.0 | 67 | 70.0 |
| MTSamples Procedures Procedural understanding - identifying and extracting surgical or medical procedures from clinical reports.Procedures | 73.8best | 71.6 | 71.5 | 72.0 |
| MedCalc Clinical calculations and medical formulas (dosage, risk scores) applied from patient vignettes.Calculations | 48.0best | 44.0 | 34.0 | 42.0 |
| ACI-Bench F1 Structured clinical note and code generation from ambient doctor-patient conversation recordings.Documentation | 85.2best | 83.4 | 83.9 | 81.6 | MedicationQA Provide accurate, accessible, and informative medication-related responses to open-ended consumer health questions.Documentation | 80.9best | 70.2 | 70.6 | 71.4 |
| MedDialog F1 Clinical dialogue comprehension - quality of understanding and response in patient-provider conversations.Dialogue | 76.3best | 75.9 | 76.1 | 76.2 |
| PubMedQA - EM Reading comprehension - answering biomedical research questions from PubMed abstract context.Comprehension | 82best | 76.0 | 76.0 | 75.0 |
| HeadQA - EM Medical knowledge assessed via multiple-choice questions from professional healthcare education exams.Knowledge | 93.9best | 90.1 | 89.8 | 91.2 |
| MedBullets Apply clinical knowledge: Answer questions similar to those found on Step 2 and Step 3 US medical board exams.Knowledge | 90.3best | 89.0 | 79.0 | 80.0 |
| EHRSQL EM Translating natural language questions into SQL queries for Electronic Health Record databases.SQL / EHR | 34.0best | 29.0 | 29.0 | 14.0 |
| MediQA General medical reasoning and question answering across complex, varied clinical scenarios.Reasoning | 78.1best | 76.1 | 76.9 | 76.5 |
| RaceBias Racial bias evaluation - ensuring equitable medical decision-making and clinical logic across demographics.Fairness | 96.0best | 91.0 | 89.0 | 90.0 |
| Med-Hallu Hallucination control - detecting and avoiding plausible but factually incorrect medical information.Hallucination | 96.0best | 92.0 | 92.0 | 90.0 |
| MMLU Clinical Knowledge Evaluates factual and practical understanding of real-world medicine.Knowledge | 97.5best | 96.5 | 97.0 | 95.5 |
| MedQA Evaluate professional clinical reasoning skills using USMLE-style multiple-choice questions.Knowledge | 96.2best | 95.0 | 93.5 | 95.0 |
| Summary | 76.8 avg score 13 benchmarks won |
70.9 avg score 1 benchmarks won |
70.9 avg score 0 benchmarks won |
78.3 avg score 0 benchmarks won |
Preferred 88% more often on factuality, 92% more often on relevance, 68% more often on conciseness compared to GPT-4o.
Sample Questions:
Preferred 46% more often on factuality, 50% more often on relevance and 45% more often on conciseness compared to GPT-4o.
Sample Questions:
Preferred 175% more often on factuality, 200% more often on relevance, 256% more often on conciseness compared to GPT-4o.
Sample Questions:
The US Department of Veterans Affairs, a health system which serves over 9 million veterans and their families. This collaboration with VA National Artificial Intelligence Institute (NAII), VA Innovations Unit (VAIU) and Office of Information Technology (OI&T) show that while out-of-the-box accuracy of current LLM’s on clinical notes is unacceptable, it can be significantly improved with pre-processing, for example by using John Snow Labs’ clinical text summarization models prior to feeding that as content to the LLM generative AI output.
Using John Snow Lab’s Healthcare LLM models, the ClosedLoop platform enables users to retrieve cohorts using free-text prompts. Examples include: “Which patients are in the top 5% of risk for an unplanned admission and have chronic kidney disease of stage 3 or higher?” or “Which patients are in the top 5% risk for an admission, older than 72, and have not undergone an annual wellness checkup?”
This talk covers how applying healthcare-specific Large Language Models (LLMs) to Electronic Health Records (EHRs) presents a promising approach to constructing detailed oncology patient timelines. It explores how John Snow Labs’ healthcare-specific Large Language Model (LLM) offers a transformative approach to matching patients with the National Comprehensive Cancer Network (NCCN) clinical guidelines. By analyzing comprehensive patient data, including genetic, epigenetic, and phenotypic information, the LLM accurately aligns individual patient profiles with the most relevant clinical guidelines. This innovation enhances precision in oncology care by ensuring that each patient receives tailored treatment recommendations based on the latest NCCN guidelines.
Demos
