John Snow Labs Medical Language Models vs. AWS Medical Comprehend part 1/3

09.01.2025

Dean Coutinho

Account Executive at John Snow Labs

Many data scientists have asked what and when to use AWS Medical Comprehend vs. John Snow Labs Healthcare NL. In this blog, we explore the aspects and differentiators of John Snow Labs’ Medical Language Models and AWS Medical Comprehend, across accuracy, richness of the libraries and capabilities, and their medical large language models (LLMs). We further highlight their respective strengths and capabilities, discuss the importance of privacy, customization, and scaling capabilities, which are critical factors for organizations in the healthcare and life science industries.

John Snow Labs Vs AWS Medical Comprehend

Accuracy:

Clinical NER

We managed to find 6 common entity types returned by AWS and mapped with the entities in Healthcare NLP using ner_jsl and ner_clinical_large models: Test, Treatment, Medication, Anatomy, Condition, Procedure.
Healthcare NLP does 13% better when it comes to Test entities, and 19% better in Treatment entities. The largest difference is observed in Anatomy entities by 24%. In all the entities compared, Healthcare NLP performs better in all of them, and exceeds AWS by 18% in average
The numbers in red puntos under each entity on the chart’s x-axis denote the number of tokens for the corresponding entity. That is, out of 10,300 Condition-labeled (clinical disorders, symptoms etc.) tokens, AWS fails to detect 1,300 of them while Healthcare NLP fails only with 300 of them (makes more than 4x less error).

For this study, we had to find an open-source dataset for the reproducibility concerns. At the time of writing this article, mtsamples.com hosts 5,003 Samples in 40 types. For this study, we randomly picked 8,000 clinical notes from various types and have human annotators (physicians having substantial experience in each domain) annotate all for named entity recognition and entity resolution tasks.

First of all, we annotated this test dataset within the annotation guideline that we used our most popular clinical NER model named ner_jsl. Then we applied the following mapping to indicate which entity from the new annotation corresponds to an entity from Healthcare NLP vs other cloud services.

As our NER models are originally trained with a large set of the internally annotated datasets and we used several open source and proprietary datasets as well as academic ones like MIMIC-III, we wanted to make sure that the NER models we’ll be using to do this comparison haven’t seen the 8,000 sentences during training. We couldn’t do the same checks with the APIs provided by other cloud providers as they are all managed services with no information about the way they train their models.

De-Identification

For the next test, we did accuracy comparison for De-identification capabilities in English language offered by John Snow Labs and AWS Medical Comprehend. Both libraries offer de-identification capabilities across multiple entities, however, for this test, the entities of focus were Patient ID, Date, Hospital, Location, Age, Patient, Doctor, and Phone. The dataset used compiled 100 random notes from the public n2c2 Deidentification dataset. The results are shown below and can be reproduced by using the following Colab notebook.

The results show that John Snow Labs’ de-identification models performed better than AWS Medical Comprehend in across all entities, except Patient ID and Phone, where the accuracy was at the same level.

John Snow Labs: Demonstrates superior accuracy in de-identification tasks, outperforming AWS Medical Comprehend by a margin of 18% to 15%.
AWS Medical Comprehend: While capable of de-identification, its accuracy is lower compared to JSL.

The tested libraries are:

Spark NLP for Healthcare (clinical_deidentification pipeline with Glove Embeddings)
AWS Comprehend Medical “detect_phi” API call
Entity Tagging Differences: Libraries use different entity tags and annotation guidelines, with a 60% similarity threshold applied for fuzzy matches.
Customization: All libraries support customization, but the benchmark focused on out-of-the-box capabilities.
Tested Labels: Eight labels were evaluated: ID, DATE, HOSPITAL, LOCATION, AGE, PATIENT, DOCTOR, PHONE.
Colab notebook

Richness of Library and Specialty Domains:

Library Size and Pre-trained Models

John Snow Labs: Offers a library including +2,450 healthcare-specific language models, embeddings and annotators, covering a wide range of healthcare domains such as oncology, radiology, medical devices, vaccines, and medical risk factors. The library supports specialized tasks like entity mapping to their corresponding codes across all taxonomies (IC10, SNOMED, RxNorm, CPT, LOINC, HCC, etc.), relation extraction (including Zero-Shot relation extraction), sentiment analysis, Zero-Shot classification, advanced de-identification, and more.

John Snow Labs Provides over 1,400 specialized NER models and pipelines, supporting diverse healthcare applications across multiple languages. Pipelines include:

Adverse drug events detection
PHI detection
Zero-shot entity recognition
Entity Mapping
Mapping healthcare data to standardized vocabularies

AWS Medical Comprehend: Provides a library with 15+ pre-trained models, primarily focused on general entity extraction, sentiment analysis, and entity linking across 3 main taxonomies. Compared to Healthcare NLP, the library is shown to appear constrained in terms of capabilities and state-of-the-art technology trained for industry-specific tasks, does not cover niche domains such as oncology or radiology, and it does not support customization tasks such as document classification, embeddings, contextual parsers, annotators for structured JSON conversion, and more. Such offers may make AWS Medical Comprehend a tempting option for testing, however, it seriously limits the ability to build and productionize robust and enterprise-grade level pipelines.

Reference: https://docs.aws.amazon.com/comprehend-medical/

Specialty Domains

John Snow Labs:

Specializes in Healthcare and Lifescience industries, making them a leader of industry-specific language capabilities across all medical branches, such as oncology, pharmacovigilance, radiology, pathology, genomics, mental health, social determinants of health, and more. It offers vast amount of ready-to-use models and pipelines to tackle specific use cases, such as cancer diagnosis and treatment planning, clinical trials selection and monitoring, adverse drug events detection, unstructured medical report summarization, and disease progression, and more. These domain-specific models provide great understanding of unstructured medical text and help analyze the information for research and clinical applications.

AWS Medical Comprehend:

AWS Medical Comprehend serves six key specialty domains, focusing on various aspects of healthcare and clinical text processing. It supports clinical documentation, enabling the extraction and analysis of unstructured text from patient records and notes. In medical coding and billing, it maps text to ontologies like ICD-10-CM, RxNorm, and SNOMED CT, streamlining coding workflows. The platform addresses pharmacology by extracting medication-related details, including dosages, frequencies, and routes. It ensures compliance in handling sensitive information by identifying and de-identifying Protected Health Information (PHI). Additionally, AWS Medical Comprehend facilitates general healthcare data analysis by extracting entities and relationships from text and enhances understanding through sentiment and syntax analysis, aiding categorization and contextual insights for healthcare data.

Reference: https://docs.aws.amazon.com/comprehend-medical/

Advanced NLP Tasks

John Snow Labs:

Supports advanced NLP tasks, including relation extraction, clinical note summarization, assertion status detection, and advanced de-identification.
Offers pipelines that directly handle complex medical terminologies, facilitating workflows in clinical and research settings.

AWS Medical Comprehend:

Primarily focuses on basic entity extraction and sentiment analysis. Advanced NLP tasks like relation extraction, summarization, and de-identification are not fully supported or are limited in scope.

Domain-specific Tools

John Snow Labs:

Offers +2450 pretrained and domain-specific models and pipelines, which are specifically designed to address complex challenges and allow for comprehensive analysis in the healthcare and life science sector. The domain-specific capabilities include extraction of medical entities (e.g. diseases, drugs, symptoms), identifying relationships between entities (e.g. drug interactions, symptom-disease associations), performing sentiment analysis (e.g. patient feedback or clinical notes), de-identification of medical notes (e.g. unstructured text, DICOM images, PDFs), building knowledge graphs from EHR medical data, clinical decision support (e.g. risk scoring and risk adjustment), clinical trial analysis (e.g. patient outcomes, patient recruitment, adverse drug events), and many more. Overall, John Snow Labs’ medical capabilities offer deep understanding of the terminology, relations, implications, challenges, and compliance requirements for medical data analysis.

AWS Medical Comprehend:

Amazon Comprehend Medical offers domain-specific tools focused on processing unstructured medical text, primarily in English. It detects medical entities such as conditions, medications, anatomy, tests, treatments, and procedures, and links them to standard ontologies like ICD-10-CM, RxNorm, and SNOMED CT. AWS Medical Comprehend also extracts relevant attributes and traits, such as dosage and frequency for medications. Its batch processing capabilities through integration with Amazon S3 make it scalable for large datasets. However, the platform’s scope is narrower, as it lacks advanced features such as relationship extraction, sentiment analysis, knowledge graph building, or capabilities for clinical trial analysis and decision support. Customization options are limited, and it does not support multi-language processing, reducing its flexibility for global healthcare applications.

Multi-Language Support

John Snow Labs: In addition to the +2450 pretrained medical language in English, their library also supports Multilingual Medical Language models and 23 world languages. For example, users can access de-identification models and named entity recognition in German language, Arabic, Spanish and many more.

AWS Medical Comprehend: Is limited to English-only models, which restricts its applicability for non-English healthcare texts.

Comparison of Terminology Mapping

John Snow Labs provides extensive terminology mapping capabilities, integrating a wide array of medical ontologies and standards. This breadth allows the platform to cater to a diverse range of healthcare use cases, from clinical data processing to advanced research applications.

Scope of Terminologies:

John Snow Labs supports an expansive list of ontologies, addressing clinical care (ICD-10-PCS, CPT), oncology (ICD-O), pharmacology (NDC), research (MeSH), and genetics (HPO).
AWS Medical Comprehend is focused on foundational terminologies: ICD-10-CM, RxNorm, and SNOMED CT.

Depth of Application:

John Snow Labs enables mapping to specialized vocabularies like HPO and ICD-O, essential for advanced fields such as oncology and precision medicine.
AWS Medical Comprehend is well-suited for general-purpose healthcare applications, like basic diagnostics, drug standardization, and interoperability, but lacks depth in specialized or research-oriented ontologies.

Customization and Scalability:

John Snow Labs offers fine-tuning capabilities and supports the creation of custom mappings, enhancing adaptability for domain-specific use cases.
AWS Medical Comprehend operates as a pre-trained, black-box API with limited customization options.

Global Usability:

With support for multiple ontologies and international standards, John Snow Labs caters to a broader audience, including researchers, clinical practitioners, and healthcare organizations worldwide.
AWS Medical Comprehend primarily addresses U.S.-centric terminologies, such as ICD-10-CM and RxNorm, making it less versatile for international healthcare settings.