Watch Healthcare NLP Summit 2024. Watch now.
was successfully added to your cart.

Fine-Tuning and Automated Parameter Optimization for Medical Named Entity Recognition

We are glad to announce that Spark NLP for Healthcare 3.1.2 has been released! This release comes with new features, new models, bug fixes, and examples.


  • Support for Fine-tuning of Ner models.
  • More built-in(pre-defined) graphs for MedicalNerApproach.
  • Date Normalizer.
  • New Relation Extraction Models for ADE.
  • Bug Fixes.
  • Support for user-defined Custom Transformer.
  • Java Workshop Examples.
  • Deprecated Compatibility class in Python.

Support for Fine Tuning of Ner models

Users can now resume training/fine-tune existing(already trained) Spark NLP MedicalNer models on new data. Users can simply provide the path to any existing MedicalNer model and train it further on the new dataset:

ner_tagger = MedicalNerApproach().setPretrainedModelPath("/path/to/trained/medicalnermodel")

If the new dataset contains new tags/labels/entities, users can choose to override existing tags with the new ones. The default behavior is to reset the list of existing tags and generate a new list from the new dataset. It is also possible to preserve the existing tags by setting the ‘overrideExistingTags’ parameter:

ner_tagger = MedicalNerApproach()\

Setting overrideExistingTags to false is intended to be used when resuming trainig on the same, or very similar dataset (i.e. with the same tags or with just a few different ones).

If tags overriding is disabled, and new tags are found in the training set, then the approach will try to allocate them to unused output nodes, if any. It is also possible to override specific tags of the old model by mapping them to new tags:

ner_tagger = MedicalNerApproach()\
.setTagsMapping("B-PER,B-VIP", "I-PER,I-VIP")

In this case, the new tags ‘B-VIP’ and ‘I-VIP’ will replace the already trained tags ‘B-PER’ and ‘I-PER’. Unmapped old tags will remain in use and unmapped new tags will be allocated to new output nodes if any.

Jupyter Notebook:

More built-in graphs for MedicalNerApproach

Seventy new TensorFlow graphs have been added to the library of available graphs which are used to train MedicalNer models. The graph with the optimal set of parameters is automatically chosen by MedicalNerApproach during training.


New annotator that normalize dates to the format YYYY/MM/DD.

This annotator identifies dates in chunk annotations, and transform these dates to the format YYYY/MM/DD.

Both the input and output formats for the annotator are chunk.



sentences = [
    ['Jan 30, 2018'],
    ['3April 2020'],
    ['next monday'],
    ['next week'],
df = spark.createDataFrame(sentences).toDF("text")
document_assembler = DocumentAssembler().setInputCol('text').setOutputCol('document')
chunksDF = document_assembler.transform(df)
aa = map_annotations_col("document"),
lambda x: [Annotation('chunk', a.begin, a.end, a.result, a.metadata, a.embeddings) for a in x], "document",
"chunk_date", "chunk")
dateNormalizer = DateNormalizer().setInputCols('chunk_date').setOutputCol('date').setAnchorDateYear(2021).setAnchorDateMonth(2).setAnchorDateDay(27)
dateDf = dateNormalizer.transform(aa)"date.result","text").show()
    |text    | date  |
    |08/02/2018 |2018/08/02|
    |11/2018   |2018/11/DD|
    |11/01/2018 |2018/11/01|
    |12Mar2021  |2021/03/12|
    |Jan 30, 2018|2018/01/30|
    |13.04.1999 |1999/04/13|
    |3April 2020 |2020/04/03|
    |next Monday |2021/06/19|
    |today    |2021/06/12|
    |next week  |2021/06/19|

New Relation Extraction Models for ADE

We are releasing new Relation Extraction models for ADE (Adverse Drug Event). The new models are available as instances of RelationExtraction and Bert based RelationExtractionDL annotators, and are capable of linking drugs with ADE mentions.



ade_re_model = new RelationExtractionModel().pretrained('ner_ade_clinical', 'en', 'clinical/models')\
.setInputCols(["embeddings", "pos_tags", "ner_chunk", "dependencies"])\
.setRelationPairs(['ade-drug', 'drug-ade'])
pipeline = Pipeline(stages=[documenter, sentencer, tokenizer, pos_tagger, words_embedder, ner_tagger, ner_converter, 
dependency_parser, re_ner_chunk_filter, re_model])
text ="""A 30 year old female presented with tense bullae due to excessive use of naproxin, and leg cramps relating to oxaprozin."""

p_model =[[text]]).toDF("text"))

result = p_model.transform(data)


|  |chunk1    |entity1  | chunk2    |entity2  |  result |
    | 0 |tense bullae |ADE    |naproxin   |DRUG   |     1 |
    | 1 |tense bullae |ADE    |oxaprozin   |DRUG   |     0 |
    | 2 |naproxin   |DRUG    |leg cramps  |ADE    |     0 |
    | 3 |leg cramps  |ADE    |oxaprozin   |DRUG   |     1 |


Model: re_ade_clinical

       precision  recall f1-score  support
         0    0.85   0.89   0.87   1670
         1    0.88   0.84   0.86   1673
     micro avg    0.87   0.87   0.87   3343
     macro avg    0.87   0.87   0.87   3343
    weighted avg    0.87   0.87   0.87   3343

Model: redl_ade_biobert

Relation      Recall Precision    F1  Support
    0          0.894   0.946   0.919   1011
    1          0.963   0.926   0.944   1389
    Avg.        0.928   0.936   0.932
    Weighted Avg.    0.934   0.934   0.933

Support for user-defined Custom Transformer

Utility classes to define custom transformers in python are included in this release. This allows users to define functions in Python to manipulate Spark-NLP annotations. These new Transformers can be added to pipelines like any of the other models you’re already familiar with.

Example of how to use the custom transformer.

          # lower case the content of the annotations
          return[a.copy(a.result.lower()) forainannotations]

Java Workshop Examples

New Java examples were added to the workshop repository.

Deprecated Compatibility class in Python

Due to active release cycle, we are adding & training new pretrained models at each release and it might be tricky to maintain the backward compatibility or keep up with the latest models and versions, especially for the users using our models locally in air-gapped networks.

We are releasing a new utility class to help you check your local & existing models with the latest version of everything we have up to date. You will not need to specify your AWS credentials from now on. This new class is now replacing the previous Compatibility class written in Python and CompatibilityBeta class written in Scala.

from sparknlp_jsl.compatibility import Compatibility
compatibility = Compatibility(spark)


[{'name': 'sentence_detector_dl_healthcare', 'sparkVersion': '2.4', 'version': '2.6.0', 'language': 'en', 'date': '2020-09-13T14:44:42.565', 'readyToUse': 'true'}, {'name': 'sentence_detector_dl_healthcare', 'sparkVersion': '2.4', 'version': '2.7.0', 'language': 'en', 'date': '2021-03-16T08:42:34.391', 'readyToUse': 'true'}]

Installing and Quickstart Instructions:

python3 -m pip install --upgrade spark-nlp-jsl==3.1.2 --user --extra-index-url

Online Quickstart (requires Internet):

Either use our quick SparkSession starter:

from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
from sparknlp.common import *
import sparknlp_jsl
spark = sparknlp_jsl.start("3.1.2-8a8d35167f9758a982fcb9d0a831267887132710")

Or create SparkSession manually:

from sparknlp.annotator import *
from sparknlp_jsl.annotator import *
from sparknlp.base import *
from sparknlp.common import *
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .master('local[*]') \
    .appName('Spark NLP') \
    .config("spark.driver.memory", "32G") \
    .config("spark.driver.maxResultSize", "2G") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .config("spark.kryoserializer.buffer.max", "2000M") \
    .config("spark.jars.packages", "com.johnsnowlabs.nlp:spark-nlp_2.12:3.1.2") \
    .config("spark.jars", "")\

Try Medical LLMs

See in action

2600+ New Models for 200+ Languages and 10+ Dimension Reduction Algorithms for Streamlit Word-Embedding visualizations in 3-D with NLU

We are extremely excited to announce the release of NLU 3.1! This is our biggest release so far and it comes with...