Don't miss the NLP Summit 2022, free and online event in October 4-6. Register for freehere.
was successfully added to your cart.

Setup Spark NLP on Databricks in 2 Minutes and get the taste of scalable NLP

What will we learn in this article?

We will set up our own Databricks cluster with all dependencies required to run Spark NLP in either Python or Java.
This tutorial consists of the following simple steps :

  1. Create a Databricks cluster
  2. Setup Python dependencies for Spark NLP in the Databricks Spark cluster
  3. Setup Java dependencies for Spark NLP in the Databricks Spark cluster
  4. Test out our cluster

The NLP domain of machine learning has been continuously breaking records in many of the NLP related tasks like part-of-speech tagging, question answering, relation extraction, spell checking, and many more!

In particular a new breed of transformer-based models that can learn and leverage context in sentences to solve these problems and have been responsible for these breakthroughs.

All the text data in the world out there is like a giant pool of oil. NLP research provides us with methods for extracting value from these valuable oily pools of text. The code resulting from NLP research is usually not scalable, which leaves us with just a very tiny and inefficient oil pump for our oil…

This is where Spark NLP comes into play. It basically gives you a huge data pump which is highly scalable and efficient!

  • Wouldn’t you like to play around these state of the art (SOTA) models in NLP?
  • Don’t you want to save time setting up tediously your local environment to get your fingers on SOTA algorithms like Bert, Albert, ELMO, XLNet, and more?
  • Should that all scale well with terabytes of data?

Then Spark-NLP is what you need!


What is Spark NLP and who uses it?

Spark NLP is an award-winning library running on top of Apache Spark distribution engine. It provides the first production-grade versions of the latest deep learning NLP research.

“The most widely used NLP library in the enterprise”

backed by O’Reilly’s most recent “AI Adoption in the Enterprise” survey in February

Widely used by fortune 500 companies


Why is Spark NLP so widely used?

It’s easy and fast to achieve results close to the SOTA in many of the NLP tasks using the latest deep learning NLP research methods which are provided by Spark NLP. All of this will be made scalable by leveraging Spark’s built-in distribution engine.


Easy and fast SOTA with Spark NLP


If you are thinking,

  • How to get started with NLP in Python quickly
  • How to set up a Spark-NLP Python Databricks cluster
  • How to configure Databricks for Python and Spark-NLP
  • I wanna be rich like an oil baron

You have come to the right place my NLP hungry friend!

In this little tutorial, you will learn how to set up your Python environment for Spark-NLP on a community Databricks cluster with just a few clicks in a few minutes! Let’s get started!


0. Login to Databricks or get an account

First login to your Databricks accounts or create one real quick. It’s completely free!


1. Create a cluster with the latest Spark version

Select the Clusters tab on the left side and click on create a new cluster. Make sure you select one of the currently supported Databricks runtimes which you can find here In This example we will be using the 6.5 runtime


2. Install Python Dependencies to cluster

Select your newly created cluster in the clusters tab and then select the Libraries tab. In that menu click on Install New and select the PyPI tab.


3. Install Java Dependencies to cluster

In the same window as before, select Maven and enter these coordinates and hit install. These dependencies are required since even if you only want to run Spark NLP in Python it will invoke computations on the JVM which depend on the Spark NLP library.


You can get the latest MVN coordinates here, the <groupId:atifactId:version> format is expected.


That’s it for setting up your cluster! After a few moments, everything should look like this


4. Let’s test out our cluster real quick

Create a new Python Notebook in Databricks and copy-paste this code into your first cell and run it. The following example demonstrates how to get token embeddings for raw input sentences
This is a fully-fledged state of the art pipeline which generates token embeddings with the official ELMO model which can be used for following downstream tasks.

<span id="2018" class="gb kk kl iz oj b dn ok ol l om" data-selectable-paragraph="">from pyspark.sql.types import StringType
#Spark NLP
import sparknlp
from sparknlp.pretrained import PretrainedPipeline
from sparknlp.annotator import *
from sparknlp.base import *</span><span id="4748" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">#If you need to set any Spark config
spark.conf.set('spark.serializer', 'org.apache.spark.serializer.KryoSerializer')</span><span id="8def" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">#Create Dataframe with Sample data
dfTest = spark.createDataFrame([
    "Spark-NLP would you be so nice and cook up some state of the art embeddings for me?",
    "Tensorflow is cool but can be tricky to get Running. With Spark-NLP, you save yourself a lot of trouble.",
    "I save so much time using Spark-NLP and it is so easy!"
], StringType()).toDF("text")</span><span id="76d1" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">#Basic Spark NLP Pipeline</span><span id="ca78" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">document_assembler = DocumentAssembler() \
    .setInputCol("text") \
      .setOutputCol("document") \</span><span id="aaa6" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">sentence_detector = SentenceDetector() \
    .setInputCols(["document"]) \
    .setOutputCol("sentence")</span><span id="8eae" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">tokenizer = Tokenizer() \
    .setInputCols(["document"]) \
    .setOutputCol("token")</span><span id="8620" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">elmo = ElmoEmbeddings.pretrained() \
      .setInputCols(["token", "document"]) \
      .setOutputCol("elmo") \
      .setPoolingLayer("elmo")</span><span id="540c" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">nlpPipeline = Pipeline(stages=[
])</span><span id="070f" class="gb kk kl iz oj b dn op oq or os ot ol l om" data-selectable-paragraph="">nlp_model =
processed = nlp_model.transform(dfTest)</span>

After running the pipeline your result should look like this



You are now ready for Spark-NLP in your Databricks cluster and enjoy the latest and most scalable NLP models out there!


Spark NLP is easy to set up, reliable and fast! It always provides the latest SOTA models in NLP shortly after they are released.


What we have learned

  • How to set up a Spark-NLP Python Databricks cluster
  • How to set up a Spark-NLP Scala Databricks cluster
  • How to set up a Spark-NLP working environment
  • How to get started with Spark-NLP in Python
  • How to configure Databricks for Python and Spark-NLP
  • How to get started with Spark-NLP easy and quickly


Next Steps:

If this got your appetite started for more NLP and Spark in Python, Scala, Java or R you should next head to these Spark-NLP tutorials: