was successfully added to your cart.

BusinessAnnouncementNatural Language Processing

Hindi WordEmbeddings, Bengali Named Entity Recognition, 30+ new models, analyze Crypto news with NLU 1.1.2

By February 18, 2021February 19th, 2021No Comments
We are very happy to announce NLU 1.1.2 has been released with the integration of 30+ models and pipelines Bengali Named Entity Recognition, Hindi Word Embeddings,

and state-of-the-art transformer based OntoNotes models and pipelines from the incredible Spark NLP 2.7.3 Release in addition to a few bugfixes.
In addition to that, there is a new NLU Webinar video showcasing in detail
how to use NLU to analyze a crypto news dataset to extract keywords unsupervised and predict sentimental/emotional distributions of the dataset and much more!

 

Python’s NLU library: 1,000+ models, 200+ Languages, State of the Art Accuracy, 1 Line of code – NLU NYC/DC NLP Meetup Webinar

Using just 1 line of Python code by leveraging the NLU library, which is powered by the award-winning Spark NLP.

This webinar covers, using live coding in real-time,
how to deliver summarization, translation, unsupervised keyword extraction, emotion analysis,
question answering, spell checking, named entity recognition, document classification, and other common NLP tasks. This is all done with a single line of code, that works directly on Python strings or pandas data frames.
Since NLU is based on Spark NLP, no code changes are required to scale processing to a multi-core or cluster environment – integrating natively with Ray, Dask, or Spark data frames.

The recent releases for Spark NLP and NLU include pre-trained models for over 200 languages and language detection for 375 languages.
This includes 20 languages families; non-Latin alphabets; languages that do not use spaces for word segmentation like
Chinese, Japanese, and Korean; and languages written from right to left like Arabic, Farsi, Urdu, and Hebrew.
We’ll also cover some of the algorithms and models that are included. The code notebooks will be freely available online.

NLU 1.1.2 New Models and Pipelines

NLU 1.1.2 New Non-English Models

Language nlu.load() reference Spark NLP Model reference Type
Bengali bn.ner ner_jifs_glove_840B_300d Word Embeddings Model (Alias)
Bengali bn.ner.glove ner_jifs_glove_840B_300d Word Embeddings Model (Alias)
Hindi hi.embed hindi_cc_300d NerDLModel
Bengali bn.lemma lemma Lemmatizer
Japanese ja.lemma lemma Lemmatizer
Bihari bh.lemma lemma Lemma
Amharic am.lemma lemma Lemma

NLU 1.1.2 New English Models and Pipelines

Language nlu.load() reference Spark NLP Model reference Type
English en.ner.onto.bert.small_l2_128 onto_small_bert_L2_128 NerDLModel
English en.ner.onto.bert.small_l4_256 onto_small_bert_L4_256 NerDLModel
English en.ner.onto.bert.small_l4_512 onto_small_bert_L4_512 NerDLModel
English en.ner.onto.bert.small_l8_512 onto_small_bert_L8_512 NerDLModel
English en.ner.onto.bert.cased_base onto_bert_base_cased NerDLModel
English en.ner.onto.bert.cased_large onto_bert_large_cased NerDLModel
English en.ner.onto.electra.uncased_small onto_electra_small_uncased NerDLModel
English en.ner.onto.electra.uncased_base onto_electra_base_uncased NerDLModel
English en.ner.onto.electra.uncased_large onto_electra_large_uncased NerDLModel
English en.ner.onto.bert.tiny onto_recognize_entities_bert_tiny Pipeline
English en.ner.onto.bert.mini onto_recognize_entities_bert_mini Pipeline
English en.ner.onto.bert.small onto_recognize_entities_bert_small Pipeline
English en.ner.onto.bert.medium onto_recognize_entities_bert_medium Pipeline
English en.ner.onto.bert.base onto_recognize_entities_bert_base Pipeline
English en.ner.onto.bert.large onto_recognize_entities_bert_large Pipeline
English en.ner.onto.electra.small onto_recognize_entities_electra_small Pipeline
English en.ner.onto.electra.base onto_recognize_entities_electra_base Pipeline
English en.ner.onto.large onto_recognize_entities_electra_large Pipeline

New Tutorials and Notebooks

 

NLU 1.1.2 Bug Fixes

  • Fixed a bug that caused NER confidences not to be extracted
  • Fixed a bug that caused nlu.load(‘spell’) to crash
  • Fixed a bug that caused Uralic/Estonian/ET language models not to be loaded properly

 

New Easy NLU 1-liners in 1.1.2

Named Entity Recognition for Bengali (GloVe 840B 300d)

#Bengali for :  It began to be widely used in the United States in the early '90s.
nlu.load("bn.ner").predict("৯০ এর দশকের শুরুর দিকে বৃহৎ আকারে মার্কিন যুক্তরাষ্ট্রে এর প্রয়োগের প্রক্রিয়া শুরু হয়'")

output :

entities token Entities_classes ner_confidence
[‘মার্কিন যুক্তরাষ্ট্রে’] ৯০ [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] এর [‘LOC’] 0.9999
[‘মার্কিন যুক্তরাষ্ট্রে’] দশকের [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] শুরুর [‘LOC’] 0.9969
[‘মার্কিন যুক্তরাষ্ট্রে’] দিকে [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] বৃহৎ [‘LOC’] 0.9994
[‘মার্কিন যুক্তরাষ্ট্রে’] আকারে [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] মার্কিন [‘LOC’] 0.9602
[‘মার্কিন যুক্তরাষ্ট্রে’] যুক্তরাষ্ট্রে [‘LOC’] 0.4134
[‘মার্কিন যুক্তরাষ্ট্রে’] এর [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] প্রয়োগের [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] প্রক্রিয়া [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] শুরু [‘LOC’] 0.9999
[‘মার্কিন যুক্তরাষ্ট্রে’] হয় [‘LOC’] 1
[‘মার্কিন যুক্তরাষ্ট্রে’] [‘LOC’] 1

Bengali Lemmatizer

#Bengali for :  One morning in the marble-decorated building of Vaidyanatha, an obese monk was engaged in the enchantment of Duis and the milk service of one and a half Vaidyanatha. Give me two to eat
nlu.load("bn.lemma").predict("একদিন প্রাতে বৈদ্যনাথের মার্বলমণ্ডিত দালানে একটি স্থূলোদর সন্ন্যাসী দুইসের মোহনভোগ এবং দেড়সের দুগ্ধ সেবায় নিযুক্ত আছে বৈদ্যনাথ গায়ে একখানি চাদর দিয়া জোড়করে একান্ত বিনীতভাবে ভূতলে বসিয়া ভক্তিভরে পবিত্র ভোজনব্যাপার নিরীক্ষণ করিতেছিলেন এমন সময় কোনোমতে দ্বারীদের দৃষ্টি এড়াইয়া জীর্ণদেহ বালক সহিত একটি অতি শীর্ণকায়া রমণী গৃহে প্রবেশ করিয়া ক্ষীণস্বরে কহিল বাবু দুটি খেতে দাও")

output :

lemma document
[‘একদিন’, ‘প্রাতঃ’, ‘বৈদ্যনাথ’, ‘মার্বলমণ্ডিত’, ‘দালান’, ‘এক’, ‘স্থূলউদর’, ‘সন্ন্যাসী’, ‘দুইসের’, ‘মোহনভোগ’, ‘এবং’, ‘দেড়সের’, ‘দুগ্ধ’, ‘সেবা’, ‘নিযুক্ত’, ‘আছে’, ‘বৈদ্যনাথ’, ‘গা’, ‘একখান’, ‘চাদর’, ‘দেওয়া’, ‘জোড়কর’, ‘একান্ত’, ‘বিনীতভাব’, ‘ভূতল’, ‘বসা’, ‘ভক্তিভরা’, ‘পবিত্র’, ‘ভোজনব্যাপার’, ‘নিরীক্ষণ’, ‘করা’, ‘এমন’, ‘সময়’, ‘কোনোমত’, ‘দ্বারী’, ‘দৃষ্টি’, ‘এড়ানো’, ‘জীর্ণদেহ’, ‘বালক’, ‘সহিত’, ‘এক’, ‘অতি’, ‘শীর্ণকায়া’, ‘রমণী’, ‘গৃহ’, ‘প্রবেশ’, ‘বিশ্বাস’, ‘ক্ষীণস্বর’, ‘কহা’, ‘বাবু’, ‘দুই’, ‘খাওয়া’, ‘দাওয়া’] একদিন প্রাতে বৈদ্যনাথের মার্বলমণ্ডিত দালানে একটি স্থূলোদর সন্ন্যাসী দুইসের মোহনভোগ এবং দেড়সের দুগ্ধ সেবায় নিযুক্ত আছে বৈদ্যনাথ গায়ে একখানি চাদর দিয়া জোড়করে একান্ত বিনীতভাবে ভূতলে বসিয়া ভক্তিভরে পবিত্র ভোজনব্যাপার নিরীক্ষণ করিতেছিলেন এমন সময় কোনোমতে দ্বারীদের দৃষ্টি এড়াইয়া জীর্ণদেহ বালক সহিত একটি অতি শীর্ণকায়া রমণী গৃহে প্রবেশ করিয়া ক্ষীণস্বরে কহিল বাবু দুটি খেতে দাও

Japanese Lemmatizer

#Japanese for :  Some residents were uncomfortable with this, but it seems that no one is now openly protesting or protesting.
nlu.load("ja.lemma").predict("これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。")

output :

lemma document
[‘これ’, ‘にる’, ‘不快’, ‘感’, ‘を’, ‘示す’, ‘住民’, ‘はる’, ‘いる’, ‘まする’, ‘たる’, ‘がる’, ‘,’, ‘現在’, ‘,’, ‘表立つ’, ‘てる’, ‘反対’, ‘やる’, ‘抗議’, ‘のる’, ‘声’, ‘を’, ‘挙げる’, ‘てる’, ‘いる’, ‘住民’, ‘はる’, ‘いる’, ‘なぐ’, ‘よう’, ‘です’, ‘。’] これに不快感を示す住民はいましたが,現在,表立って反対や抗議の声を挙げている住民はいないようです。

Aharic Lemmatizer

#Aharic for :  Bookmark the permalink.
nlu.load("am.lemma").predict("መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።")

output :

lemma document
[‘‘, ‘መጽሐፍ’, ‘ኡ’, ‘ን’, ‘‘, ‘አስያዝ’, ‘ኧ’, ‘ኣት’, ‘።’] መጽሐፉን መጽሐፍ ኡ ን አስያዛት አስያዝ ኧ ኣት ።

Bhojpuri Lemmatizer

#Bhojpuri for : In this event, participation of World Bhojpuri Conference, Purvanchal Ekta Manch, Veer Kunwar Singh Foundation, Purvanchal Bhojpuri Mahasabha, and Herf - Media.
nlu.load("bh.lemma").predict("एह आयोजन में विश्व भोजपुरी सम्मेलन , पूर्वांचल एकता मंच , वीर कुँवर सिंह फाउन्डेशन , पूर्वांचल भोजपुरी महासभा , अउर हर्फ - मीडिया के सहभागिता बा ।")

output :

lemma document
[‘एह’, ‘आयोजन’, ‘में’, ‘विश्व’, ‘भोजपुरी’, ‘सम्मेलन’, ‘COMMA’, ‘पूर्वांचल’, ‘एकता’, ‘मंच’, ‘COMMA’, ‘वीर’, ‘कुँवर’, ‘सिंह’, ‘फाउन्डेशन’, ‘COMMA’, ‘पूर्वांचल’, ‘भोजपुरी’, ‘महासभा’, ‘COMMA’, ‘अउर’, ‘हर्फ’, ‘-‘, ‘मीडिया’, ‘को’, ‘सहभागिता’, ‘बा’, ‘।’] एह आयोजन में विश्व भोजपुरी सम्मेलन , पूर्वांचल एकता मंच , वीर कुँवर सिंह फाउन्डेशन , पूर्वांचल भोजपुरी महासभा , अउर हर्फ – मीडिया के सहभागिता बा ।

Named Entity Recognition – BERT Tiny (OntoNotes)

nlu.load("en.ner.onto.bert.small_l2_128").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.8536999821662903, 0.7195000052452087, 0.746…] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’]

Named Entity Recognition – BERT Mini (OntoNotes)

nlu.load("en.ner.onto.bert.small_l4_256").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.835099995136261, 0.40450000762939453, 0.331…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘ORG’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘GPE’, ‘GPE’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition – BERT Small (OntoNotes)

nlu.load("en.ner.onto.bert.small_l4_512").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.964900016784668, 0.8299000263214111, 0.9607…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘the 1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition – BERT Medium (OntoNotes)

nlu.load("en.ner.onto.bert.small_l8_512").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.916700005531311, 0.5873000025749207, 0.8816…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘the 1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition – BERT Base (OntoNotes)

nlu.load("en.ner.onto.bert.cased_base").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.504800021648407, 0.47290000319480896, 0.462…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘the 1970s and 1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition – BERT Large (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_small").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.7213000059127808, 0.6384000182151794, 0.731…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition – ELECTRA Small (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_small").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadella.""",output_level = "document")

output :

ner_confidence Entities_classes entities
[0.8496000170707703, 0.4465999901294708, 0.568…] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’]

Named Entity Recognition – ELECTRA Base (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_base").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadellabase.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.5134000182151794, 0.9419000148773193, 0.802…] [‘William Henry Gates III’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘the 1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’]

Named Entity Recognition – ELECTRA Large (OntoNotes)

nlu.load("en.ner.onto.electra.uncased_large").predict("""William Henry Gates III (born October 28, 1955) is an American business magnate,
 software developer, investor, and philanthropist. He is best known as the co-founder of Microsoft Corporation. During his career at Microsoft,
  Gates held the positions of chairman, chief executive officer (CEO), president and chief software architect,
   while also being the largest individual shareholder until May 2014.
    He is one of the best-known entrepreneurs and pioneers of the microcomputer revolution of the 1970s and 1980s. Born and raised in Seattle, Washington, Gates co-founded Microsoft with childhood friend Paul Allen in 1975, in Albuquerque, New Mexico;
     it went on to become the world's largest personal computer software company. Gates led the company as chairman and CEO until stepping down as CEO in January 2000, but he remained chairman and became chief software architect.
     During the late 1990s, Gates had been criticized for his business tactics, which have been considered anti-competitive. This opinion has been upheld by numerous court rulings. In June 2006, Gates announced that he would be transitioning to a part-time
      role at Microsoft and full-time work at the Bill & Melinda Gates Foundation, the private charitable foundation that he and his wife, Melinda Gates, established in 2000.
 He gradually transferred his duties to Ray Ozzie and Craig Mundie.
  He stepped down as chairman of Microsoft in February 2014 and assumed a new post as technology adviser to support the newly appointed CEO Satya Nadellabase.""",output_level = "document")

output :

ner_confidence entities Entities_classes
[0.8442000150680542, 0.26840001344680786, 0.57…] [‘William Henry Gates’, ‘October 28, 1955’, ‘American’, ‘Microsoft Corporation’, ‘Microsoft’, ‘Gates’, ‘May 2014’, ‘one’, ‘1970s’, ‘1980s’, ‘Seattle’, ‘Washington’, ‘Gates co-founded’, ‘Microsoft’, ‘Paul Allen’, ‘1975’, ‘Albuquerque’, ‘New Mexico’, ‘largest’] [‘PERSON’, ‘DATE’, ‘NORP’, ‘ORG’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘CARDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘GPE’, ‘PERSON’, ‘ORG’, ‘PERSON’, ‘DATE’, ‘GPE’, ‘GPE’, ‘GPE’]

Recognize Entities OntoNotes – BERT Tiny

nlu.load("en.ner.onto.bert.tiny").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.994700014591217, 0.9412999749183655, 0.9685…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes – BERT Mini

nlu.load("en.ner.onto.bert.mini").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.996399998664856, 0.9733999967575073, 0.8766…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes – BERT Small

nlu.load("en.ner.onto.bert.small").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9987999796867371, 0.9610000252723694, 0.998…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’, ‘Parliament’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’, ‘ORG’]

Recognize Entities OntoNotes – BERT Medium

nlu.load("en.ner.onto.bert.medium").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9969000220298767, 0.8575999736785889, 0.995…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes – BERT Base

nlu.load("en.ner.onto.bert.base").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.996999979019165, 0.933899998664856, 0.99930…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008 to 2016’, ‘Parliament’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’, ‘ORG’]

Recognize Entities OntoNotes – BERT Large

nlu.load("en.ner.onto.bert.large").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9786999821662903, 0.9549000263214111, 0.998…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008 to 2016’, ‘Parliament’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’, ‘ORG’]

Recognize Entities OntoNotes – ELECTRA Small

nlu.load("en.ner.onto.electra.small").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9952999949455261, 0.8589000105857849, 0.996…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

Recognize Entities OntoNotes – ELECTRA Base

nlu.load("en.ner.onto.electra.base").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9987999796867371, 0.9474999904632568, 0.999…] [‘Johnson’, ‘first’, ‘2001’, ‘Parliament’, ‘eight years’, ‘London’, ‘2008’, ‘2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘ORG’, ‘DATE’, ‘GPE’, ‘DATE’, ‘DATE’]

Recognize Entities OntoNotes – ELECTRA Large

nlu.load("en.ner.onto.large").predict("Johnson first entered politics when elected in 2001 as a member of Parliament. He then served eight years as the mayor of London, from 2008 to 2016, before rejoining Parliament.",output_level="document")

output :

ner_confidence entities Entities_classes
[0.9998000264167786, 0.9613999724388123, 0.998…] [‘Johnson’, ‘first’, ‘2001’, ‘eight years’, ‘London’, ‘2008 to 2016’] [‘PERSON’, ‘ORDINAL’, ‘DATE’, ‘DATE’, ‘GPE’, ‘DATE’]

NLU Installation

# PyPi
!pip install nlu pyspark==2.4.7
#Conda
# Install NLU from Anaconda/Conda
conda install -c johnsnowlabs nlu

Additional NLU Resources

preloader