What is Lemmatization?

Lemmatization is a natural language processing (NLP) technique for identifying any word's "lemma" or base dictionary form. Most languages contain many related words that derive from other existing words to communicate changes in tense, plurality, or grammatical category. For example, "sleeping," "slept," "nap," and "slumber" all relate to the root form "sleep."

Lemmatization groups these words with their roots so machine learning and artificial intelligence models can complete language tasks accurately and quickly. Instead of processing many word variations, AI only needs to analyze the root forms, leading to efficient text analytics.

Lemmatization has several applications in natural language processing tasks.

Natural language understanding and processing

Applications like machine translation and voice recognition require models that can understand nuanced meanings of text. Lemmatization reduces words to their original dictionary form to aid with contextual understanding. For example, consider the two sentences:

He looked at his watch to check the time

She likes to watch movies on weekends.

Lemmatization reduces the watch in the first sentence to the root word "watch" (clock) but reduces the "watch" in the second sentence to the dictionary root "see." This improves accuracy in language processing tasks.

Lemmatization also allows natural language processing systems to focus on keywords relevant to their current task. For example, in a sentiment analysis task, lemmatization enables the model to focus on the underlying emotional tone of the text rather than being distracted by word variations, such as "running" or "ran."

Analytics

Lemmatization helps normalize text data, making it easier to analyze and extract insights from vast amounts of unstructured content. It converts raw text data into organized groups of similar words for applications such as topic modeling and trend analysis. Similarly, lemmatization eliminates redundancy and simplifies processing for text classification and clustering tasks like spam detection and document summarization.

Information retrieval

Search engines use lemmatization to understand different forms of user queries that often refer to the same content. For example, users searching for "annual leave policy," "annual leave policies," and "annual leaves" all get similar results because their queries have the same stem. Lemmatization thus expands the scope of search results retrieved without requiring precise keyword matching.

Chatbot and AI interaction

Chatbots and conversational AI rely heavily on lemmatization to improve their understanding of user inputs. By recognizing the contextual meaning of words, lemmatization helps these systems accurately interpret sentences and provide relevant responses. For instance, whether a user says "is running" or "ran," the chatbot can identify the intent behind the statement, leading to smoother and more human-like interactions.

Lemmatization seeks to analyze words morphologically by studying their structure and formation. It focuses on how words are built from smaller units of meaning. It looks for inflected words formed by adding prefixes and suffixes to the same root. For example, "player," "playing," and "played" are inflected forms of the word "play." It also looks for synonyms and word categories within the context of parts of speech and the broader grammar and vocabulary of the language. The goal is to normalize different inflected forms and words with similar meanings for further text analysis and comparison.

Lemmatization algorithms take words as input and return their root forms as output. The root form has a dictionary meaning and is a valid word in the language being analyzed. The typical steps are as follows.

Tokenization

The first step decomposes the input text into individual words called tokens. This can be done using various methods, such as dividing the text based on spaces.

POS tagging

Parts-of-speech tagging assigns a grammatical category (like noun, adjective, etc.) to every token. This step is essential because, in many languages, base word forms vary depending on their grammatical role in the sentence. For example, "play" has different meanings when used as a noun and a verb. As a noun, it means "theatrical production," but as a verb, it means the action of playing.

Rule application

Once each token is assigned a part-of-speech tag, the lemmatization algorithm uses a lexicon or linguistic rules to determine the word's lemma. The lemma is another, typically shorted dictionary word that accurately represents the actual language word. For example, the lemma of "eating" is "eat," while the lemma of "are" is "be." Linguistic rules and patterns are used to make the correct lemmatization decision for complex situations, such as irregular verbs or words with multiple possible lemmas.

We give some examples of lemmatization algorithms below. You can implement them using Python libraries.

WordNet lemmatization

WordNet lemmatization uses the WordNet lexical database to reduce words to their base forms while considering their part of speech. WordNet organizes words into sets of synonyms (synsets) and includes other relationships such as hypernyms and hyponyms. For example, the word "animal" is the hypernym of the word "cat." You can access WordNet lemmatization functions from the Python NLTK library.

Statistical lemmatization

Statistical lemmatization relies on probabilistic models, such as Hidden Markov Models (HMMs), to determine the lemma based on word context. These models are trained on labeled datasets to learn how words change in different grammatical structures. This method benefits morphologically rich languages where context significantly determines word forms.

While direct implementations of statistical lemmatization are not available, Python tools like spaCy implicitly use statistical models in their lemmatization process:

Finite state transducers

Finite state transducers (FSTs) are state machine models that implement a language's morphological rules. They match words to their lemma by following a predefined set of transitions and rules. FSTs are highly efficient for large-scale lemmatization tasks. Although not natively supported in Python, libraries like Apertium can process text using transducers.

Neural lemmatization

Neural lemmatization uses machine learning to predict lemmas. These models are trained on large annotated training data and use contextual embeddings to achieve high accuracy, especially for ambiguous or irregular words. Flair, a popular NLP library in Python, can be used for neural lemmatization:

Hybrid lemmatization

Hybrid lemmatization uses multiple approaches, including rule-based methods, statistical models, and neural networks. For example, a hybrid system might first apply rule-based lemmatization to regular words and then use a neural model for irregular cases or unknown words.

Stemming and lemmatization functions are often found together in text mining and analytics systems. They strip words down to their core form and pass the resulting word roots for further processing. However, lemmatization is more advanced in its method and output than stemming.

Lemmatization offers the following benefits over stemming algorithms.

• It more precisely and accurately determines the word lemma than the stemming root.
• It understands full sentence input and considers grammar rules before choosing the lemma.
• It can understand a term based on its contextual use.

How they work

Stemming algorithms only remove word suffixes by checking word endings against a pre-defined list of common suffixes. For example, common suffixes like "ing," "ed," and "er" can be removed from most words as they typically indicate inflectional forms. So stemming algorithms return "burn" for the inflected words "burning," "burned," and "burner." However, this approach can result in errors. For example, the word "nothing" is reduced to "noth," which is inaccurate. Similarly, all the different forms of the word "carry," like "carries" and "carried," reduce to "carri" instead of the actual word "carry."

Unlike stemming, lemmatization takes a more nuanced approach to word reduction. It carefully considers the word's context and correct meaning before returning an output. Several words may also have a common root form unrelated to their spelling. For example, "better" and "good" share a semantic root but differ significantly in spelling.

AWS pre-trained artificial intelligence (AI) services easily integrate with your applications to address common NLP tasks. For example,

• Amazon Comprehend identifies the language, key phrases, subject, sentiment, and main topics in any document library.
• Amazon Kendra intelligently searches multiple enterprise knowledge sources to retrieve information for employees and generative AI models.
• Amazon Lex is an AI chat builder that allows users to interact with any application using natural language voice or chat.
• Amazon Textract automatically extracts printed text, handwriting, layout elements, and data from any document.
• Amazon Translate analyzes and translates large volumes of text for all your localization needs.

Get started with natural language processing on AWS by creating a free account today.