Natural Language Processing (NLP)

Why is NLP needed given the scale of text data today

Human language carries rich, complex information, and with millions of documents, tweets, and conversations generated, manual analysis is impossible

What is “unstructued data” and why is it challenging

Text, speech, and social media data that doesn’t fit nearly into tows/columns, it’s messy and ambiguous

How has the goal of text analysis shifted over time

From interpreting text based on keywords to understanding actual meaning behind the words

What is NLP

A field if AI focused on understanding human language, the ability to read, understand, and derive meaning from it, Combining CS, Linguistics, and Machine Learning

What factors are driving the growth of NLP

Explosion of available data and increased computational power, with growing applications across healthcare, finance, media, and HR

How can NLP help in healthcare

By extracting disease information and patient history from unstructured doctor’s notes and electronic health records, aiding disease prediction and treatment suggestions

What is sentiment analysis used for

Identifying and extracting information from sources like social media to determine what customers are saying about a product or service and their decision drivers

How do voice assistants like Siri and Alexa use NLP

They respond to vocal prompts to perform tasks like finding a shop, giving weather forecasts, suggesting routes, or controlling smart home devices

How is NLP used in spam filtering

Companies analyze the text in emails flowing through their servers to classify and filter out spam before it reaches the inbox

How is NLP used in recruitment

To identify the skills of potential hires during search and selection, and to spot prospects before they become active job seekers

What notable healthcare discovery did Microsoft make using NLP on search queries

By analyzing large samples of search engine queries, they could identify internet users suffering from pancreatic cancer even before a formal diagnosis

What are the four main stages of the NLP pipeline

1) text processing

2) feature extraction

3) modeling/learning

4) evaluation

Why is language processing broken into multiple distinct learning techniques rather than one step

Because language is extremely complex and tricky to understand and manipulate, so different techniques are used to handle different challenges before combining them.

What is Bag of Words (BoW)

A model that counts all words in a text, creating an occurrence matrix for a sentence of document while disregarding grammar and word order; these frequencies become classifier features.

What are the main limitations of Bag of Words

It lacks context or meaning, treats all words independently, lets common words like “the” or “is” dominate, has poor sematic understanding

What problem does TFIDF solve compared to plain Bag of Words

It rescales word frequency by how often the word appears across all documents, penalizing common words that appear frequently everywhere while rewarding unique or rare items

Does TFIDF solve BoW’s lack of context/semantics

No TFIDF improves on BoW through weighting, but it still has no context or semantic understanding

What is tokenization

The process of segmenting running text into sentences and words (tokens), typically discarding certain characters like punctuation

Why is tokenization harder than just splitting on blank spaces

Not all languages are segmented like English, even in english, splitting on spaces can incorrectly break up multiword names (San Francisco)

Why is tokenization especially trikcky in biomedical text

Because biomedical text contains lots of hyphens, parenthesis and other punctuation marks

What is stop word removal

The process of removing common, low information words like articles, pronouns, and prepositions from text before processing

Why is there no universal stop word list

Because what counts as an uninformative word depends on context; lists can be pre defined or built from scratch

What’s a key risk of removing stop words

It can wipe out relevant information and change a sentence’s meaning

What is stemming

The process of slicing the end of beginning of words to remove affixes (prefixes or suffixes)

What is a key limitation of stemming

Because stemmers use algorithmic rule based approaches, the result may not be an actual word or may even change the word’s or sentence’s meaning

Despite its limitations, why is stemming still used

It can help correct spelling errors in tokens, and stemmers are simple and very fast, which matters when speed and performance are priorities in an NLP pipeline

Lemmatization

The process of reducing a word to its base dictionary form (lemma) and grouping different forms of the same word together (went → go, best → good)

How does lemmatization differ from stemming in its approach

Lemmatization resolves words to their actual dictionary form using detailed dictionaries, rather than just algorithmically slicing affixes like stemming does

How does lemmatization handle word disambiguation

By taking context into account, (using a part of speech tag) it can distinguish between identical words with different meaning

Why is lemmatization more resource intensive than stemming

Because it considers context, dictionaries, and part of speech information rather than performing simple string slicing

What is Named Entry Recognition (NER)

An NLP technique that scans text to automatically detect and classify key entities into predefined categories

What techniques are commonly combined to build entity detection algorithms

Rule based parsing, dictionary lookups, POS tagging and dependency parsing

What are other names is NER known by

Entity Identification, entity extraction, or entity chunking

What are some applications of NER

Automated chatbots, content analyzers, and consumer insight tools

What are some commonly used open source libraries for NER

NLTK, spaCy, and Stanford NER