Language Models: Static Word Embeddings

Turing Test

If a machine pretend to be a human well enough in a (blind) conversation to fool a human into thinking it is human

Language Models

A language model is a probability distribution over sequences of words. Using it, we can predict the occurrence of words.

Language Models examples

● n-grams

● Static embeddings: Word2vec, GloVe, FastText, etc.

● Contextualized word embeddings: BERT, Llama, GPT-2 / GPT-3 / GPT-4, DeepSeek R1, etc.

Applications of Language Models

● Natural language understanding (NLU): Question answering systems, Chatbots

● Natural language generation (NLG): Text summarization, Automated journalism, Chatbots

● Machine translation

● Text classification

● Named entity recognition

● Opinion mining

● Sentiment analysis

Word similarity applications

● Spell checking: Similarity between individual words

● Search: Similarity between a sentence and the content of a document

● Duplicate detection: Similarity between two documents

● Summarization: Removal of redundant (= similar) sentences in a document

● Translation: Finding a similar word in a different language

WordNet Limitations

• missing nuance

• missing new meanings of words until they are added

• is subjective and biased by the annotators’ perspectives

• Requires constant human labor to create and update

Similaeity in Vector spaced model

● Represent the query as a tf-idf encoded vector

● Represent each document as a tf-idf encoded vector

● Compute cosine similarities between query and documents

One Hot Vector Encodings

● The dimension of the vector space is equal to the vocabulary size

● Each word (or lemma, or stem) corresponds to exactly one dimension

● A single word is encoded by a one-hot vector:

○ All but one vector components are equal to 0 (these components are “cold”)

○ Only the component in the dimension that corresponds to the word itself has a value of 1 (the component is “hot”)

● Individual word vectors are orthogonal by definition

● There is no notion of word similarity in the vector space model

Issues with Vector space model

The vector space model is designed to compare sequences of text based on the words they contain.

=>Historical solution in information retrieval: Query expansion

Keep a dictionary of similar words and use it to expand the querie

Query expansion

a (set of) heuristic(s) specifically designed to work around the inability of the vector space model to represent semantic meaning for document search

Term-concept matrix

in which each word is ranked according to the strength of its relation to a semantic concept.

The rows of such a matrix could then be used as vector representations of the words. Similar words would have similar vectors.

Embedding of words

Given a training corpus, the embedding of words is the process of assigning a vector (in a latent space) to each word, based on the content of the corpus (by using some algorithm)

Word embedding:

The vector that is assigned to a word as the result of the above process is often called the embedding of the word.

Latent Space

A latent space (also called embedding space) is a vector space in which items are placed in such a way that similar items are placed in proximity. Typically (but not always), latent spaces have a lower rank (dimensionality) than the original space of the data.

Embedding Pipeline

● Place similar and/or related words in close proximity in the latent space

● Use word (co-)occurrence information from the corpus

● Work without supervision (labeling data at this scale is infeasible)

Dimensionality Reduction For Term Doc Matrix

Use dimensionality reduction techniques to reduce the corpus dimension to a manageable size but leave the vocabulary dimension as it is. The resulting row vectors are dense and capture information regarding the occurrence of words in the documents

Latent Semantic Analysis (LSA)

● Compute a singular value decomposition A = UDV of the term-document matrix A

● Reduce the number of dimensions to retain only the k most important dimensions

● Use the row vectors of U_k as word embeddings

Disadvantages of LSA

● It works solely based on global cooccurrence statistics (it only knows whether words occur in the same documents)

● It does not enable us to use compositional semantics

Distributional Semantics:

● The meaning of words is indicated by the context in which they occur

=> using word cooccurrence information.

● Word embeddings derived from this principle are called distributed representations

Cooccurrence: Context and Context Windows

● When a word w appears in a text, its context is the set of words that appear nearby(before n after)

● We use the combined contexts to learn the representation of word w

Cloze Test

The cloze test (also called cloze deletion test) is a fill-in-the-blanks style examination task.

word2vec word embeddings variations

● Continuous Bag of Words or CBOW: Given some context, learn to predict a missing word.

● Skipgram: Given a word, learn to predict the context

Skipgram Training Data

We generate training data by:

● Iterating over the corpus step-by-step with a fixed window size

● Extracting pairs of the input word in the center of the window and one target word

Generating training data for CBOWworks the same way, except input and target words are reversed

Skipgram: Negative Sampling

add negative examples by picking random output words

Word2vec Model Initialization:

• Word2vec is a shallow neural network architecture with just two layers.

• The layers are an embedding layer (used for learning representations of the input words) and a context layer (used for learning representations of the output words.

• The matrices are initialized with random values.

• The embedding size is the selected dimensionality of our latent space (typically 300 dimensions).

Word2vec Model Training

repeatedly iterate over the training data.

● For each input word, we find the corresponding row in the embedding matrix and retrieve the current embedding

● For the positive and negative output words, we find the corresponding rows in the context matrix and retrieve the current embeddings

For each embedding-context pair, we compute the dot product (≈ cosine similarity)

● The similarity is used to predict the likelihood of the target occurring in the context

● We use the error to update the embeddings in both matrices

● This process is repeated for multiple cycles over all data points until it converges

Word2vec Output

After convergence is reached, we discard the context matrix and use the embedding matrix for our word embeddings. Each row contains the embedding of the corresponding word in the vocabulary.

Intuition of word2vec

● During initialization, each word starts at a random location in the latent space

● During training, each word is pulled closer towards frequently cooccurring words

● During training, each word is pushed away from non-cooccurring words

● Words end up in the proximity of other words that occur in similar contexts

Additive Compositionality

Word2vec was created with the idea of supporting arithmetic solution of analogy tasks

Compositional Semantics

Common semantic concepts are implicitly encoded as directions in the latent space

○ Gender

○ Verb tenses

○ Country/capital relations

○ etc.

Caution abt similarity

Similarity=/= Relatedness

CBOW

● Predicts target words from the surrounding context words.

● Statistically, CBOW smoothes over a lot of the distributional information (by treating an entire context as one observation).

● Works good with high number of samples (bigger dataset).

● Several times faster to train than the skip-gram, slightly better accuracy for the frequent words.

Skipgram

● Predicts surrounding context words from the target words (inverse of CBOW).

● Statistically, skip-gram treats each context-target pair as a new observation.

● Works well with low number of samples (smaller dataset).

● Represents well even rare words or phrases.

Improve accuracy of word2vec

○ the choice of model architecture (CBOW (faster) or Skip-Gram (better))

○ increasing the training data set

○ increasing the number of vector dimensions

○ increasing the window size of words considered by the algorithm

Each of these improvements comes with the cost of increased computational complexity and therefore increased model generation time