Embeddings-2
Distributed Word Representations
Presented by Radu-Casian Mihailescu, Associate Professor.
Contact: r.mihailescu@hw.ac.uk
Outline
Sparse Embedding/Count-based Algorithms:
1-hot encoding
TF-IDF
Dense Embeddings/Predict-based Algorithms:
Word2vec
CBOW (Continuous Bag of Words)
Skip-gram
Embeddings Properties and Visualization
How Do We Represent Words?
Need to learn embeddings from words to vectors.
Function f(word): returns a vector encoding for any word.
Sparse Representations:
1-of-K or 1-hot encoding
TF-IDF
Limitations:
Scalability issues.
Lacks encoding of relationships between words.
One-hot Encoding
Words represented by one-hot vectors.
Example sentence: "The motel room is modern."
Vocabulary example:
Vocabulary = {the, motel, room, is, modern, I, reserved, a, at, Hilton, hotel}
Vector dimension equals the number of words in the vocabulary (e.g., 500K).
One-hot representation:
the = [1,0,0,...]
motel = [0,1,0,...]
room = [0,0,1,...]
hotel = [0,0,...,1]
Drawback: No encoding of similarity between words.
Distributional Hypothesis
Words in similar contexts have similar meanings.
Use context vectors to represent meanings:
Each word is a vector.
Similar words reside "nearby" in space.
Learning Word Embeddings:
Context-counting methods.
Context-predicting methods.
Context-counting Methods
Term-document matrix:
Cells represent the count of word occurrences in documents.
Matrix size: Vocabulary size |V| x Document size |D|.
Each document as a vector of words.
Word Vectors:
Examples:
"battle" appears in certain contexts (e.g. Julius Caesar).
"fool" appears in comedies.
Term-Term Matrix:
Represents co-occurrences of words in context windows.
Size: |V|x|V|.
Limitations of Raw Frequency:
Overly frequent words (e.g., "the") are less informative.
TF-IDF (Term Frequency-Inverse Document Frequency):
A solution for word weighting.
dft: Number of documents containing the term.idft: Emphasizes discriminative words (inverse frequency).Formula for significance measurement.
Sparse vs. Dense Embeddings
TF-IDF:
Sparse embeddings (most elements = 0).
Longer vectors (20,000 to 50,000 elements).
Dense Embeddings:
Shorter vectors (50 to 1000 elements), often non-zero.
More efficient in storage and computation, better performance.
Context-predicting Methods - Word2vec
Input:
Large text corpus, predefined vocabulary, and vector dimension.
Process:
Use a classifier to predict context words for target words.
The focus is on learned classifier weights as word embeddings.
Word2vec Variants
CBOW (Continuous Bag of Words):
Predict target word based on surrounding context words.
Skip-gram:
Predict surrounding context words based on a target word.
CBOW Methodology**
Context window (e.g., +/- 2 words).
Predict target word using its context without considering word order.
Learn from input layer to hidden layer to output.
Skip-gram Methodology**
Treat target word and neighboring words as positive, while random samples are negative.
Adjust word vectors to maximize similarity for positive pairs and minimize it for negative pairs.
Properties of Word Embeddings
Parallelogram model of analogical reasoning.
Example:
king - man + womanapproximatesqueen.Relational Meaning: Captures concepts and relationships.
Vectors illustrate relationships between countries and their capitals.
PCA projection of vectors shows automatic organization.
Visualizing Word Embeddings
Clusters can be visualized using 2-D t-SNE plots.
Summary
Embeddings: Vector models reflecting language use from text corpora.
Applications: Good for modeling similarity and analogy in NLP tasks.
Types: Sparse and dense embeddings, derived from count or predictive methods.