Embeddings-2

Distributed Word Representations

  • Presented by Radu-Casian Mihailescu, Associate Professor.

  • Contact: r.mihailescu@hw.ac.uk

Outline

  • Sparse Embedding/Count-based Algorithms:

    • 1-hot encoding

    • TF-IDF

  • Dense Embeddings/Predict-based Algorithms:

    • Word2vec

    • CBOW (Continuous Bag of Words)

    • Skip-gram

  • Embeddings Properties and Visualization

How Do We Represent Words?

  • Need to learn embeddings from words to vectors.

  • Function f(word): returns a vector encoding for any word.

  • Sparse Representations:

    • 1-of-K or 1-hot encoding

    • TF-IDF

  • Limitations:

    • Scalability issues.

    • Lacks encoding of relationships between words.

One-hot Encoding

  • Words represented by one-hot vectors.

  • Example sentence: "The motel room is modern."

  • Vocabulary example:

    • Vocabulary = {the, motel, room, is, modern, I, reserved, a, at, Hilton, hotel}

  • Vector dimension equals the number of words in the vocabulary (e.g., 500K).

  • One-hot representation:

    • the = [1,0,0,...]

    • motel = [0,1,0,...]

    • room = [0,0,1,...]

    • hotel = [0,0,...,1]

  • Drawback: No encoding of similarity between words.

Distributional Hypothesis

  • Words in similar contexts have similar meanings.

  • Use context vectors to represent meanings:

    • Each word is a vector.

    • Similar words reside "nearby" in space.

  • Learning Word Embeddings:

    • Context-counting methods.

    • Context-predicting methods.

Context-counting Methods

  • Term-document matrix:

    • Cells represent the count of word occurrences in documents.

    • Matrix size: Vocabulary size |V| x Document size |D|.

    • Each document as a vector of words.

  • Word Vectors:

    • Examples:

      • "battle" appears in certain contexts (e.g. Julius Caesar).

      • "fool" appears in comedies.

  • Term-Term Matrix:

    • Represents co-occurrences of words in context windows.

    • Size: |V|x|V|.

  • Limitations of Raw Frequency:

    • Overly frequent words (e.g., "the") are less informative.

  • TF-IDF (Term Frequency-Inverse Document Frequency):

    • A solution for word weighting.

    • dft: Number of documents containing the term.

    • idft: Emphasizes discriminative words (inverse frequency).

    • Formula for significance measurement.

Sparse vs. Dense Embeddings

  • TF-IDF:

    • Sparse embeddings (most elements = 0).

    • Longer vectors (20,000 to 50,000 elements).

  • Dense Embeddings:

    • Shorter vectors (50 to 1000 elements), often non-zero.

    • More efficient in storage and computation, better performance.

Context-predicting Methods - Word2vec

  • Input:

    • Large text corpus, predefined vocabulary, and vector dimension.

  • Process:

    • Use a classifier to predict context words for target words.

    • The focus is on learned classifier weights as word embeddings.

Word2vec Variants

  • CBOW (Continuous Bag of Words):

    • Predict target word based on surrounding context words.

  • Skip-gram:

    • Predict surrounding context words based on a target word.

CBOW Methodology**

  • Context window (e.g., +/- 2 words).

  • Predict target word using its context without considering word order.

  • Learn from input layer to hidden layer to output.

Skip-gram Methodology**

  • Treat target word and neighboring words as positive, while random samples are negative.

  • Adjust word vectors to maximize similarity for positive pairs and minimize it for negative pairs.

Properties of Word Embeddings

  • Parallelogram model of analogical reasoning.

  • Example: king - man + woman approximates queen.

  • Relational Meaning: Captures concepts and relationships.

  • Vectors illustrate relationships between countries and their capitals.

  • PCA projection of vectors shows automatic organization.

Visualizing Word Embeddings

  • Clusters can be visualized using 2-D t-SNE plots.

Summary

  • Embeddings: Vector models reflecting language use from text corpora.

  • Applications: Good for modeling similarity and analogy in NLP tasks.

  • Types: Sparse and dense embeddings, derived from count or predictive methods.