Solving Analogies with Embeddings

Solving Analogies with Embeddings

Word Analogy Task

• Given two pairs of words that share a relation (e.g., “man:woman”, “king:queen”).
• The identity of the fourth word (“queen”) is hidden, and the task is to infer it based on the other three by answering “man is to woman as king is to — ?”.
• More generally, represented as a:a* as b:b*.

Vector Arithmetic

• a:a* as b:b. b is a hidden vector.
• b* should be similar to the vector $b - a + a^*$.
• Example: vector('king') - vector('man') + vector('woman') ≈ vector('queen').

Analogy and Embeddings

• Embeddings capture relational meaning.
• $b^<em>$ should be similar to the vector $b - a + a^</em>$.
• Example: vector('king') - vector('man') + vector('woman') ≈ vector('queen').
• This analogy question can be solved by optimizing the vector arithmetic.
• Example:vector('Paris') - vector('France') + vector('Italy') ≈ vector('Rome')

Representing Phrases with Vectors

• Mikolov et al. constructed representations for phrases as well as for individual words.
• To learn vector representations for phrases, find words that appear frequently together but infrequently in other contexts, and represent these n-grams as single tokens.
• For example, “New York Times” becomes NewYorkTimes and “Toronto Maple Leafs” becomes TorontoMapleLeafs.
• A bigram like "this is" remains unchanged.

Analogical Reasoning Task for Phrases

• Examples:
  • Newspapers: New York :: New York Times
  • Newspapers: Baltimore :: Baltimore Sun
  • NHL Teams: Boston :: Boston Bruins
  • NBA Teams: Detroit :: Detroit Pistons
  • Airlines: Austria :: Austrian Airlines
  • Company executives: Steve Ballmer :: Microsoft
• The goal is to compute the fourth phrase using the first three.
• A model achieved an accuracy of 72% on this dataset.

Vector Compositionality

• Mikolov et al. experiment with using element-wise addition to compose vectors.
• Examples:
  • Czech + currency = koruna
  • Vietnam + capital = Hanoi
  • German + airlines = airline Lufthansa
  • Russian + river = Moscow
  • French + actress = Juliette Binoche
• To maximize accuracy on the phrase analogy task, the amount of training data was increased by using a dataset with about 33 billion words.
• Using hierarchical softmax, dimensionality of 1000, and the entire sentence for the context resulted in a model that reached an accuracy of 72%.
• Accuracy was 66% when the size of the training dataset was reduced to 6B words, suggesting that a large amount of training data is crucial.

Sentence Embeddings

• Use word embeddings computed using one of the popular methods on unlabeled corpus like Wikipedia.
• Represent the sentence by a weighted average of the word vectors.

Embeddings and Word History

• Train embeddings on old books to study changes in word meaning.
• Diachronic word embeddings for studying language change.

Visualizing Changes

• Project 300 dimensions down into 2 for visualization.
• Examples of word changes over time include:
  • "gay": daft (1900s) → witty (1950s) → homosexual (1990s)
  • "broadcast": seed (1850s) → newspapers (1900s) → television (1990s)
  • "awful": solemn (1850s) → horrible (1900s) → weird (1990s)

Evolution of Sentiment Words

• Example: "terrific" shifted from causing terror to extremely good over time.

Embeddings and Bias

• Embeddings reflect cultural bias.
• Examples:
  • Paris : France :: Tokyo : x (x = Japan)
  • father : doctor :: mother : x (x = nurse)
  • man : computer programmer :: woman : x (x = homemaker)

Implicit Association Test

• Measures how associated concepts (flowers, insects) are with attributes (pleasantness, unpleasantness).
• Studied by measuring timing latencies for categorization.
• Psychological findings on US participants:
  • African-American names are associated with unpleasant words more than European-American names.
  • Male names are associated more with math, female names with arts.
  • Old people's names with unpleasant words, young people with pleasant words.

Embeddings and Bias Replication

• Caliskan et al. replication with embeddings:
  • African-American names (Leroy, Shaniqua) had a higher GloVe cosine with unpleasant words (abuse, stink, ugly).
  • European American names (Brad, Greg, Courtney) had a higher cosine with pleasant words (love, peace, miracle).
• Embeddings reflect and replicate all sorts of pernicious biases.

Debiasing Algorithms

• Debiasing algorithms for embeddings.
• Use embeddings as a historical tool to study bias.

Historical Analysis with Embeddings

• The cosine similarity of embeddings for decade X for occupations (like teacher) to male vs female names.
• Is correlated with the actual percentage of women teachers in decade X.

Gender Bias in Embeddings

• Embeddings for competence adjectives are biased toward men (smart, wise, brilliant, etc.).
• This bias is slowly decreasing.

Princeton Trilogy Experiments

• Attitudes toward ethnic groups (1933, 1951, 1969) scores for adjectives (industrious, superstitious, nationalistic, etc.).
• Cosine of Chinese name embeddings with those adjective embeddings correlates with human ratings.

Change in Linguistic Framing

• Change in association of Chinese names with adjectives framed as "othering" (barbaric, monstrous, bizarre).

Asian Bias Over Time

• Top Asian adjectives in 1910, 1950, and 1990 by relative norm difference in the COHA embedding.

Conclusion

• Embeddings = vector models of meaning.
• More fine-grained than just a string or index.
• Especially good at modeling similarity/analogy.
• Download them and use cosines.
• Can use sparse models (tf-idf) or dense models (word2vec, GLoVE).
• Useful in practice but know they encode cultural stereotypes.