Contextualized Embeddings Notes

Contextualized Embeddings

Introduction

  • Distributional word representations (word embeddings) like Word2Vec are trained on large corpora and widely used in modern NLP tasks.
  • Traditional word embeddings obtain a single global representation for each word, disregarding context.
  • Contextual embeddings move beyond global word representations, achieving significant performance improvements in various NLP tasks.
  • Each token is associated with a representation that is a function of the entire input sequence.
  • Context-dependent representations capture syntactic and semantic properties of words in diverse linguistic contexts.

Word Embeddings

  • A text corpus is represented as a sequence S of tokens (t<em>1,t</em>2,…,tN)(t<em>1, t</em>2, …, t_N).
  • Distributed representations of words associate each token t<em>it<em>i with a dense feature vector h</em>ih</em>i.
  • Traditional word embedding techniques aim to learn a global word embedding matrix E∈RV×DE ∈ ℝ^{V×D}, where V is the vocabulary size and D is the number of dimensions.
  • Each row eie_i of E corresponds to the global embedding of word i in the vocabulary V.

Contextualized Word Embeddings

  • Contextualized Word Embeddings associate each token t<em>it<em>i with a representation that is a function of the entire input sequence S , i.e., h</em>i=f(e<em>1,e</em>2,…,ei)h</em>i = f(e<em>1, e</em>2, …, e_i)
  • Each input token t<em>it<em>i is usually mapped to its non-contextualized representation e</em>ie</em>i first, before applying an aggregation function ff.
  • There are many model architectures for ff.

Unsupervised Pre-training via Language Modeling

  • The prototypical way to learn distributed token embeddings is via language modeling: a probability distribution over a sequence of tokens.
  • Given a sequence of N tokens (t<em>1,t</em>2,…,t<em>N)(t<em>1, t</em>2, …, t<em>N), a language model factorizes the probability of the sequence as: p(t</em>1,t<em>2,…,t</em>N)=∏<em>i=1Np(t</em>i∣t<em>1,t</em>2,…,ti−1)p(t</em>1, t<em>2, …, t</em>N ) = \prod<em>{i=1}^{N} p(t</em>i | t<em>1, t</em>2, …, t_{i-1})
  • Language modeling uses maximum likelihood estimation (MLE) to estimate model parameters.
  • L(θ)=∏<em>i=1Nf(x</em>i∣θ)L(θ) = \prod<em>{i=1}^{N} f(x</em>i | θ)
  • A left-to-right language model considers the left context (t<em>1,t</em>2,…,t<em>i−1)(t<em>1, t</em>2, …, t<em>{i-1}) of t</em>it</em>i for estimating the conditional probability.
  • Pre-training contextual embeddings can be divided into unsupervised methods (e.g., language modeling and its variants) or supervised methods (e.g., machine translation and natural language inference).

ELMO

  • ELMO stands for Embeddings from Language Models.
  • ELMO generalizes traditional word embeddings by extracting context-dependent representations from a bidirectional language model.
  • A forward L-layer LSTM and a backward L-layer LSTM are applied to encode the left and right contexts, respectively.

ELMO: Bidirectional Language Model

  • Bidirectional Language Model (BiLM) consists of forward and backward language models.

ELMO - Representations

  • At each layer jj, the contextualized representations are the concatenation of the left-to-right and right-to-left representations, obtaining N hidden representations (h<em>1,j,h</em>2,j,…,hN,j)(h<em>{1,j}, h</em>{2,j}, …, h_{N,j}) for a sequence of length N.

ELMO: Forward and Backward Language Models

  • Forward:
    • Given a sequence of N tokens, (t<em>1,t</em>2,…,t<em>N)(t<em>1, t</em>2, …, t<em>N), a forward language model computes the probability of the sequence by modeling the probability of token t</em>kt</em>k given the history (t<em>1,…,t</em>k−1)(t<em>1, …, t</em>{k-1}).
    • At each position k, each LSTM layer outputs a context-dependent representation.
    • The top layer LSTM output is used to predict the next token tk+1t_{k+1}.
  • Backward:
    • The process is similar to the forward model but proceeds in the reverse direction.

ELMO - Log-likelihood

  • A biLM combines both a forward and backward LM.
  • The formulation jointly maximizes the log likelihood of the forward and backward directions.
  • Separate parameters are maintained for the LSTMs in each direction.

ELMO: Task Specific

  • To use ELMo in downstream tasks, the (L+1)(L + 1)-layer representations (including the global word embedding) for each token kk are aggregated as:
    ELMo<em>ktask=γtask∑</em>j=0Ls<em>jtaskh</em>k,jELMo<em>{k}^{task} = γ^{task} \sum</em>{j=0}^{L} s<em>{j}^{task} h</em>{k,j}
  • stasks^{task} are layer-wise weights normalized by the softmax used to linearly combine the (L+1)(L + 1)-layer representations of the token kk, and γtaskγ^{task} is a task-specific constant.
  • Given a pre-trained ELMo, it is straightforward to incorporate it into a task-specific architecture for improving the performance.
  • Supervised models use global word representations x<em>kx<em>k in their lowest layers, these representations can be concatenated with their corresponding context-dependent representations ELMo</em>ktaskELMo</em>{k}^{task}.

ELMO: Results on Common Tasks

  • ELMo enhanced neural models show improvements over state-of-the-art single model baselines across six benchmark NLP tasks.
  • Tasks include Question Answering (SQUAD), Textual entailment (SNLI), Semantic role labeling (SRL), Coreference resolution (Coref), Named entity recognition (NER), and Sentiment analysis (SST-5).
  • The performance metric varies across tasks, including accuracy and F1 score.

ELMO: Availability

  • Pre-trained ELMo models are available at https://allenai.org/allennlp/software/elmo
  • AllenNLP is a deep library on top of PyTorch.
  • ELMo can process character-level inputs.
  • Japanese, Chinese, and Korean ELMo models are possible.

BERT

  • BERT stands for Bidirectional Encoder Representations from Transformers.
  • It was released in late 2018 by researchers at Google AI.
  • BERT offers an advantage over models like Word2Vec because Word2Vec has a fixed representation of each word regardless of the context.
  • BERT produces word representations that are dynamically informed by the words around them.
  • For example, Word2Vec would produce the same word embedding for the word “bank” in both sentences, while BERT would produce a different word embedding for “bank” for each sentence:
    • “The man was accused of robbing a bank.”
    • “The man went fishing by the bank of the river.”

BERT: Core Idea

  • The core idea behind BERT is to “fill in the blank” based on context.
  • BERT produces labeled data for originally unlabeled data. This technique is called Self-Supervised Learning.
  • For instance, complete the sentence: “The woman went to the store and bought a _ of shoes.”

BERT: Masked Language Model

  • Instead of predicting the next word in a sequence, BERT uses Masked Language Model (MLM).
  • It randomly masks words in the sentence and then tries to predict them.
  • Masking means that the model looks in both directions and it uses the full context of the sentence, both left and right surroundings, in order to predict the masked word.
  • Unlike ELMo, it takes both the previous and next tokens into account at the same time.
  • BERT is also able to perform next-sentence prediction.
  • BERT uses encoders only.

BERT: Main Goals

  • BERT uses the encoder part of the transformer architecture so that it understands semantic and syntactic language information.
  • The output of BERT are embeddings, not predicted next words.
  • To leverage these embeddings, other layer(s) need to be added on top, for example text classification or questions and answers.
  • Fine-tuning is required to adapt BERT to specific tasks.

BERT: Transformer

  • BERT is based on a Transformer.
  • Transformers consist of an encoder to read the text input and a decoder to produce a prediction for the task.
  • Self-attention learns the weight of the relationship between each word in an input sequence to other words in the input sequence.
  • For example: The animal took too long to cross the road because it was too slow.

BERT: Input

  • BERT needs the input to be massaged and decorated with some extra metadata:
    1. Token embeddings: A [CLS] token is added to the input word tokens at the beginning of the first sentence, and a [SEP] token is inserted at the end of each sentence.
    2. Segment embeddings: A marker indicating Sentence A or Sentence B is added to each token. This allows the encoder to distinguish between sentences.
    3. Positional embeddings: A positional embedding is added to each token to indicate its position in the sentence.

BERT: Input - Special Tokens

  • [SEP]: At the end of every sentence, we need to append the special [SEP] token.
    • This token is an artifact of two-sentence tasks, where BERT is given two separate sentences and asked to determine something (e.g., can the answer to the question in sentence A be found in sentence B?).
  • [CLS]: For classification tasks, we must prepend the special [CLS] token to the beginning of every sentence.
    • This token has special significance. BERT consists of 12 Transformer layers. Each transformer takes in a list of token embeddings and produces the same number of embeddings on the output (but with the feature values changed, of course!).
    • On the output of the final (12th) transformer, only [CLS] is used by the classifier for downstream task!

BERT: Input - Constraints

  • BERT has two constraints:
    • All sentences must be padded or truncated to a single, fixed length.
    • The maximum sentence length is 512 tokens.
  • Padding and Truncation:
    • Padding adds a special padding token to ensure shorter sequences will have the same length as either the longest sequence in a batch or the maximum length accepted by the model.
    • Truncation works in the other direction by truncating long sequences.

BERT: Masking

  • Masking original idea:
    • Replace 15% of the words in the input with a [MASK] token.
    • Run the entire sequence through the BERT attention-based encoder.
    • Predict only the masked words (based on the context provided by the other non-masked words in the sequence).
  • Problem: the model only tries to predict when the [MASK] token is present in the input, while we want the model to try to predict the correct tokens regardless of what token is present in the input.

BERT: Masking Solution

  • To deal with this issue, out of the 15% of the tokens selected for masking:
    • 80% of the tokens are actually replaced with the token [MASK].
    • 10% of the time, tokens are replaced with a random token.
    • 10% of the time, tokens are left unchanged.

BERT: Subwords

  • BERT uses subwords.
  • It is a compromise between meaningful embeddings and acceptable memory consumption between a character-level (small vocabulary) and a word-level tokenizer (large vocabulary).
  • Underlying principle: frequently used words should not be split into smaller subwords, but rare words should be decomposed into meaningful subwords.
  • For instance,