Contextualized Embeddings Notes
Contextualized Embeddings
Introduction
- Distributional word representations (word embeddings) like Word2Vec are trained on large corpora and widely used in modern NLP tasks.
- Traditional word embeddings obtain a single global representation for each word, disregarding context.
- Contextual embeddings move beyond global word representations, achieving significant performance improvements in various NLP tasks.
- Each token is associated with a representation that is a function of the entire input sequence.
- Context-dependent representations capture syntactic and semantic properties of words in diverse linguistic contexts.
Word Embeddings
- A text corpus is represented as a sequence S of tokens (t<em>1,t</em>2,…,tN).
- Distributed representations of words associate each token t<em>i with a dense feature vector h</em>i.
- Traditional word embedding techniques aim to learn a global word embedding matrix E∈RV×D, where V is the vocabulary size and D is the number of dimensions.
- Each row ei of E corresponds to the global embedding of word i in the vocabulary V.
Contextualized Word Embeddings
- Contextualized Word Embeddings associate each token t<em>i with a representation that is a function of the entire input sequence S , i.e., h</em>i=f(e<em>1,e</em>2,…,ei)
- Each input token t<em>i is usually mapped to its non-contextualized representation e</em>i first, before applying an aggregation function f.
- There are many model architectures for f.
Unsupervised Pre-training via Language Modeling
- The prototypical way to learn distributed token embeddings is via language modeling: a probability distribution over a sequence of tokens.
- Given a sequence of N tokens (t<em>1,t</em>2,…,t<em>N), a language model factorizes the probability of the sequence as:
p(t</em>1,t<em>2,…,t</em>N)=∏<em>i=1Np(t</em>i∣t<em>1,t</em>2,…,ti−1)
- Language modeling uses maximum likelihood estimation (MLE) to estimate model parameters.
- L(θ)=∏<em>i=1Nf(x</em>i∣θ)
- A left-to-right language model considers the left context (t<em>1,t</em>2,…,t<em>i−1) of t</em>i for estimating the conditional probability.
- Pre-training contextual embeddings can be divided into unsupervised methods (e.g., language modeling and its variants) or supervised methods (e.g., machine translation and natural language inference).
ELMO
- ELMO stands for Embeddings from Language Models.
- ELMO generalizes traditional word embeddings by extracting context-dependent representations from a bidirectional language model.
- A forward L-layer LSTM and a backward L-layer LSTM are applied to encode the left and right contexts, respectively.
ELMO: Bidirectional Language Model
- Bidirectional Language Model (BiLM) consists of forward and backward language models.
ELMO - Representations
- At each layer j, the contextualized representations are the concatenation of the left-to-right and right-to-left representations, obtaining N hidden representations (h<em>1,j,h</em>2,j,…,hN,j) for a sequence of length N.
ELMO: Forward and Backward Language Models
- Forward:
- Given a sequence of N tokens, (t<em>1,t</em>2,…,t<em>N), a forward language model computes the probability of the sequence by modeling the probability of token t</em>k given the history (t<em>1,…,t</em>k−1).
- At each position k, each LSTM layer outputs a context-dependent representation.
- The top layer LSTM output is used to predict the next token tk+1.
- Backward:
- The process is similar to the forward model but proceeds in the reverse direction.
ELMO - Log-likelihood
- A biLM combines both a forward and backward LM.
- The formulation jointly maximizes the log likelihood of the forward and backward directions.
- Separate parameters are maintained for the LSTMs in each direction.
ELMO: Task Specific
- To use ELMo in downstream tasks, the (L+1)-layer representations (including the global word embedding) for each token k are aggregated as:
ELMo<em>ktask=γtask∑</em>j=0Ls<em>jtaskh</em>k,j - stask are layer-wise weights normalized by the softmax used to linearly combine the (L+1)-layer representations of the token k, and γtask is a task-specific constant.
- Given a pre-trained ELMo, it is straightforward to incorporate it into a task-specific architecture for improving the performance.
- Supervised models use global word representations x<em>k in their lowest layers, these representations can be concatenated with their corresponding context-dependent representations ELMo</em>ktask.
ELMO: Results on Common Tasks
- ELMo enhanced neural models show improvements over state-of-the-art single model baselines across six benchmark NLP tasks.
- Tasks include Question Answering (SQUAD), Textual entailment (SNLI), Semantic role labeling (SRL), Coreference resolution (Coref), Named entity recognition (NER), and Sentiment analysis (SST-5).
- The performance metric varies across tasks, including accuracy and F1 score.
ELMO: Availability
- Pre-trained ELMo models are available at https://allenai.org/allennlp/software/elmo
- AllenNLP is a deep library on top of PyTorch.
- ELMo can process character-level inputs.
- Japanese, Chinese, and Korean ELMo models are possible.
BERT
- BERT stands for Bidirectional Encoder Representations from Transformers.
- It was released in late 2018 by researchers at Google AI.
- BERT offers an advantage over models like Word2Vec because Word2Vec has a fixed representation of each word regardless of the context.
- BERT produces word representations that are dynamically informed by the words around them.
- For example, Word2Vec would produce the same word embedding for the word “bank” in both sentences, while BERT would produce a different word embedding for “bank” for each sentence:
- “The man was accused of robbing a bank.”
- “The man went fishing by the bank of the river.”
BERT: Core Idea
- The core idea behind BERT is to “fill in the blank” based on context.
- BERT produces labeled data for originally unlabeled data. This technique is called Self-Supervised Learning.
- For instance, complete the sentence: “The woman went to the store and bought a _ of shoes.”
BERT: Masked Language Model
- Instead of predicting the next word in a sequence, BERT uses Masked Language Model (MLM).
- It randomly masks words in the sentence and then tries to predict them.
- Masking means that the model looks in both directions and it uses the full context of the sentence, both left and right surroundings, in order to predict the masked word.
- Unlike ELMo, it takes both the previous and next tokens into account at the same time.
- BERT is also able to perform next-sentence prediction.
- BERT uses encoders only.
BERT: Main Goals
- BERT uses the encoder part of the transformer architecture so that it understands semantic and syntactic language information.
- The output of BERT are embeddings, not predicted next words.
- To leverage these embeddings, other layer(s) need to be added on top, for example text classification or questions and answers.
- Fine-tuning is required to adapt BERT to specific tasks.
- BERT is based on a Transformer.
- Transformers consist of an encoder to read the text input and a decoder to produce a prediction for the task.
- Self-attention learns the weight of the relationship between each word in an input sequence to other words in the input sequence.
- For example: The animal took too long to cross the road because it was too slow.
- BERT needs the input to be massaged and decorated with some extra metadata:
- Token embeddings: A [CLS] token is added to the input word tokens at the beginning of the first sentence, and a [SEP] token is inserted at the end of each sentence.
- Segment embeddings: A marker indicating Sentence A or Sentence B is added to each token. This allows the encoder to distinguish between sentences.
- Positional embeddings: A positional embedding is added to each token to indicate its position in the sentence.
- [SEP]: At the end of every sentence, we need to append the special [SEP] token.
- This token is an artifact of two-sentence tasks, where BERT is given two separate sentences and asked to determine something (e.g., can the answer to the question in sentence A be found in sentence B?).
- [CLS]: For classification tasks, we must prepend the special [CLS] token to the beginning of every sentence.
- This token has special significance. BERT consists of 12 Transformer layers. Each transformer takes in a list of token embeddings and produces the same number of embeddings on the output (but with the feature values changed, of course!).
- On the output of the final (12th) transformer, only [CLS] is used by the classifier for downstream task!
- BERT has two constraints:
- All sentences must be padded or truncated to a single, fixed length.
- The maximum sentence length is 512 tokens.
- Padding and Truncation:
- Padding adds a special padding token to ensure shorter sequences will have the same length as either the longest sequence in a batch or the maximum length accepted by the model.
- Truncation works in the other direction by truncating long sequences.
BERT: Masking
- Masking original idea:
- Replace 15% of the words in the input with a [MASK] token.
- Run the entire sequence through the BERT attention-based encoder.
- Predict only the masked words (based on the context provided by the other non-masked words in the sequence).
- Problem: the model only tries to predict when the [MASK] token is present in the input, while we want the model to try to predict the correct tokens regardless of what token is present in the input.
BERT: Masking Solution
- To deal with this issue, out of the 15% of the tokens selected for masking:
- 80% of the tokens are actually replaced with the token [MASK].
- 10% of the time, tokens are replaced with a random token.
- 10% of the time, tokens are left unchanged.
BERT: Subwords
- BERT uses subwords.
- It is a compromise between meaningful embeddings and acceptable memory consumption between a character-level (small vocabulary) and a word-level tokenizer (large vocabulary).
- Underlying principle: frequently used words should not be split into smaller subwords, but rare words should be decomposed into meaningful subwords.
- For instance,