Explainability Issues in Large Language Models

Explainability Issues

Introduction

• Explainability: The capability to explain or present the behavior of models in human-understandable terms.
• Explainability of LLMs is crucial for two key reasons:
  • General End Users: Builds appropriate trust by showing the reasoning mechanism behind model predictions in an intuitive and understandable way.
  • Researchers and Developers: Explaining model behaviors provides insight to identify unintended biases, risks, and areas for performance improvements.

Explainability Paradigms

• LLM explainability can be categorized into two major paradigms:
  • Traditional Fine-Tuning
  • Prompting
• These paradigms encompass different kinds of explainability techniques associated with LLMs.

Explainability Based on Traditional Fine-Tuning

• Focuses on two key areas:
  • Understanding how self-supervised pre-training enables models to acquire a foundational understanding of language (e.g., syntax, semantics, and contextual relationships).
  • Analyzing how the fine-tuning process equips these pre-trained models with the capability to solve downstream tasks effectively.

Explanation Techniques for Traditional Fine-Tuning

• Traditional fine-tuning = pre-training + downstream fine-tuning
• Two types of explanations:
  • Local Explanation: Aims at providing an understanding of how a language model makes a prediction for a specific input instance.
  • Global Explanation: Understands how the LLM works overall.

Local Explanation

• Encompasses four main types of approaches:
  • Feature Attribution-Based Methods:
    • Measure the relevance of each input feature (e.g., words, phrases) to a model’s prediction.
    • Given an input text $x$ comprised of $n$ words $\lbrace x<em>1, x</em>2, …, x<em>n \rbrace$ and a fine-tuned language model $f$ generating an output $f(x)$, the feature attribution-based methods assign a relevance score $R(x</em>i)$ to the input word $x_i$ to reflect its contribution to the model prediction $f(x)$.
  • Perturbation Methods:
    • Perturb input and evaluate output changes.
  • Gradient-Based Methods:
    • Use partial derivatives of the output with respect to each input dimension.
    • Example: Integrated Gradients.
  • Surrogate Models:
    • Use human-comprehensible models like LIME and SHAP.

Integrated Gradients

• Given a model $F$, an input $x ∈ ℝ^n$ and a reference (zero) baseline $x' ∈ ℝ^n$, attribution scores are calculated.
• Gradients of the model’s output with respect to each input feature $x$ are computed.
  • $x$: Actual input.
  • $x'$: Baseline input.
  • $F$: Model.
  • $i$: Feature index.

Integrated Gradients - Steps

1. Select a Baseline Input: A neutral input sequence like a sequence of [PAD] tokens or all zeros.
2. Interpolate Inputs: Create a sequence of inputs that smoothly transitions from the baseline to the actual input.
3. Calculate Gradients: For each interpolated input, calculate the gradient of the output with respect to the input.
4. Integrate the Gradients: Sum the gradients along the path to get the total contribution of each input feature.
5. Multiply and Obtain Attribution: Multiply the average gradient by the difference between the actual input and the baseline.

Integrated Gradients - Examples

• Example sentence: "The movie was surprisingly good and well-acted."
• Analyzed using:
  • BERT: Used for classification (sentiment analysis).
  • GPT: Used for generation (predicting next tokens).

Integrated Gradients (BERT)

• Shows Token Importance Score (IG) for each token.
• Visual representation of importance (███).
• The most influential tokens contribute significantly.

Integrated Gradients (BERT) - Examples

• Examples demonstrating attribution scores, predicted labels, and word importance.
• Shows how the model interprets positive and negative sentiments.

Integrated Gradients (GPT)

• Identifies which tokens contribute most to predicting the next token.
• Example: "The movie was surprisingly good" contributing to the prediction of "and".
• Shows Token Importance Score (IG) and visual representation.

Integrated Gradients - Comparison

LIME

• LIME: Locally Interpretable Model-Agnostic Explanations.
• A post-hoc model-agnostic explanation technique.
• Aims to approximate any black box machine learning model with a local, interpretable model to explain each individual prediction.
• The explanations provided by LIME for each observation $x$ is obtained as follows:
  • $L$ = locality aware loss (e.g. MSE)
  • $G$ = interpretable models (e.g. decision tree)
  • $\Omega(x)$ = model complexity

LIME - Steps

1. Choose a Prediction to Explain:
   • Input: "The food was delicious, but the service was slow."
   • Prediction: Neutral.
2. Perturb the Input:
   • LIME creates many slightly modified versions of the input (called "perturbations").
   • It randomly removes or masks words:
     • "The food was delicious, but the service was ___."
     • "___ food was delicious, but the service was slow."
3. Evaluate How the Prediction Changes w.r.t. Each Perturbed Input:
   • Example: Input: "The food was delicious, but the service was _." Prediction: Positive.
4. Assign a Weight to Each Perturbed Sentences:
   • Perturbations that are more similar to the original input get more weight (usually using cosine similarity or some kernel function).
5. Fit a Simple Model:
   • Like linear regression on the weighted perturbations.

LIME - Evaluating Prediction Changes

• Shows how predictions change with perturbed inputs.
• Example:
  • Perturbed Input: The food was delicious, but the service was
  • Prediction: Neutral (0.52)
• These input-output pairs tell us how the model behaves around the original input.

LIME - Assigning Weights

• Computes similarity scores between each perturbed input and the original.
• Cosine similarity over binary word presence vectors.
• Example:
  • Perturbed Input: The food was delicious, but the service was
  • Similarity Score (to original): 0.95

LIME - Fitting a Simple Model

• Step 5: Fit a simple model (like linear regression) on the weighted perturbations.
• Inputs: binary vectors showing which words were kept (1) or removed (0)
• Targets: the model's prediction for each perturbed input
• Weights: how similar each perturbed input is to the original
• LIME fits a weighted linear regression model:

LIME - Interpreting Coefficients

• Example:
  • Word: delicious, Coefficient (β): +0.45, Interpretation: Strongly positive
  • Word: slow, Coefficient (β): -0.35, Interpretation: Strongly negative

LIME - Example Visualization

• Shows prediction probabilities for positive, negative, and neutral sentiments.
• Text with highlighted words indicating their impact on the prediction.

LIME vs Integrated Gradients

SHAP

• SHAP: SHapley Additive exPlanations.
• Based on the game theoretically optimal Shapley values.
• Basic idea: SHAP measures the impact of variables taking into account the interaction with other variables.

SHAP - Simplified Inputs

• SHAP makes use of simplified inputs: binary variables corresponding to the presence of words.
• Those simplified inputs can be (locally) mapped back to original inputs through a mapping function $h_x$.
• Example: "I am happy" $x=(0.523,8.35,…)$ $x'=(1,1,1)$

SHAP - Local Surrogate Model

• SHAP generates a local surrogate model $g$ that is locally similar to the original model.
• From the interpretable model $g$, feature attribution consists in distributing the prediction outcome between the different variables:
  • $z_i ∈ \lbrace 0,1 \rbrace ^M$ coalition vector
  • $\phi_j ∈ ℝ$ is the feature attribution for a feature $j$
  • $\phi_0$ complete absence of words

SHAP - Interpreting Feature Attribution

• For the sentence "I am happy", $\phi<em>0$ would be the prediction in the complete absence of words, $\phi</em>0 + \phi_1$ would be the local model prediction $g(1,0,0)$ for a virtual example with only the word "I", etc.

SHAP - Example

• Let’s consider the sentence: "The movie was excellent but too long."
• Suppose the model gives a score: 0.65 (positive).
• SHAP tries to break down this 0.65 into contributions from each word.

SHAP - Shapley Value Concept

• Each word/token is a player in a cooperative game.
• The model’s prediction is the payout.
• SHAP computes how much each feature contributes on average to all possible combinations of features.

SHAP - Coalitions and Contributions

1. Creates many subsets of the input features (like different combinations of tokens).
2. For each subset, it masks out the missing features and gets the model’s output.
3. It then adds each feature in, one at a time, and sees how the prediction changes.

• Example:
  • Subsets: "The movie was ___ " Prediction: 0.50
  • Subsets: "The movie was excellent" Prediction: 0.75
  • Contribution of "excellent": +0.25
• Repeat for all possible subsets.

SHAP - Average Marginal Contribution

• SHAP computes the average marginal contribution of each token/word across all possible subsets of features:
  • 0 words are present (empty set)
  • 1 word is present
  • 2 words are present
  • … N−1 words are present (all except one)
• Repeat for all possible subsets

SHAP - Example of Marginal Contribution

• Coalition: [The, movie, was, surprisingly] Prediction: 0.60 Marginal Contribution of "good": +0.10
• Coalition: [The, movie, was] Prediction: 0.50 Marginal Contribution of "good": +0.20
• Coalition: [The, movie, surprisingly] Prediction: 0.55 Marginal Contribution of "good": +0.15
• Coalition: [surprisingly, good] Prediction: 0.80 Marginal Contribution of "good": +0.25
• Exact SHAP is exponential in feature size
• In practice, SHAP samples a subset of possible word combinations (e.g., 1000–5000) to estimate contributions.

SHAP - Formula

• The shap value for each token/word $i$:

$\phi<em>i(f, x) = \sum</em>{S \subseteq M \setminus \lbrace i \rbrace} \frac{|S|! (|M| - |S| - 1)!}{|M|!} (f<em>S \cup \lbrace i \rbrace (x</em>S \cup \lbrace i \rbrace) - f<em>S(x</em>S))$

• Where:
  • $M$: Full set of words/tokens
  • $S$: Subset of features that does not include word/token $i$
  • $f<em>S(x</em>S)$: Model prediction using the words/tokens in $S$
  • $f<em>{S \cup \lbrace i \rbrace} (x</em>{S \cup \lbrace i \rbrace})$: Model prediction when word/token $i$ is added to $S$

SHAP - Describing the Formula

• $\frac{|S|! (|M| - |S| - 1)!}{|M|!}$: weight based on the number of subsets
• $(f<em>S \cup \lbrace i \rbrace (x</em>S \cup \lbrace i \rbrace) - f<em>S(x</em>S))$: marginal contribution of word/token i

SHAP - Visualization Example

• Shows SHAP values (impact on model output) for different features (words).
• Features are ranked by their SHAP value (impact on model output).

Local Explanation Techniques Recap

• Encompasses four main types of approaches:
  • Feature Attribution-Based Methods:
    • Measure the relevance of each input feature (e.g., words, phrases) to a model’s prediction.
  • Perturbation Methods:
    • Perturb input and evaluate output changes.
  • Gradient-Based Methods:
    • Use partial derivatives of the output with respect to each input dimension.
    • Example: Integrated Gradients.
  • Surrogate Models:
    • Use human-comprehensible models like LIME and SHAP.