MLLMU-Bench: Multimodal Unlearning Benchmark

Overview

• Privacy risk in multimodal large language models (MLLMs) due to memorization of confidential data from large web-scale pretraining.

• MLLMU-Bench: a synthetic privacy-unlearning benchmark for MLLMs.

• Purpose: evaluate how well unlearning methods remove private knowledge while preserving model utility, across both visual and textual modalities.

• Key findings (high level): unimodal unlearning methods often excel in generation and cloze tasks; multimodal unlearning methods can be stronger for classification with multimodal inputs; there is a trade-off between unlearning strength and model utility.

Benchmark composition and scope

• Profiles: 500 fictitious profiles (GPT-4o generated) + 153 real celebrity profiles (CelebA) for utility tests.

• Questions: 14 questions per profile (7 image+text, 7 text-only) → 20,754 total questions across all data.

• Datasets within the benchmark:

  • Forget Set: targets for unlearning (5%, 10%, 15% of fictitious profiles).

  • Test Set: paraphrased questions and transformed images to assess generalization of unlearning.

  • Retain Set: remaining profiles used to measure model utility after unlearning.

  • Real Celebrity Set: real public figures used to assess unintended consequences on neighboring concepts.

• Modalities:

  • Multimodal: image + text inputs.

  • Unimodal: text-only inputs.

• Base MLLMs used in experiments:

  • LLaVA-1.5-7B

  • Idefics2-8B

• Fine-tuning baseline: vanilla model (fine-tuned on all data) used as reference.

• Forget data splits: 5%, 10%, 15% (Df) with 95%, 90%, 85% retained (Dr).

• Evaluation framework covers three axes: unlearning efficacy, unlearning generalizability, model utility.

Evaluation metrics and tasks

• Tasks and settings

  • Classification: predict key attributes (e.g., birthplace, occupation) using input ⟨image, x, y⟩ in multimodal or ⟨∅, x, y⟩ in unimodal.

  • Generation: free-form answers; measure factual and textual quality.

  • Cloze: fill-in-the-blank style, using only a name as prompt to test retention of sensitive details.

• Metrics

  • Classification accuracy: ext{Acc} = rac{1}{|X|} isplaystyle \sum_{x \u2208 X} \mathbb{I}(\, ext{pred}(x) = ext{ground-truth}(x)\,)

  • Multimodal classification: $\hat{y} = \underset{y \in Y}{\mathrm{argmax}}\ P(y \mid \text{image}, x, M)$

  • Generation quality: ROUGE-L recall and F1 (ROUGE-L score), plus a Factuality score via GPT-4o on a 1–10 scale.

  • ROUGE-L recall/precision: $\text{Recall} = \frac{\text{LCS}( ext{ground truth}, \text{generated})}{|\text{ground truth}|}, \quad \text{Precision} = \frac{\text{LCS}( ext{ground truth}, \text{generated})}{|\text{generated}|}, \quad \text{ROUGE-L} = \text{F1} = \frac{2 \cdot \text{Prec} \cdot \text{Rec}}{\text{Prec} + \text{Rec}}.$n- Factuality Score: GPT-4o assigns a score in [1, 10] based on factual alignment with ground truth.

  • Cloze: exact-match accuracy against ground-truth details.

• Additional utility benchmarks: MMMU (reasoning) and LLaVA-Bench (helpfulness/reasoning) to assess model capability beyond unlearning targets.

Unlearning methods and baselines

• Gradient-based unlearning methods (typical baselines from LLM unlearning):

  • Gradient Ascent (GA): maximize loss on forget set to encourage forgetting.

  • Gradient Difference: balance forgetting on forgetting set with preserving performance on retain set.

  • KL Minimization: minimize KL divergence on retain set to preserve original behavior while maximizing loss on forget set.

  • Negative Preference Optimization (NPO): cast unlearning as a preference optimization problem to reduce dependence on forget data while stabilizing utility.

• Prompt-based baseline: appending system prompts to discourage generation of private data (e.g., a prompt saying not to output personal information).

• Dataset-level approach: forget set vs retain set splits across multiple forget percentages to study robustness.

• Data- and model-level details:

  • Unlearning inputs can be multimodal (image+text) or text-only.

  • Some methods perform better in multimodal settings for classification; others perform better in unimodal settings for generation/cloze.

Data creation and quality control

• Profile generation:

  • Fictitious profiles generated by GPT-4o with diverse attributes.

  • Ground-truth data for each profile used to generate 14 questions per profile.

• Images:

  • All fictitious profile images generated via ThisPersonDoesNotExist-style workflows (StyleGAN-based) to ensure privacy.

  • Test-set images modified with Arc2Face to generate varied poses for generalization tests.

• Real celebrities:

  • 153 real CelebA-derived profiles used to evaluate model utility on real-world knowledge.

• Quality control:

  • Human experts validate image quality, profile-attribute alignment, and question-answer consistency.

  • Celebrities’ data cross-checked against trusted sources.

• Data organization:

  • Forget Set, Retain Set, Test Set, Real Celebrity Set cover multimodal and unimodal evaluation.

  • Each profile has 14 questions; 20k+ total questions across the dataset.

• Fine-tuning and data usage:

  • Vanilla model is fine-tuned on the entire dataset D for baseline comparison.

  • LoRA enabled; gradient accumulation used for some models; hyperparameters tuned per base model.

Experimental setup and hyperparameters

• Base models and hardware:

  • LLaVA-1.5-7B and Idefics2-8B; experiments run on NVIDIA L40 GPUs (48 GB).

• Fine-tuning setup (example hyperparameters):

  • LLaVA-1.5-7B: 4 epochs, 4 steps, batch size 2, optimizer Adam, LoRA True, gradient accumulation True, learning rate $2 \times 10^{-5}$.

  • Idefics2-8B: 4 epochs, 2 steps, batch size 4, optimizer Adam, LoRA True, gradient accumulation 4, learning rate $1 \times 10^{-5}$.

• Data splits for forget/retain:

  • Forget Set: 5%, 10%, 15% of the fictitious profiles.

  • Retain Set: 95%, 90%, 85% respectively.

Key findings and takeaways

• Unlearning effectiveness vs. model utility

  • GA and Gradient Difference often achieve strongest unlearning on forget sets but can significantly reduce Retain Set accuracy (utility).

  • KL Minimization and NPO tend to be more balanced, preserving utility while providing reasonable forgetting.

  • Prompt-based unlearning helps avoid generating sensitive content but generally underperforms gradient-based methods for forgetting strength, especially on Forget and Test Sets.

• Modality-specific trends

  • Multimodal unlearning tends to excel in classification when both image and text are used as inputs.

  • Unimodal unlearning (text-only) tends to perform better in generation and cloze tasks, but may degrade multimodal performance when evaluated jointly.

  • There is a trade-off: improving forget accuracy often harms retain accuracy and reasoning/human-like abilities; improving utility can reduce forgetting strength.

• Generalization and robustness

  • Unlearning generalizability tested via the Test Set (paraphrased questions and Arc2Face-transformed images) to simulate transformed or paraphrased data.

  • Results indicate varying degrees of generalization across methods and base models; no single method dominates across all tasks and modalities.

• Real-world implications

  • Multimodal entanglement makes unimodal-only unlearning less effective for comprehensive forgetting; modality-aware unlearning is needed.

  • There is a notable trade-off between forgetting private data and preserving neighboring concepts and general model capability.

Practical insights and recommendations

• For tight forgetting with acceptable utility, consider NPO or KL Minimization with careful tuning; avoid over-reliance on prompting alone for strong forgetting.

• If classification with multimodal inputs is primary, explore multimodal unlearning approaches; for generation/cloze emphasis, unimodal approaches may be more effective but watch for drop in multimodal tasks.

• Always evaluate across Forget Set, Test Set, Retain Set, and Real Celebrity Set to capture forgetting, generalization, and utility jointly.

• Consider multi-faceted evaluation (ROUGE-L, Factuality via GPT-4o, cloze accuracy, and external reasoning benchmarks) for a holistic view of unlearning performance.

• Be mindful of potential unseen leakage through in-context learning (ICL) and plan mitigation strategies beyond the unlearning stage.

Limitations and future directions

• Root-cause gaps: while modality-aware results are observed, the underlying reasons why unimodal vs multimodal approaches differ are not fully understood and warrant theoretical analysis.

• Group forget and targeted attribute unlearning: current setup focuses on removing all information about an individual; future work could explore selectively forgetting specific attributes (e.g., residence) while preserving others.

• Real-world attack exposure: future benchmarking could incorporate data-extraction/ jailbreak-style attacks to test robustness of unlearning under adversarial conditions.

• Certified unlearning: development of formally certified/unverifiable unlearning guarantees for MLLMs remains an open area for future work.

Appendix highlights (metrics and procedures)

• A.1 Unlearning Efficacy:Forget Set evaluates whether the model behaves as if target data never existed, across multimodal and text-only settings; checks via multiple-choice, generation, and cloze tasks.

• A.2 Unlearning Generalizability: Test Set assesses performance on transformed/paraphrased data to ensure forgetting extends beyond the exact forget data.

• A.3 Model Utility: Retain Set and Real Celebrity Set measure performance on non-forgotten data and neighboring concepts; use MMMU and LLaVA-Bench for reasoning and helpfulness.

• A.4 ROUGE-L score: explained via Longest Common Subsequence (LCS) metrics; ROUGE-L F1 balances precision and recall.

• A.5 Factuality Score: GPT-4o-based evaluation on a 1–10 scale comparing generated outputs to ground-truth facts; few-shot prompts included.

• B Appendix: Data generation workflow details, including how GPT-4o was prompted to create profiles and how the vanilla model was fine-tuned.

• C Appendix: Unlearning algorithms formal definitions and objective functions (GA, Gradient Difference, KL Minimization, NPO, and prompting-based prevention).

• D Appendix: Additional experiments across modalities and forget data splits; trade-off analyses for Idefics2-8B and LLaVA models.

• E Appendix: Case studies and error analysis showing where unlearning methods may incorrectly retrieve neighboring facts.

This set of notes captures the essential structure, methods, metrics, and findings of the MLLMU-Bench work, focusing on quick recall for exam review and high-level understanding of multimodal unlearning in LLM-based systems.

• Privacy in Healthcare: Why is privacy crucial/relevant?

  • Crucial for Multimodal Large Language Models (MLLMs) due to the risk of memorization of confidential data from large web-scale pretraining.

• Tell me about PPML techniques

  • The note focuses on "unlearning methods" to remove private knowledge, which align with Privacy-Preserving Machine Learning (PPML) techniques:

    • Gradient-based methods: Gradient Ascent (GA), Gradient Difference, KL Minimization, Negative Preference Optimization (NPO).

    • Prompt-based baseline: Appending system prompts to discourage generation of private data.

    • Dataset-level approach: Managing forget set vs. retain set splits.

• Any privacy applications/concerns to (healthcare) chatbots in particular?

  • MLLMs, which can be core components of advanced chatbots (including those in healthcare), face significant privacy risks from memorizing confidential data during pretraining.

  • Unlearning methods are designed to mitigate these risks by removing private knowledge while preserving model utility.

• Methodological issues - why is privacy neglected? I.e. the performance tradeoffs?

  • Performance Trade-offs:

    • There is a notable trade-off between unlearning strength (how well private data is forgotten) and overall model utility (preserving non-private knowledge and capabilities).

    • Methods achieving the strongest unlearning (e.g., GA, Gradient Difference) often significantly reduce accuracy on retained data, impacting general model performance.

    • Improving forgetting accuracy can harm retain accuracy and the model's reasoning or human-like abilities, and vice versa.

  • Methodological Challenges:

    • Multimodal entanglement means that unimodal-only unlearning is less effective for comprehensive forgetting; modality-aware unlearning is required for MLLMs.