Back to Feed
Benchmarks & Evals / Multimodal

Testing How AI Forgets Multimodal Data

Original: Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation
By Chunlin Liu, Junnian Chen, Haitong Jiang, Jianyu Zhao, Yingsen Pang, Jingchen Li, Jiabiao He, Youming Lu, Jinhe Bi, Yuntao Du Published 2026-08-04

Listen to the summary

Uses a voice available on your device

Audio options
On this page

Key Takeaways

  • Multimodal unlearning, or the process of removing learned associations from a model, transfers more effectively to text-only evaluation than text-only unlearning transfers to image-based evaluation.
  • The UNLINK-VL benchmark provides a standardized, human-verified pipeline for testing knowledge removal across 250 visually identifiable entities.
  • Among tested methods, NPO provides the strongest forgetting, DPO preserves the most useful neighboring knowledge, and RT offers the best overall performance balance.
  • The research highlights a consistent penalty in unlearning effectiveness when moving between different input modalities.

Summary & Methodology Analysis

The researchers developed the UNLINK-VL benchmark using a four-stage pipeline to quantify how effectively models forget specific knowledge. They began by selecting 250 entities from Wikidata and creating four distinct evaluation subsets: Forget, Multi-Hop, Rewrite, and Retain. Each subset includes aligned text-only and text-image counterparts, allowing the team to measure unlearning efficacy through controlled cross-modal experiments involving Gradient Ascent, Direct Preference Optimization (a technique for aligning models using chosen and rejected pairs), Negative Preference Optimization, and Refusal Training. The team tested these methods across architectures like Qwen3-VL and Llama-3.2-Vision to isolate baseline-normalized transfer asymmetry.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the main problem this paper investigates?

It examines whether deleting information in one input format, like text, actually removes it from the model when the user queries it using another format, like images.

Q2. What is UNLINK-VL?

It is a new benchmark for evaluating how well multimodal models can effectively unlearn specific entities across different modalities.

Q3. Why does this matter for software engineers?

As models are increasingly used in production, being able to reliably scrub sensitive or outdated information is a safety and compliance necessity.

Q4. Which unlearning method is best for preserving other knowledge?

Direct Preference Optimization is the most effective at preserving neighboring knowledge that should not be forgotten.

Q5. What is the primary technical limitation of these unlearning methods?

Fine-tuning (the process of updating a pre-trained model on a smaller, specific dataset) may not actually delete the knowledge, as it might still be recoverable via paraphrasing or adversarial queries.

Q6. Does the paper compare its results against other existing benchmarks?

Yes, it references benchmarks including UMU-Bench, PPU-Bench, MME, MMBench, SEED, OCRBench, ScienceQA, MIA-Bench, MathVista, MathVision, POPE, and HallusionBench.

Q7. What performance penalties were observed in cross-modal transfer?

Text-only unlearning transferring to text-image evaluation showed a 6.4 to 19.2 point penalty, while multimodal unlearning transferring to text-only evaluation showed a 15.9 to 27.4 point penalty.

Q8. Does the benchmark require the original training data to function?

No, it focuses on post-hoc unlearning where the original training data is unavailable, which makes it difficult to distinguish true forgetting from weak prior knowledge.

Q9. How were the datasets validated for quality?

The researchers used a human-verified quality control process as the final stage of their four-stage benchmark construction pipeline.