Back to Feed
Benchmarks & Evals / Multimodal

Evaluating Image Captions with Dense Question Answering

Original: CapProbe: Evaluating Detailed Image Captions via Full-Scene Dense Question Answering
By Mouxiao Huang, Qiangyu Yan, Borui Jiang, Han Shu Published 2026-08-11

Listen to the summary

Uses a voice available on your device

Audio options
On this page 4 sections
Related concepts 2 concepts

Key Takeaways

  • CapProbe improves factual evaluation by using a hierarchical structure of 346 images, 1,868 regions, and 25,650 QA pairs.
  • Gemini-3.1-Pro emerged as the top performer among 13 models with an overall accuracy of 68.72 percent and a coverage of 73.68 percent.
  • The evaluation pipeline uses automated deduplication based on semantic embeddings, reducing raw QA sets by 23.2 percent.
  • The framework moves beyond standard n-gram metrics by providing a fine-grained, region-based diagnostic tool for captioning models.

Summary & Methodology Analysis

CapProbe approaches caption evaluation by decomposing images into semantic regions rather than relying on global text-to-text comparison. The pipeline begins by selecting images across 37 L1 domains and 219 L2 sub-domains from LVIS, Places365, and OpenImagesV7. Foreground objects are isolated using YOLOv26-seg, while SAM3 handles background elements. For every identified region, Gemini-3.1-Pro generates structured metadata and multiple-choice questions spanning 10 categories. The framework maintains quality through an automated deduplication step using Qwen3-Embedding-8B to filter redundancy, followed by manual human review to verify clarity and accuracy.

Interactive System Flowchart

Click diagram to expand and zoom

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary problem with existing captioning metrics?

Current metrics like n-gram overlap and standard LLM-as-a-scorer approaches often fail to detect specific factual hallucinations or omissions in detailed image captions.

Q2. How does CapProbe evaluate a caption?

It uses a three-stage protocol that involves generating questions about specific image regions, having a language judge answer these questions based on the caption, and computing final metrics for competency and efficiency.

Q3. Does this benchmark require a massive compute budget?

The paper does not specify the total compute budget, though it does note the efficiency of the automated pipeline in reducing raw QA volume by 23.2 percent.

Q4. What role does YOLOv26-seg and SAM3 play?

They are used for full-scene image decomposition where YOLOv26-seg extracts foreground objects and SAM3 isolates background or stuff elements.

Q5. How were the QA pairs validated?

The initial set was processed through automated deduplication using Qwen3-Embedding-8B, followed by manual quality assurance from human annotators who edited or deleted pairs to ensure correctness.

Q6. What is the scale of the CapProbe dataset?

It includes 346 images, 1,868 regions, and 25,650 total QA pairs, providing an average of 74 questions per image.

Q7. How does CapProbe address redundancy?

It employs hierarchical deduplication based on semantic embedding similarity, which reduced the initial raw set of 50,902 QA pairs to 39,127.

Q8. Are there any known limitations to the benchmark?

The taxonomy-diverse set of 346 images is limited and does not support stable performance estimates for individual L2 sub-domains.

Q9. Which models performed best in the evaluation?

Among 13 models tested, Gemini-3.1-Pro achieved the highest overall accuracy of 68.72 percent and the highest coverage of 73.68 percent.

Browse the archive →
Multimodal / Benchmarks & Evals Removing Person Knowledge from Multimodal Models The paper introduces a corpus-free framework and benchmark for deleting specific person-related knowledge from multimodal large language models without needing the original training data. Multimodal / Benchmarks & Evals Evaluating Vision Models on Pragmatic Incongruity The paper introduces a benchmark called PragMatch to test if large vision-language models can distinguish between genuine pragmatic sarcasm and simple image-text mismatches. Multimodal / Safety & Alignment Programmable Backdoor Attacks on Vision Models The paper introduces a flexible backdoor paradigm that enables dynamic, post-training control over a Vision Language Model output by injecting trigger patterns into training data.