Benchmarking Vision-Language Model Confidence

Summary & Methodology Analysis

The researchers addressed the difficulty of assessing model reliability in intelligent document processing by creating ConfBench. They applied 20 distinct degradation pipelines, leveraging tools like Augraphy, to 75 verified FCC-format invoices from the RealKIE-FCC-Verified dataset. This generated 1,346 variations of these documents, creating a controlled environment that spans the full accuracy spectrum to test how models respond to low-quality or difficult-to-parse inputs. The evaluation framework includes standard metrics like extraction accuracy, AUROC, ECE, Brier Score, and a custom error-capture metric called ECARB. By comparing four proprietary and three open-weight models, the researchers were able to quantify how well models can signal when they are likely to fail. The team also performed per-model post-hoc corrections to absolute confidence values, which is a technique used to calibrate predicted probabilities after the model has completed its primary inference. For open-weight models, they specifically compared different log-probability estimation methods, including first-token, mean-token, and margin aggregation. A key finding is that log-probability using first-token aggregation is the most effective approach for measuring confidence. Furthermore, the paper demonstrates that input modality matters, as systems utilizing both OCR and image data significantly outperform image-only configurations, particularly in smaller models. However, the study is limited by its scope, focusing exclusively on a narrow domain of FCC-format invoices, a single layout family, and one language. Because of this, the authors warn that these findings should not be treated as universal claims about general document understanding capabilities. The paper does not provide specific data regarding latency, GPU memory usage, or dollar costs associated with running these models.