Detecting Vision Language Model Hallucinations

Summary & Methodology Analysis

Vision language models often generate information that is not supported by the input image, a failure known as hallucination. Current detection methods usually focus on only one pattern of failure, but these models use a shifting mixture of patterns depending on the task. To address this, the authors developed a process called source allocation, where they measure how each internal part of the model, called an attention head, balances input from four specific areas: system instructions, images, user text, and the model's own previously generated text. By comparing these measurements to a baseline of faithful behavior, the team identified Role Break, a signal that highlights when a model deviates from its reliable pattern of processing information. They use a mathematical technique called isometric log ratio, or a way to translate complex compositional data into simpler coordinates, to create a standardized residual, which is essentially the measurement of how far a head has strayed from its expected role. This residual is then fed into a simple logistic regression probe, a statistical tool that calculates the probability of a hallucination. This pipeline works for both direct answer tasks and generative tasks where the model produces longer text. In simple yes or no scenarios, the researchers can even automatically flip the model response if the probe detects a high likelihood of hallucination. The paper acknowledges that they have not performed controlled interventions, so it remains unclear whether these Role Break patterns are the direct cause of hallucinations or merely a downstream reaction by the model. Furthermore, while the current method effectively corrects binary answers, it does not yet provide a solution for generating longer, free form text, leaving this as a goal for future research.