Better Model Output Selection via Consilience

Summary & Methodology Analysis

The Consilience framework shifts away from simple confidence maximization, which the paper demonstrates is an unreliable heuristic for complex reasoning. By sampling multiple completions, the method isolates the reasoning phase from final answer tokens using specific delimiters. It then calculates token-level confidence using negative mean log-probability, a metric representing how likely the model deems each token. The framework specifically compares initial confidence, calculated over a window after an initial sequence of tokens, against final confidence, calculated over a window at the end of the generation. By calculating the Consilience Score as the difference between final confidence and weighted initial confidence, the system explicitly prioritizes outputs that demonstrate a specific temporal pattern of growth in certainty. This approach is intended to filter out outputs that remain overly confident from the start, a common symptom of hallucinations in complex reasoning tasks. The paper demonstrates this effectively on models like DeepSeek-R1-Qwen-7B-Distill and various Qwen3 variants, achieving higher accuracy on benchmarks such as GPQA-Diamond and SWE-bench-Verified. While effective, the paper notes that Consilience is a fixed heuristic rather than a universal rule, as task settings can have varying valid confidence trajectories. Additionally, the implementation is computationally expensive for production environments because it requires generating multiple samples for each prompt to determine the optimal completion.