Solving Logical Reasoning Failures in Large Language Models

Summary & Methodology Analysis

When large language models evaluate answer options containing explicit logical operators such as And, Or, and Neither/Nor, they frequently fail even if they correctly understand the underlying facts. To address this, the paper presents a structured compositional reasoning framework that prevents the model from ever seeing the compound option directly. Instead, the method deterministically parses each option into a triplet and collects all unique atomic answers appearing in the instance. For every unique atomic answer, the system constructs a pair of opposing natural language hypotheses conditioned on the context, representing whether the atomic answer satisfies the context or not. By presenting these opposing hypotheses as choices within a single prompt, the system estimates the model local evidence using normalized log probabilities. These raw preference scores are then calibrated using Platt scaling, isotonic calibration, or relative calibration.

The core inference layer relies on a binary integer linear program, which operates over the inferred atomic statuses and compound option validity variables. The algorithm encodes operator constraints and a strict uniqueness constraint requiring the sum of valid options to equal one using linear inequalities. It then selects the assignment that satisfies these constraints with the strongest total evidence by maximizing an objective function combining supporting and opposing evidence. By shifting the complexity from a single holistic generation step to a constrained optimization problem over calibrated atomic local evidence, the system prevents logical contradiction and dramatically improves accuracy on complex compound structures.

The evaluation and application of this framework come with explicit boundaries specified in the paper. The experiments are limited to a single model, Llama-3.1-8B-Instruct, which is a language model configured to follow instructions, and two benchmarks with explicit binary operators over pairs of atomic answers. Consequently, the results do not establish that the same gains will hold for other model families, larger models, longer logical expressions, or operators such as implication and exclusive disjunction. Furthermore, the benchmarks enforce exactly one valid compound option, whereas real tasks may permit multiple valid answers or no valid answer at all. Finally, because the framework depends entirely on the quality of the atomic evidence supplied to the inference layer, any errors in commonsense interpretation, passage grounding, or source annotations will propagate directly to the final prediction.