Improving Accuracy in Medical Question Answering

Summary & Methodology Analysis

ConRub-Med addresses the difficulty of evaluating open-ended medical responses by implementing a multi-step automated rubric generation process. The system generates three atomic criteria using heterogeneous language models and filters these criteria through a review model to ensure they receive semantic support from all three generators. This creates a more reliable automated verifier than relying on single-model evaluations, which often struggle with medical nuance and clinical accuracy. The method uses a Three-State scoring approach to classify output into correct coverage, missing information, or incorrect claims, applying negative credit directly to incorrect information to steer the model towards safer responses. For optimization, the framework utilizes GRPO (Group Relative Policy Optimization, a reinforcement learning method that optimizes a policy by comparing multiple generated outputs within a group to compute a relative advantage). When groups share tied rewards, the system adds a pairwise judge to assess sequence advantages only if both candidates agree on the ranking, ensuring more robust training signal in ambiguous scenarios. For groups without ties, the system defaults to standard GRPO. This approach allows the model to achieve state-of-the-art performance, ranking first on six of nine benchmarks including high scores on the HealthBench-Hard dataset. A primary limitation noted is the high cost of involving physicians for rubric validation in every instance, which prevents scaling manual verification to every potential query.