Measuring Cross-Lingual Consistency in AI Agents
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 6 concepts
Key Takeaways
- Most frontier models retain 71 to 73 percent of their action policies when switching languages under greedy decoding.
- A model's measured policy gap is highly correlated with its self-consistency, showing a correlation of r = +0.97.
- Models under 10 billion parameters show a significant breakdown in consistent policy retention.
- Agents consistently default to English reasoning even when explicitly instructed to reason in other languages, with a 99 percent refusal rate.
- Trace extraction methods can introduce artifacts, such as a regex-based error that increased measured accuracy twenty-sixfold in specific tests.
Summary & Methodology Analysis
The researchers developed a protocol to evaluate agent behavior by focusing on the executed trace, which consists of the intermediate tool-use actions rather than just the final text output. This approach addresses limitations where simple output metrics mask variations in latency, cost, and auditability. To ensure rigorous measurement, the protocol accounts for five specific confounds: lack of baseline for model self-consistency, trace length differences, empty traces, model reproducibility ceilings, and chance floors. They calculate cross-lingual policy retention by comparing same-language agreement to across-language agreement and normalizing this via self-consistency. Causal reliance on English is further verified by manipulating tool availability and ordering constraints. The study tested models including Gemma-3-27B, Sarvam-M, Qwen3-235B, and Llama-4-Maverick.
Evaluation involved a suite of datasets such as FLORES-200, XQuAD, XNLI, Belebele, and XCOPA. The experimental design requires generating two replicates for every task in every language under identical decoding and budget constraints. This allows for a robust assessment of whether agents exhibit the same decision-making logic regardless of the input language. Greedy decoding is the primary inference method, where the model always selects the highest probability token at each step of generation. The results show that while frontier models maintain 71 to 73 percent retention, this stability is heavily dependent on model scale. Models below approximately 10 billion parameters fail to maintain this level of regularity, indicating that cross-lingual policy stability is a property of larger scale models.
Key limitations include a sharp decline in performance for smaller models and a persistent, near-total refusal to reason in non-English languages. Furthermore, one vendor model, Aya-Expanse-8B, was excluded from the final analysis because it produced high rates of non-parseable traces, effectively failing the adherence requirements of the protocol. The study also warns that methodology matters: a regex used to extract traces caused a massive twenty-sixfold accuracy inflation in GPT-OSS-120B due to the inclusion of worked examples in the output. Developers should be aware that self-consistency is a strong predictor of this policy gap, with an r = +0.97 correlation under greedy decoding, making it a critical metric for evaluating the reliability of agentic workflows.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. Why is the final output of an AI agent insufficient for evaluation?
Looking only at the final output masks intermediate tool-use actions, which hide differences in cost, latency, failure modes, and auditability.
Q2. What is the primary goal of this research?
The researchers aimed to determine if tool-using models execute the same action policy when switching between different languages.
Q3. What happens when you ask an agent to reason in a language other than English?
The agents almost always maintain English reasoning, with a refusal rate of over 99 percent.
Q4. How do you calculate cross-lingual policy retention?
The researchers calculate the ratio of across-language agreement to same-language agreement, which is normalized by dividing the two values.
Q5. What specific confounds does the new measurement protocol address?
The protocol addresses the lack of baseline for self-consistency, trace length differences, empty traces, model reproducibility ceilings, and chance floors.
Q6. What is the impact of model size on policy retention?
The consistent 71 to 73 percent policy retention observed in larger models breaks down for models below approximately 10 billion parameters.
Q7. How does greedy decoding affect the results?
Under greedy decoding, there is a very high correlation of r = +0.97 between a model's self-consistency and its measured policy gap.
Q8. Which models were tested in this study?
The tested models included Gemma-3-27B, Sarvam-M, Qwen3-235B, Llama-4-Maverick, GPT-OSS-120B, Gemma-3-4B, Qwen3-8B, and Aya-Expanse-8B.
Q9. Are there any known artifacts in trace measurement?
Yes, the paper notes that a regex trace-extraction method caused a twenty-sixfold increase in measured accuracy for GPT-OSS-120B.