Verifying Integrity of Vendor Hosted LLMs
Listen to the summary
Uses a voice available on your device
Audio options
On this page 4 sections
Related concepts 2 concepts
Key Takeaways
- Ventor-QTest detects behavioral deviations in black-box LLM APIs without requiring probability access.
- The framework uses two components: a repeated-request test for output distribution analysis and a long-sequence test for extreme deviation tracking.
- Performance metrics show a strong correlation of 0.971 with standard logprob-based analytical methods.
- The audit identified significant degradation in task performance, such as a drop from 82.6 percent to 13.6 percent pass rate on Terminal-Bench in specific routing configurations.
Summary & Methodology Analysis
Ventor-QTest treats hosted model routing as a stochastic process where requests are sampled across different backend configurations. The framework employs a two-part audit strategy. The repeated-request component sends frozen contexts multiple times to build a categorical output distribution. It then calculates the Average Fidelity Loss (AFL), a statistic that corrects for null-bias to measure within-window coarsened-KL, which is a method for comparing probability distributions between models. The second part, the long-sequence component, executes independent runs to calculate a reference-centered-surprisal statistic. This generates an Extreme Fidelity Loss (EFL) metric based on the upper tail of these performance deviations.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the main purpose of Ventor-QTest?
It is an audit framework used to identify when a vendor-hosted LLM behaves differently than a trusted reference model.
Q2. Does this tool require access to the LLM internal probabilities?
No, it functions as a black-box tester and does not require access to target-side probability data.
Q3. Can this test tell me exactly why a model's behavior changed?
No, the test cannot identify specific causes for deviations, such as quantization (the process of reducing the precision of model weights to save memory) or intentional vendor tampering.
Q4. What metrics does Ventor-QTest produce?
It produces Average Fidelity Loss (AFL) for repeated request patterns and Extreme Fidelity Loss (EFL) for sequence-level variations.
Q5. How reliable is the AFL metric?
AFL demonstrated strong descriptive agreement with logprob-derived coarsened-KL benchmarks, reaching a Pearson correlation of 0.971.
Q6. What datasets were utilized for testing?
The study utilized GPQA-Diamond and Terminal-Bench.
Q7. Does this audit predict if an LLM will succeed on downstream tasks?
The paper does not establish a causal or monotonic relationship between the AFL or EFL metrics and success on downstream tasks.
Q8. Are there specific models tested in this study?
The study tested DeepSeek-V4-Flash, DeepSeek-V4-Flash-0731, Kimi K2, Codex, and Claude Code.
Q9. Is the audit comprehensive for all model deviations?
No, the audit is restricted to the specific probe distributions used and cannot detect deviations outside of those sets.