How Voice and Typing Errors Impact LLM Performance
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- Token survival, which is the preservation of original input tokens after noise, serves as a primary predictor of performance loss with a correlation of 0.79.
- Reasoning models that utilize a thinking budget can effectively recover from keyboard-based input errors.
- Voice transcription errors introduce spoken register perturbations that reasoning models struggle to recover from, even with extended thinking budgets.
- The research introduces HIVE, a suite of 17 operators designed to simulate real world user input variations.
Summary & Methodology Analysis
The researchers developed HIVE, the Human Input Variation Engine, to model input noise across three channels: voice transcription, QWERTY keyboard errors, and copy-paste control data. By applying 17 distinct perturbation operators, the team tested five instruction-tuned models across six benchmarks to determine how input degradation impacts task accuracy. They measured token survival rates to quantify whether the removal of original input tokens or the injection of new, erroneous tokens caused the most significant accuracy drops.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary focus of this research?
This paper investigates how input errors from typing and voice transcription pipelines influence the performance accuracy of LLM agents.
Q2. Did the researchers test real human audio?
No, the paper notes a limitation where spoken registers were simulated using LLM verbalizers rather than actual human audio recordings.
Q3. Can models recover from input errors?
Yes, reasoning models can recover from most keyboard errors using a thinking budget, though they struggle to fix voice transcription perturbations.
Q4. What is HIVE?
HIVE is the Human Input Variation Engine, a suite of 17 operators and two controls used to model realistic human input variations.
Q5. What models were included in the evaluation?
The models evaluated were Llama-3.1-8B, Qwen2.5-7B, Mistral-7B-v0.3, Qwen3-8B, and phi-4.
Q6. What datasets were used to measure performance?
The researchers used GSM8K, GSM-Symbolic, GSM1k, HumanEval, MMLU-Pro, and TruthfulQA.
Q7. How did they measure the impact of input errors?
They performed within-item comparisons to measure accuracy changes and calculated token survival rates to see how much of the original input survived the perturbation.
Q8. What mitigation strategies did they test?
They tested self-distillation with LoRA adapters and assessed the impact of model thinking budgets on error recovery.
Q9. Are there limitations to the keyboard analysis?
Yes, the typing analysis is limited solely to the QWERTY keyboard layout.