Adapting LLM Inference to Bursty Workloads
Listen to the summary
Uses a voice available on your device
Audio options
On this page
Key Takeaways
- The original WAIT algorithm performs poorly under realistic, non-stationary traffic because it relies on fixed-rate assumptions.
- The new method uses exponential moving average and Savitzky-Golay filtering to estimate local arrival rates in real time.
- Dynamic batching thresholds are adjusted based on these estimates to maintain higher system throughput.
- The modified approach outperforms Sarathi-Serve, ORCA, and vLLM in low arrival-rate shift scenarios.
- Latency remains a challenge, and the algorithm requires further refinement to handle high-intensity arrival-rate shifts.
Summary & Methodology Analysis
The standard WAIT algorithm for LLM inference relies on fixed batching thresholds, which are calculated using fluid dynamics assuming a constant arrival rate. This design is fundamentally mismatched with the non-stationary, bursty traffic patterns typical in production environments. To address this, the authors implemented a preprocessing pipeline that applies an exponential moving average and Savitzky-Golay filtering to interarrival times. This produces a real-time local estimate of the arrival rate, allowing the system to observe fluctuations rather than assuming a static flow. The architecture modifies the WAIT batching thresholds dynamically, specifically setting the threshold n_j(t) to be inversely proportional to the current local arrival rate estimate. This feedback loop allows the system to adjust its batching behavior to better match the bursty nature of incoming requests.
Interactive System Flowchart
Cross-Examination & FAQs
A deeper dive clarifying mechanics, constraints, and baseline evaluations.
Q1. What is the primary problem this paper addresses?
The paper addresses the failure of existing LLM inference scheduling algorithms, specifically WAIT, to manage bursty and non-stationary traffic patterns.
Q2. What is the main contribution of this work?
The authors created a modified version of the WAIT algorithm that uses real-time local arrival rate estimation to dynamically update batching thresholds.
Q3. How does this approach compare to existing systems?
It achieves higher throughput than Sarathi-Serve, ORCA, and vLLM in low arrival-rate shift scenarios and performs comparably or better than state-of-the-art systems during moderate shifts.
Q4. How are local arrival rates estimated?
The method uses a preprocessing pipeline that applies an exponential moving average and Savitzky-Golay filtering to request interarrival times.
Q5. Does the modification resolve latency issues?
No, the paper notes that latency remains suboptimal across all settings for both the original and modified versions of the WAIT algorithm.
Q6. Can this algorithm handle high arrival-rate shifts?
The paper states that both the original and modified versions of the WAIT algorithm require further improvements to handle high arrival-rate shift scenarios effectively.
Q7. What specific models or datasets were tested?
The paper references a wide range of systems and models including ChatGPT, Claude, Gemini, LLaMA, Mistral, vLLM, FasterTransformer, DeepSpeed-MII, BurstGPT, ELIS, WAIT, Sarathi, Sarathi-Serve, ORCA, GPT-3, GPT-4, Splitwise, DistServe, and Microsoft Vidur.
Q8. How are the batching thresholds updated?
The thresholds are updated by setting the batching parameter inversely proportional to the estimated local arrival rate.
Q9. What mathematical model was the original WAIT algorithm based on?
The original WAIT algorithm utilized fixed batching thresholds calculated via fluid dynamics based on known, fixed arrival rates.