Finding Hidden Failures in LLM Training

Summary & Methodology Analysis

During synchronous pre-training, large language models are trained across many parallel computing units. When a component experiences a hang, straggler, or silent data corruption, the failure is often obscured because the synchronous nature of the job forces the entire system to stall or output corrupted data. SCOUT addresses this by establishing the Consensus Collective Communication abstraction. This allows the system to compare diagnostic payloads via a strict-majority consensus mechanism among ranks that are performing identical computations, such as those handling the same tensor or pipeline shards. To keep monitoring lightweight, it employs trainer-independent CPU observers that monitor collective fingerprints through shared memory without adding significant overhead to the primary training loop.

The core of the recovery process involves in-situ replay, where the system re-runs training layers, kernels, and communication patterns directly on the live accelerators. To manage this effectively for models using mixture of experts (a technique that routes inputs to different sub-networks based on the data), it uses a coverage-based replay scheduler. This ensures the most representative input shapes are tested while minimizing impact on training performance. Once the evidence is collected, the system uses a certification process to verify model checkpoints, ensuring they are only promoted to a verified state after passing consecutive check cycles.

Despite its effectiveness, SCOUT has specific operational limitations. It does not perform root-cause analysis for low-level infrastructure failures, such as specific network cables, ports, or internal switch hardware. Furthermore, its replay mechanism is designed to capture recurring faults; it cannot localize one-shot transient faults that fail to reappear during the replay phase. The system is designed to work with standard industry frameworks like PyTorch, Megatron-Core, and DeepSpeed, but its diagnostic capability is strictly tied to the evidence it can reproduce during its controlled replay cycles.