Efficient CPU Inference for Large Models

Summary & Methodology Analysis

The paper addresses the performance bottleneck of single-token autoregressive decoding on CPUs, where memory bandwidth rather than compute limits throughput. By co-designing the model architecture and the inference runtime, the author implements cflow: an engine that organizes weight storage to match the compute consumption order. This design utilizes a vertical pipeline execution schedule where weights for multiple layers are read and executed together, effectively overlapping I/O operations for expert tiles with the computation of preceding layers in Mixture of Experts (MoE) architectures, which select a small number of active experts from a large pool for each token.

To evaluate the engine, the author targets large-scale models like the Gemma 4 26B-A4B and benchmarks them against existing tools. On a 30.9-billion-parameter pipeline-native MoE, cflow delivers 5.94 tokens per second on a 32-vCPU Ice Lake server. This throughput surpasses llama.cpp, which reaches 4.75 tokens per second, and the vLLM CPU backend, which achieves 1.65 tokens per second on comparably sized dense models. The methodology relies on precise management of expert tiles and asynchronous scheduling to maximize the efficiency of the CPU memory hierarchy.

Despite these gains, the research highlights significant limitations regarding specific optimization claims. For instance, the PREFETCHT0 explicit prefetch instruction and the stage-major disk layout failed to show measurable benefits when storage-to-RAM bandwidth became the primary bottleneck. Furthermore, the author notes that these techniques are not guaranteed to generalize to batched inference scenarios, leaving this as an area for further investigation. The paper provides a clear assessment of where these optimizations provide value and where they fail to produce gains in bandwidth-constrained environments.