Mixture of experts

What it is

Instead of passing every token through the entire parameter space of a neural network, a mixture of experts model uses a lightweight routing layer to dispatch tokens to a small subset of specialized sub-networks called experts. For instance, a model might contain eight total experts per layer but route each token to only two of them. This allows the total model parameter count to scale up significantly to hundreds of billions, while the active compute cost per token remains roughly equivalent to a much smaller dense model. The router makes these routing decisions dynamically on every single token, meaning different words in the same prompt can trigger entirely different expert combinations.

Why it matters

You should care about mixture of experts because it dictates the economics, speed, and hosting characteristics of the models you select via API. These models typically offer higher accuracy and broader knowledge bases than dense models of a similar runtime cost, making them attractive for cost-sensitive production traffic. However, you cannot evaluate them purely by parameter size since their memory footprint is massive even if their active compute per token is low. Ignoring this distinction can lead to poor hardware sizing decisions if you self-host, or unexpected pricing cliffs if you rely on third-party APIs.

In practice

When using third-party APIs, you generally do not configure the routing mechanism directly, but you will see models designated as mixture of experts in provider documentation and pricing tables. You might observe bimodal latency profiles where certain requests trigger rare expert paths that take slightly longer to fetch from memory. If you self-host these models, your primary operational hurdle is the high VRAM requirement, because every expert must reside in memory simultaneously even though only a few are active per token.

The tradeoff

The main tradeoff is high memory consumption for low compute cost, and people commonly assume these models run as fast as their active parameter count rather than their total parameter footprint.

Where it appears

Research summaries that use Mixture of experts, each linked to its source paper.

• Efficient Mixture of Experts Text Embeddings
  Giga-Embeddings: Mixture-of-Experts Encoders for High-Throughput Text Embeddings

  The authors introduce a family of sparse embedding models that leverage Mixture-of-Experts architectures and distillation to achieve high throughput and competitive retrieval performance.

• Efficient CPU Inference for Large Models
  Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode

  The paper introduces cflow, a model architecture and runtime co-design that optimizes autoregressive decoding on CPUs by prioritizing memory bandwidth constraints.

• Recovering Compressed 4 Bit LLMs
  Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs

  The paper introduces Quantization-Aware Healing, a practical recipe for recovering compressed 4-bit large language models, and uses it to produce the open-weight model Hypernova-60B.

• Improving MoE Model Steering Control
  RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

  The paper introduces RARE, a method to steer Mixture of Experts models by decoupling control interventions from the model router mechanism to maintain performance and reliability.

• Efficient Hyperparameter Optimization for Large Models
  Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

  The paper introduces a method to extrapolate optimal learning rates for Mixture of Experts models using small-scale proxy runs to avoid expensive full-scale sweeps.

• Running Large AI Models on Consumer Hardware
  FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

  FreeToken enables efficient serving of frontier-scale Mixture of Experts models on consumer hardware by adapting execution strategies to available memory and bandwidth.

• DeaMoE: Reducing Memory Load for MoE
  DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding

  DeaMoE is a decoding efficient architecture for Mixture of Experts models that reduces the amount of weight data loaded per inference step.

• Explaining Neural Network Learning Dynamics
  Neural Quadratic Forms: A Unified Minimal Model for Sudden Learning and Scaling Laws

  The authors derive a mathematical framework that models how neural network training converges using a small set of variables regardless of the model size.

• Finding Hidden Failures in LLM Training
  SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

  SCOUT is a system that identifies and localizes latent hardware or communication failures during distributed large language model pre-training by comparing the behavior of identical parallel processing units.

• Distributed Training Without Sharing Private Data
  DistMoE: Private-data Rehearsal-free Routing in Mixture-of-Experts for Distributed Instruction Tuning

  DistMoE enables visual instruction tuning across distributed systems using a mixture of experts approach that eliminates the need to rehearse or share private datasets.

• Scaling Intelligence with Mixture of Experts
  Motif 3: Technical Report

  Motif 3 is a 314 billion parameter model that utilizes a specialized architecture and multi-stage post-training to improve intelligence and task generalization.

• Building Open Models for Continual Learning
  Macaron-V1: Towards Open Continual Learning with Self-Improvement and Mixture-of-LoRA

  Macaron-V1 introduces a framework for deploying persistent agent models that update themselves through specialized adapters and recursive self-improvement loops.

• Scaling Neural Networks with Mixture of Experts
  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

  The paper introduces the Sparsely-Gated Mixture-of-Experts Layer, which dramatically increases network capacity and performance while maintaining computational efficiency.

• Optimizing MoE Inference on Edge Hardware
  OrderMoE: An expert similarity driven distributed edge MoE inference

  OrderMoE reduces cross-server communication overhead for Mixture of Experts models by grouping experts based on similarity and intelligently choosing between local and remote execution.

• Agent-Driven Framework for Simulating LLM Serving
  Simthesizer: An Agent-Driven Simulation Framework for LLM Serving Systems

  Simthesizer utilizes a coding agent to automatically extend simulators for complex LLM serving systems, achieving higher throughput accuracy than existing approaches.

• Improving MoE Model Steering Control
  RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models

  The paper introduces RARE, a method to steer Mixture of Experts models by decoupling control interventions from the model router mechanism to maintain performance and reliability.

• Running Large AI Models on Consumer Hardware
  FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

  FreeToken enables efficient serving of frontier-scale Mixture of Experts models on consumer hardware by adapting execution strategies to available memory and bandwidth.

• DeaMoE: Reducing Memory Load for MoE
  DeaMoE: Efficient MoE Structure for Fast Small-Batch Decoding

  DeaMoE is a decoding efficient architecture for Mixture of Experts models that reduces the amount of weight data loaded per inference step.

• Optimizing MoE Inference on Edge Hardware
  OrderMoE: An expert similarity driven distributed edge MoE inference

  OrderMoE reduces cross-server communication overhead for Mixture of Experts models by grouping experts based on similarity and intelligently choosing between local and remote execution.

• Efficient Memory Management for MoE LLMs
  PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

  PagedWeight manages GPU memory for Mixture-of-Experts models by dynamically quantizing weights at runtime to balance model precision against KV cache requirements.

• Precise Action Recognition Using Expert Models
  Fine-Grained Action Recognition with Cross-Attentive Latent Sparse Experts

  The FineX framework improves fine-grained human action recognition by fusing distinct visual and pose signals through a mixture-of-experts architecture.

• Learning Shared Robot Control Across Embodiments
  DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation

  DyPES-VLA unifies robot control by using shared dynamics priors learned from video to enable action generation across different robot types without manual alignment.

• Efficient Recurrent Transformer Architecture and Scaling
  Loop the Loopies!

  The paper introduces a novel looped Transformer architecture called Loopie that maximizes pre-training compute efficiency to achieve strong reasoning benchmark performance.

• Efficient Memory Management for MoE LLMs
  PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization

  PagedWeight manages GPU memory for Mixture-of-Experts models by dynamically quantizing weights at runtime to balance model precision against KV cache requirements.