Mixture of experts

What it is

Instead of passing every token through the entire parameter space of a neural network, a mixture of experts model uses a lightweight routing layer to dispatch tokens to a small subset of specialized sub-networks called experts. For instance, a model might contain eight total experts per layer but route each token to only two of them. This allows the total model parameter count to scale up significantly to hundreds of billions, while the active compute cost per token remains roughly equivalent to a much smaller dense model. The router makes these routing decisions dynamically on every single token, meaning different words in the same prompt can trigger entirely different expert combinations.

Why it matters

You should care about mixture of experts because it dictates the economics, speed, and hosting characteristics of the models you select via API. These models typically offer higher accuracy and broader knowledge bases than dense models of a similar runtime cost, making them attractive for cost-sensitive production traffic. However, you cannot evaluate them purely by parameter size since their memory footprint is massive even if their active compute per token is low. Ignoring this distinction can lead to poor hardware sizing decisions if you self-host, or unexpected pricing cliffs if you rely on third-party APIs.

In practice

When using third-party APIs, you generally do not configure the routing mechanism directly, but you will see models designated as mixture of experts in provider documentation and pricing tables. You might observe bimodal latency profiles where certain requests trigger rare expert paths that take slightly longer to fetch from memory. If you self-host these models, your primary operational hurdle is the high VRAM requirement, because every expert must reside in memory simultaneously even though only a few are active per token.

The tradeoff

The main tradeoff is high memory consumption for low compute cost, and people commonly assume these models run as fast as their active parameter count rather than their total parameter footprint.

Where it appears

Research summaries that use Mixture of experts, each linked to its source paper.

• Learning Shared Robot Control Across Embodiments
  DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation

  DyPES-VLA unifies robot control by using shared dynamics priors learned from video to enable action generation across different robot types without manual alignment.

• Dynamic Modality Selection for 3D Scenes
  SmartMage: Dynamic Modality Orchestration for 3D Scene Understanding

  SmartMage optimizes 3D scene understanding by dynamically routing input modalities based on query requirements to improve reasoning performance and efficiency.

• Teaching Nemotron Models Modern Greek
  Teaching Nemotron Greek: Mining a Corpus, Adapting Retrieval, and Grounding Generation for Modern Greek across Specialist Domains

  The paper adapts NVIDIA Nemotron models for Modern Greek retrieval and generation in specialized domains and introduces the HERA benchmark.

• Optimizing Mixture of Experts Diffusion Models
  LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

  Researchers established scaling laws for Mixture-of-Experts diffusion language models and used them to train the high-performing LLaDA MoE v2 model.

• Balancing Vision and Language for Models
  Relax Within, Balance Across: Geometry-Guided Load Balancing for Vision-Language Mixture-of-Experts

  The researchers developed a new load balancing method called ReBA that stabilizes how vision-language models process data by treating image and text inputs separately.

• Improving Neural Network Classification Using Weight Specialization
  Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification

  The researchers developed a hierarchical expert system that organizes neural network weights into specialized groups to improve how computers classify images represented as mathematical functions.

• Smarter Resource Allocation for AI Experts
  Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Experts

  The paper introduces a routing system that determines when adding more specialized AI processing steps will actually improve an answer versus when it is a waste of effort.

• Improving High Definition Image Quality Efficiently
  CoDe-SSM: Context-Detail Decoupled State Space Model for Efficient UHD Image Restoration

  The researchers developed a new model called CoDe-SSM that restores ultra-high definition images by balancing global context with fine structural details using a highly efficient architecture.

• Scaling Neural Networks with Sparse Experts
  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

  The researchers implemented a Sparsely-Gated Mixture of Experts layer to increase neural network capacity to 137 billion parameters without a proportional increase in computational cost.