MoE (Mixture of Experts)
MoE (Mixture of Experts) is an architecture that contains multiple "expert" subnetworks within a model, activating only a subset of them for each input, thereby increasing the total number of parameters while keeping inference costs low.
Fast Because It Doesn't Use Everything
Why can massive models like GPT-4 and Llama 4 perform inference at relatively practical speeds? One answer is the MoE architecture.
In a standard Transformer model (Dense model), every input token passes through all parameters. For a 100B-parameter model, 100B weights are involved in every calculation. With MoE, even if the total model size is 2 trillion parameters, only around 170B are actually used in a single inference pass — the remaining experts are skipped as "not needed this time."
The component that decides which experts to use is the "gating network" (router). It examines the features of the input token and selects the 2–4 most appropriate experts. A useful mental model: logic-oriented experts are selected for math problems, while language-oriented experts are chosen for translation tasks.
Models Currently in Use
Meta's Llama 4 adopts this architecture in Scout (17B active / 109B total) and Maverick (17B active / 400B total). Google's Gemini series is also reported to be MoE-based. Mistral's Mixtral 8x7B bundles eight 7B-parameter experts together, using only two of them during inference.
What these models share is that the active parameters during inference are dramatically fewer than the total parameter count. This allows them to maintain the model's knowledge capacity while keeping inference speed and cost within practical bounds.
When to Use Dense Models vs. MoE
Dense models, which use all parameters, are simple and easy to work with at small to medium scales. Fine-tuning is also straightforward. MoE is an architecture that truly shines at large scale, and for models below several tens of billions of parameters, the overhead may not be worth it.
Additionally, fine-tuning MoE models requires care to avoid unintended effects across all experts, and combining them with PEFT methods such as LoRA demands its own set of know-how.
Articles covering this term
- Latency Budget Design for AI Agents — How to Control the Trade-off Between Thinking Time and Response TimeExplore how "thinking time" causes response delays in multi-step reasoning agents, and compare latency budget allocation strategies and implementation patterns based on task complexity.
- What Are Liquid Neural Networks? Next-Generation AI Inference with Dynamic BehaviorLearn how LNNs dynamically adapt time-constants during inference, how they differ from traditional models, and their edge AI potential—all while keeping trained weights fixed.
- Local LLM / SLM Deployment Comparison — AI Utilization Without Cloud API DependencyCompare open-weight models like GPT OSS, Phi-4, and Llama 4 Scout against cloud APIs on GPU requirements, task accuracy, and TCO. A local AI deployment guide for data sovereignty and cost optimization.
- What is Model Merging? A Technique to Combine Multiple LLMs Without Training to Improve PerformanceExplore Model Merging: mathematically combine fine-tuned models to integrate multiple skills while reducing computational costs.
Related Terms
Context Window
A context window refers to the maximum number of tokens an LLM can process at one time, indicating t
Fine-tuning
Fine-tuning refers to the process of providing additional training data to a pre-trained machine lea
Generative AI (Generative AI)
Generative AI is a collective term for AI models capable of autonomously generating content such as
LoRA
LoRA (Low-Rank Adaptation) is a technique that inserts low-rank delta matrices into the weight matri