MoE (Mixture of Experts)

Mixture of Experts (MoE) — the model has many 'expert' components inside it, but for each question, only a few of them get activated. The overall model is huge, but only a portion of it does the work at any given time, which is the trick behind getting big-model performance at a fraction of the compute cost.

Think of a hospital with 100 doctors, but each patient only sees the two or three specialists relevant to their case. DeepSeek and Mistral became well known for building 'cheap but powerful' models this way, and many of today's leading large models use this approach. When a release notes something like '600 billion total parameters, 37 billion active parameters,' that's a sign it's using MoE.