Kimi K3 just became the largest open-weight model in history at 2.8 trillion parameters, yet it only uses 1.8% of itself at a time. The idea behind that trick is thirty-five years old, took three attempts to work, and, according to a recent theory paper, may have been hiding inside dense models all along.
On July 16, Moonshot AI released Kimi K3, and the headline number did what headline numbers do: 2.8 trillion parameters, the largest open-weight AI system ever announced, big enough that Elon Musk commented “Impressive” and xAI announced a rival 2-trillion-parameter training run the following day. But the number that actually explains the model is much smaller. K3 contains 896 expert modules, of which exactly 16 activate for any given token, roughly 1.8 percent of the network firing at once. You get the knowledge of a 2.8-trillion-parameter model at a fraction of the per-token compute, which is why Moonshot claims 2.5 times better computational efficiency than its previous generation, and why a model this size is commercially servable at all.
The architecture making that arithmetic possible is called Mixture of Experts, or MoE, and Kimi K3 is only its most recent and most extreme expression. DeepSeek-V3, whose training cost disclosures shook the industry in late 2024, is an MoE model. Mixtral, the open-weight model that put European AI on the map, is one. The MiniMax M3 model covered on this blog a few weeks ago pairs its sparse attention mechanism with the same philosophy. Across the industry, the models making headlines increasingly share a common architectural core: for every token processed, only a small, dynamically chosen subset of the model’s parameters does any work.
This post covers what MoE is, why an idea from 1991 needed three decades to matter, what researchers discovered when they finally looked inside working MoE models, and a recent theoretical result that reframes the whole story: dense models, the supposedly “normal” ones, appear to be doing mixture-of-experts computation secretly anyway. Along the way it will also complicate the most natural reading of Kimi K3’s announcement. Coverage of the launch describes 896 “specialized” experts, and the word does real marketing work: it suggests a committee of savants, a math department and a coding department and a poetry department inside the weights. What the research actually shows about expert specialization is stranger than that, and more interesting.
One honest caveat before diving in: Kimi K3’s open weights are promised for July 27, a few days after this article’s publication, and its benchmark claims await independent verification. The numbers cited here are Moonshot’s own. The architectural facts, which are what this article needs from it, are the parts least likely to move.
A Thirty-Year History in Three Acts
The intellectual origin is precise and surprisingly old. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton published “Adaptive Mixtures of Local Experts” in Neural Computation, proposing a system in which multiple small networks specialize in different regions of the input space, with a gating network learning to dispatch each input to the right specialist. The idea was elegant and, for the hardware and datasets of the era, largely a curiosity. It joined the long list of neural network concepts waiting for their moment.
The second act came in 2017, and it is no coincidence that Hinton’s name appears again alongside a team at Google Brain. “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer” by Shazeer et al. demonstrated that the 1991 idea could be made to work inside modern deep networks at then-unprecedented scale, up to 137 billion parameters, by activating only a few experts per input and training the gating network jointly with everything else. The paper introduced most of the machinery still in use today: top-k selection, noisy gating, and auxiliary losses to keep the workload balanced across experts.
Transformers then absorbed the idea in two steps. GShard (2020) showed how to shard MoE layers across accelerators and introduced the notion of expert capacity, making 600-billion-parameter translation models trainable in practice. The Switch Transformer (2021) then did something counterintuitive: it simplified. Where prior designs routed each token to two or more experts, Switch routed to exactly one, cut the auxiliary machinery to a minimum, and demonstrated stable training at trillion-parameter scale while keeping per-token compute roughly constant. The design philosophy, simplify to scale, shaped everything that followed.
The third act is the one we are living in. Mixtral 8x7B (2024) proved that an open-weight MoE model could match or beat much larger dense models while activating only about 13 billion of its 47 billion parameters per token. DeepSeekMoE pushed toward many small, fine-grained experts rather than a few large ones, a design choice that fed directly into DeepSeek-V3’s celebrated efficiency. By 2025-2026, the question for a frontier lab was no longer whether to use MoE but how, and Kimi K3’s 896-expert, 16-active configuration marks the current outer edge of that trajectory: an unusually aggressive sparsity ratio that would have been untrainable a few years ago. Like the agentic commerce story this blog covered recently, it is a pattern worth noticing: the idea was never wrong, it was early, and it waited for its infrastructure.
How It Actually Works, Without the Hype
Strip away the terminology and the mechanism is simple. In a standard Transformer, each layer contains a feed-forward network (FFN), a dense block that every token passes through. In an MoE model, that single FFN is replaced by many parallel FFNs, the “experts,” plus a small routing network, the “gate.” For each token, the router scores all experts, the top k are selected, the token is processed by only those, and their outputs are combined, weighted by the router’s scores. Everything else in the Transformer, attention included, typically stays dense.
The economics follow immediately. The model’s capacity, its total store of learned parameters, can grow enormously, while the compute per token grows barely at all, because compute is determined by the activated parameters, not the total. This is the decoupling that all three major surveys of the field, Cai et al., Mu and Lin, and the 2025 comprehensive review, identify as MoE’s foundational advantage: substantially scaling model capacity with minimal computation overhead.
A Worked Example: One Token Through a Top-2 Router
Concrete numbers make the mechanics click. Imagine an MoE layer with 8 experts using top-2 routing, roughly Mixtral’s configuration, processing the token “bank” in the sentence “she sat on the river bank.”
The token arrives at the layer as a hidden vector x, already enriched by attention with the context around it. The router is nothing exotic: a single learned linear layer that takes x and produces 8 raw scores, one per expert. Say those scores, after a softmax, come out as:
Expert 1: 0.02, Expert 2: 0.05, Expert 3: 0.42, Expert 4: 0.03, Expert 5: 0.08, Expert 6: 0.06, Expert 7: 0.31, Expert 8: 0.03
Top-2 routing keeps only the two highest: Expert 3 (0.42) and Expert 7 (0.31). The other six experts are not computed at all for this token. The two retained weights are renormalized to sum to one: 0.42 / (0.42 + 0.31) = 0.58 for Expert 3, and 0.31 / 0.73 = 0.42 for Expert 7. The token’s vector is processed by both expert FFNs independently, and the layer’s output is the weighted blend:
output = 0.58 × Expert3(x) + 0.42 × Expert7(x)
That is the entire trick. For this token, the layer performed 2/8 = 25 percent of the FFN computation a dense equivalent would have needed, while the model as a whole retains the knowledge stored in all 8 experts, because the next token, or the same word in a different context, may route entirely differently. Scale the same arithmetic up and you get Mixtral: 47 billion parameters of capacity, roughly 13 billion active per token, and inference costs closer to a 13B model than a 47B one.
Two complications keep this from being a free lunch.
First, nothing forces the router to spread work evenly; left alone, it tends to collapse onto a few favorite experts, starving the rest of training signal. Production systems add an auxiliary load-balancing loss that penalizes uneven routing, a fix that dates back to Shazeer’s 2017 paper and that every survey lists among the field’s persistent headaches, since the balancing objective mildly fights the quality objective. The problem gets harder as sparsity gets more extreme, which is why Kimi K3, activating just 16 of 896 experts, needed a new mechanism the Moonshot team calls Quantile Balancing, deriving expert allocation directly from router-score quantiles rather than from a tuned auxiliary loss; at that sparsity, the paper-era heuristics stop being good enough.
Second, one clarification the surveys are at pains to make: the “experts” are not domain specialists. There is no math expert, no French expert, no biology expert, whatever the launch coverage of “896 specialized experts” may conjure. Specialization, to the extent it exists, is far stranger, which is what researchers found when they opened the box.
What Researchers Found When They Opened the Box
For years, MoE models were deployed faster than they were understood. The empirical paper in this article’s foundation, “A Closer Look into Mixture-of-Experts in Large Language Models” by Lo et al., studied three popular open MoE models to characterize what routers and experts actually do, and its findings systematically undercut the folk picture of tidy specialists.
Three observations stand out. First, neurons act like fine-grained experts: the meaningful unit of specialization appears to sit below the expert level, at groups of neurons within experts, suggesting the expert boundary is a coarse container around finer structure. Second, the router usually selects experts whose outputs have larger norms: a partly geometric criterion rather than an obviously semantic one. Third, expert diversity increases with depth, experts becoming more differentiated in later layers, except in the last layer, a consistent outlier the authors flag and probe with a targeted experiment.
Subsequent work has sharpened the geometric picture. A 2026 study titled “Routers Learn the Geometry of Their Experts” documents what it calls geometric coupling in sparse MoE: routing weights align with the directions in activation space that experts are best conditioned to process. Routing, on this evidence, is less “send the math question to the math department” and more “send this vector to the transformation whose geometry fits it.” The metaphor of experts survives in the architecture’s name; inside the weights, something more alien and more interesting is going on.
These findings have practical teeth. Lo et al. derive concrete suggestions for router design and expert allocation from their observations, and the norm-based selection finding raises an uncomfortable question the field has not fully answered: if routers partly select on output magnitude, are they choosing the best expert or the loudest one?
The Twist: Your Dense Model Is Secretly an MoE
The most conceptually surprising entry in this article’s source list is the one published in December 2025. Enric Boix-Adserà published “Secret Mixtures of Experts Inside Your LLM”, which advances a hypothesis that reframes the entire MoE story: the dense MLP layers of ordinary, non-MoE language models secretly perform an approximately sparse computation, and can be well approximated by sparsely-activating MoE layers.
The theoretical route to this claim runs through an unexpected neighbor: Sparse Autoencoders, the tool the interpretability community uses to decompose model activations into sparse, sometimes human-legible features. Boix-Adserà establishes a formal connection between MoE structure and SAE structure in activation space, then validates the hypothesis empirically on pretrained LLMs. The detail that gives the result its force is the negative control: the approximation does not hold for Gaussian data. It works only because real neural network activations have structure, the same kind of sparse, feature-like organization SAEs exploit. Dense MLPs, in other words, are not uniformly dense in what they do; the density is in the wiring, not the computation.
If this holds up, it changes what MoE is. The standard telling, the one in the first half of this article, presents MoE as a clever efficiency trick bolted onto Transformers: activate less, save compute. The secret-MoE result suggests something deeper: sparse, expert-like computation may be what these networks naturally converge to, and explicit MoE architectures simply make the implicit structure explicit, harvesting an efficiency the dense model was leaving on the table. It would help explain a fact the field has mostly accepted without explanation: why MoE models match dense models of similar total capacity so consistently, across labs, scales, and modalities. They are not doing something different. They are doing the same thing, honestly.
The paper also earns its keep practically, suggesting low-rank routers as a direction for more efficient MoE designs, derived from the structure of the approximation. Theory papers that end with an engineering suggestion tend to be the ones that get read.
The Trade-offs Nobody Puts in the Launch Post
An honest account of MoE has to include the reasons it took this long to become the default, because the difficulties are real and several remain unsolved. The surveys converge on a consistent list.
Memory is the tax that sparsity does not pay. Every parameter of every expert must be resident in accelerator memory, even though few are active per token. Mixtral computes like a 13B model but must be stored and served like a 47B one. MoE saves FLOPs, not VRAM, which is why the architecture shines for high-throughput serving, where the memory cost amortizes across many requests, and disappoints hobbyists trying to run it on a single GPU.
Training is touchier than for dense models. Expert collapse, routing instability early in training, and the standing tension between the load-balancing loss and the actual objective all require care; the fact that Switch Transformer’s headline contribution was stability tells you what the binding constraint was. Fine-tuning MoE models is similarly delicate, with routing patterns learned in pretraining liable to degrade under narrow downstream data. And the 2025 review adds a deployment-side list: ensuring expert diversity, accurate calibration, and reliable inference aggregation, the unglamorous conditions for the architecture’s theoretical advantages to survive contact with production.
There is also a systems bill. Distributing experts across devices turns routing into a communication problem, all-to-all exchanges that can dominate step time if mismanaged, and an entire engineering literature, from GShard’s sharding schemes onward, exists to pay it down.
Finally, the efficiency story connects to a theme from this . Per-token efficiency gains of the kind MoE delivers are a major reason the IEA describes AI’s energy efficiency as improving at a rate “unprecedented in energy history”. They are also a textbook Jevons-paradox accelerant: cheaper tokens invite vastly more tokens, and total consumption rises even as unit costs fall. MoE is simultaneously one of the best things to happen to AI’s energy intensity and one of the reasons demand keeps outrunning it.
Where This Is Heading
Three directions seem most consequential. The first is granularity: the trend from a few large experts toward many small ones, championed by DeepSeekMoE and echoed in Lo et al.’s finding that neurons already behave like fine-grained experts, suggests the field is converging on finer decompositions from both the engineering and the interpretability side. The second is routing itself: low-rank routers from the secret-MoE work, geometry-aware designs informed by the coupling results, and routing beyond FFNs into attention and multimodal fusion, directions all three surveys flag as active. The third is the interpretability question hiding under everything: if routing is geometric rather than semantic, what would it take to build MoE models whose experts are genuinely modular, separable, and auditable? The gap between the architecture’s name and its behavior is, for now, a research agenda.
The historical symmetry is worth savoring. An idea published in 1991 waited a quarter century for its hardware, another few years for its infrastructure, and now sits inside most of the models defining the field, so thoroughly absorbed that researchers are finding its fingerprints even in the dense models that never adopted it. The experts were always in the mixture. It just took thirty-five years to route the question to them.
Key sources:
Cai, W. et al. “A Survey on Mixture of Experts in Large Language Models.” arXiv:2407.06204.
“Mixture-of-Experts in Large Language Models: A Comprehensive Review.” arXiv:2507.11181, 2025.
Lo, K. M. et al. “A Closer Look into Mixture-of-Experts in Large Language Models.” arXiv:2406.18219.
Boix-Adserà, E. “Secret Mixtures of Experts Inside Your LLM.” arXiv:2512.18452, December 2025.
Jiang, A. Q. et al. “Mixtral of Experts.” arXiv:2401.04088, 2024.
IEA. “Key Questions on Energy and AI.” April 2026, on the pace of AI efficiency improvements.
Kimi K3 Guide. Specs, architecture, and access details for Moonshot AI’s Kimi K3.