{"id":115,"date":"2026-10-14T14:22:59","date_gmt":"2026-10-14T14:22:59","guid":{"rendered":"https:\/\/evoxia.online\/?p=115"},"modified":"2026-10-14T14:22:59","modified_gmt":"2026-10-14T14:22:59","slug":"mixture-of-experts-large-models-that-only-use-part-of-themselves-for-each-token","status":"publish","type":"post","link":"https:\/\/evoxia.online\/?p=115","title":{"rendered":"Mixture of Experts: Large Models That Only Use Part of Themselves for Each Token"},"content":{"rendered":"\n

Paying for Every Parameter<\/strong><\/p>\n\n\n\n

In a standard dense neural network, every parameter takes part in every prediction. If a language model has 70 billion weights, all 70 billion are multiplied against each token it reads, whether the token is a word in a legal contract, a line of Python or a comma. Capacity and cost are tied together: a model that knows more also costs more to run, in a nearly fixed proportion.<\/p>\n\n\n\n

A mixture of experts (MoE) model breaks that link. Instead of one large feed-forward block, a layer holds many smaller blocks, the experts, plus a small network called a router that decides which of them should process the current input. Only the chosen experts run. The model can store a very large number of parameters while spending, per token, the compute of a much smaller one. Many of the largest open-weight language models released since 2024 are built this way, and the reasons it works (and the ways it misbehaves) say a lot about how these systems are trained.<\/p>\n\n\n\n

Local Experts in 1991<\/strong><\/p>\n\n\n\n

The idea is older than deep learning. In 1991, Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton published \u00abAdaptive Mixtures of Local Experts\u00bb in Neural Computation<\/em>. Their system trained several small networks together with a gating network that assigned each input to the expert most likely to handle it well. Because the experts competed for training cases, each one drifted toward a different region of the input space.<\/p>\n\n\n\n

They tested it on vowel discrimination, and the paper reports that the procedure split the task into subtasks, each simple enough for a very small expert network to solve. A companion paper by Nowlan and Hinton found that the modular version generalized better than a single backpropagation network across many variations of the problem. The motivation then was specialization and cleaner learning, not saving compute. That second motivation took another 26 years to arrive.<\/p>\n\n\n\n

Noisy Top-k Gating<\/strong><\/p>\n\n\n\n

In 2017, Noam Shazeer and colleagues at Google published \u00abOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer\u00bb at ICLR. They placed a layer with up to thousands of feed-forward experts between stacked LSTM layers and let a trainable gate pick a sparse set of them for each example. The largest configuration had 137 billion parameters, an enormous number for 2017, and the authors reported more than a thousandfold increase in model capacity with only minor losses in computational efficiency on GPU clusters.<\/p>\n\n\n\n

The gate works like this. It computes a score for every expert, keeps the top k scores, sets the rest to minus infinity and applies a softmax, so only k experts receive nonzero weight. The layer’s output is the weighted sum of those k experts’ outputs. Shazeer’s team added tunable Gaussian noise to the scores before taking the top k. The noise gives experts that are slightly behind a chance to be selected and receive gradient, which matters because an expert that is never chosen never learns.<\/p>\n\n\n\n

The Load Balancing Problem<\/strong><\/p>\n\n\n\n

That last point is the central difficulty of the whole approach. Routing is learned, and learning is greedy. Early in training, a few experts happen to perform slightly better, so the router sends them more tokens, so they improve faster, so the router sends them even more. Left alone, the router collapses onto a handful of experts and the rest of the parameters sit idle. The model then has the memory cost of a huge network and the quality of a small one.<\/p>\n\n\n\n

The usual fix is an auxiliary loss added to the training objective that penalizes uneven routing. Shazeer’s 2017 paper used a term based on how much total gate weight each expert received across a batch. Later work simplified it to a product of two quantities per expert: the fraction of tokens actually sent to it and the average probability the router assigned to it. The loss is smallest when both are uniform. The coefficient has to be small (values around 0.01 are common), because a strong balancing term forces the router to spread tokens evenly even when an uneven split would produce better predictions.<\/p>\n\n\n\n

Switch Transformer and One Expert per Token<\/strong><\/p>\n\n\n\n

Google’s next step, the Switch Transformer by William Fedus, Barret Zoph and Noam Shazeer (2021, later in JMLR<\/em>), took sparsity further. Earlier designs such as GShard routed each token to two experts. Switch sent each token to exactly one, reasoning that this cuts both computation and the communication between devices. The authors reported large gains in pre-training speed over dense T5 models at the same compute, including about a fourfold speedup over a tuned T5-XXL, and trained models up to roughly 1.6 trillion parameters.<\/p>\n\n\n\n

Two practical details from that paper show up in almost every MoE system since. The first is the capacity factor. Hardware wants fixed tensor shapes, so each expert gets a buffer sized to the average number of tokens it would receive under perfect balance, multiplied by a factor such as 1.25. Tokens that arrive after an expert’s buffer is full are dropped: they skip the expert and pass through the layer’s residual connection unchanged. The second is selective precision. Most of the model trained in bfloat16, but the router ran in float32, because the exponentials in its softmax were a frequent source of training instability. A follow-up, ST-MoE (Zoph and colleagues, 2022), added a \u00abrouter z-loss\u00bb that keeps the router’s raw scores small for the same reason.<\/p>\n\n\n\n

Case: Mixtral 8x7B<\/strong><\/p>\n\n\n\n

The model that made MoE familiar outside large labs was Mixtral 8x7B, released by Mistral AI in December 2023 under the Apache 2.0 license. Each of its 32 layers has eight feed-forward experts, and the router sends each token to two of them. According to Mistral, the model has 46.7 billion parameters in total but uses 12.9 billion per token. The name can mislead: it is not eight 7B models glued together, since the attention layers and embeddings are shared and only the feed-forward blocks are duplicated.<\/p>\n\n\n\n

Mistral reported that Mixtral matched or outperformed Llama 2 70B and GPT-3.5 on most of the benchmarks they tested, with roughly a fifth of Llama 2 70B’s active parameters. The paper’s routing analysis is the more interesting part. The authors checked whether experts specialized by subject, using text from sources like arXiv papers, biology abstracts and philosophy, and found no clear pattern by topic. What they did find was that routing followed structure: consecutive tokens often went to the same expert, and some tokens in code (such as indentation) were assigned consistently. The experts divided the work along lines that made sense to the optimizer, not to a librarian.<\/p>\n\n\n\n

Finer Experts and a Bias Instead of a Loss<\/strong><\/p>\n\n\n\n

DeepSeek’s line of models changed the recipe in two ways. The DeepSeekMoE paper (Dai and colleagues, ACL 2024) argued that eight or sixteen large experts are too coarse, because each one ends up covering a mix of unrelated knowledge. Its answer was fine-grained segmentation: split each expert into several smaller ones and activate more of them, so the compute per token stays the same while the number of possible combinations grows. It also isolated a few shared experts that every token passes through, so common knowledge does not have to be copied into every routed expert.<\/p>\n\n\n\n

DeepSeek-V3, described in a technical report released in December 2024, has 671 billion parameters with 37 billion active per token, trained on 14.8 trillion tokens using 2.788 million H800 GPU hours. Its most cited routing change is what the authors call auxiliary-loss-free load balancing. Each expert gets a bias term that is added to its score only when deciding which experts to select; the weights used to combine the outputs still come from the original scores. After each step, the bias of an overloaded expert is lowered and the bias of an underused one is raised, by a fixed small amount. Balance is enforced by a dial outside the gradient, so the language modeling loss is not pulled in two directions. A tiny sequence-level balance loss is kept only as a safety net against extreme cases.<\/p>\n\n\n\n

Memory and Communication<\/strong><\/p>\n\n\n\n

Sparse computation does not mean sparse memory. All 46.7 billion of Mixtral’s parameters must be loaded to serve it, because any token might need any expert. On a single machine, an MoE model needs the memory of its total size and gives the speed of its active size. That trade is favorable for providers serving many users at once and less favorable for someone running a model on one consumer GPU.<\/p>\n\n\n\n

At larger scale, the experts live on different accelerators, an arrangement called expert parallelism. Every MoE layer then requires two all-to-all exchanges: tokens are sent to whichever devices hold their chosen experts and the results are sent back. GShard (Lepikhin and colleagues, ICLR 2021) was largely a paper about making this practical, and it trained a 600-billion-parameter multilingual translation model, covering 100 languages into English, on 2,048 TPU v3 cores in four days. Much of the engineering in recent MoE reports is about hiding this communication behind computation, because on real clusters the network, not the arithmetic, is often the bottleneck.<\/p>\n\n\n\n

A Final Observation<\/strong><\/p>\n\n\n\n

The word \u00abexpert\u00bb invites a picture of a panel of specialists: one for medicine, one for law, one for code. The evidence points elsewhere. The 1991 vowel experiments, Mixtral’s routing analysis and DeepSeek’s push for finer granularity all show experts organizing around whatever division of labor lowers the loss and keeps the load even, which is often syntactic, positional or simply hard to name. A mixture of experts is better understood as a way of renting capacity: the model keeps a very large set of parameters and pays for a small slice of them at a time. The specialists are a metaphor that the training process never agreed to.<\/p>\n\n\n\n

By: Max Johnson B.<\/p>\n","protected":false},"excerpt":{"rendered":"

Paying for Every Parameter In a standard dense neural network, every parameter takes part in every prediction. If a language model has 70 billion weights, all 70 billion are multiplied against each token it reads, whether the token is a word in a legal contract, a line of Python or a comma. Capacity and cost […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-115","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/115","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=115"}],"version-history":[{"count":0,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/115\/revisions"}],"wp:attachment":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}