Also known as: MOE
Mixture of Experts (MOE) is a model architecture that routes inputs to selected specialized submodels, or experts, rather than using the entire model for every input. This can increase model capacity while controlling computation, because only a subset of experts is activated for a given example.
A scheme to increase neural network efficiency by using only a subset of its parameters (known as an expert ) to process a given input token or example. A gating network routes each input token or example to the proper expert(s). For details, see either of the following papers: - Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer - Mixture-of-Experts with Expert Choice Routing