Also known as: MOE
Mixture of Experts (MOE) es una arquitectura de modelo que enruta entradas a submodelos especializados seleccionados, o expertos, en lugar de usar todo el modelo para cada entrada. Esto puede aumentar la capacidad del modelo mientras controla el cómputo, porque solo se activa un subconjunto de expertos para un ejemplo determinado.
A scheme to increase neural network efficiency by using only a subset of its parameters (known as an expert ) to process a given input token or example. A gating network routes each input token or example to the proper expert(s). For details, see either of the following papers: - Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer - Mixture-of-Experts with Expert Choice Routing