Un attention mechanism es una técnica que permite a un modelo calcular pesos dependientes del contexto sobre elementos de entrada y usar esos pesos al generar una salida. En sistemas basados en transformer, la atención a menudo se implementa en múltiples heads para que el modelo pueda atender diferentes relaciones en paralelo. El mecanismo apoya comprensión del lenguaje, generación y aprendizaje de representaciones sensibles al contexto.
Machine learning-based attention is a mechanism mimicking cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. It can do it either in parallel (such as in transformers) or sequentially (such as in recursive neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards. Multiple attention heads are used in transformer-based large language models.