An attention mechanism is a technique that allows a model to compute context-dependent weights over input elements and use those weights when generating an output. In transformer-based systems, attention is often implemented in multiple heads so that the model can attend to different relationships in parallel. The mechanism supports context-sensitive language understanding, generation, and representation learning.
Machine learning-based attention is a mechanism mimicking cognitive attention. It calculates "soft" weights for each word, more precisely for its embedding, in the context window. It can do it either in parallel (such as in transformers) or sequentially (such as in recursive neural networks). "Soft" weights can change during each runtime, in contrast to "hard" weights, which are (pre-)trained and fine-tuned and remain frozen afterwards. Multiple attention heads are used in transformer-based large language models.