Attention is a neural network mechanism that assigns different levels of importance to parts of an input when producing an output. It helps models focus on relevant tokens, features, or regions and is a core building block of transformer architectures. Attention can reduce the burden of representing all input information equally by weighting context according to task relevance.
A mechanism used in a neural network that indicates the importance of a particular word or part of a word. Attention compresses the amount of information a model needs to predict the next token/word. A typical attention mechanism might consist of a weighted sum over a set of inputs, where the weight for each input is computed by another part of the neural network. Refer also to self-attention and multi-head self-attention, which are the building blocks of Transformers. See LLMs: What's a large language model? in Machine Learning Crash Course for more information about self-attention.