Also known as: Multi head Self attention Β· Multi-head Self Attention
Multi-head self-attention is an attention mechanism that computes several attention patterns in parallel over the same input sequence. Each head can capture different relationships among tokens, and the combined output helps transformer models represent complex contextual dependencies.
An extension of self-attention that applies the self-attention mechanism multiple times for each position in the input sequence. Transformers introduced multi-head self-attention.