Also known as: Transformers
A Transformer is a neural network architecture that relies heavily on attention mechanisms to process sequences and relationships among tokens. Transformers are the foundation of many modern language, vision, audio, and multimodal AI systems because they scale effectively and can model long-range dependencies.
A type of deep learning architecture that exploits a multi-head attention mechanism. Transformers address some of the limitations of long short-term memory, and became widely used in natural language processing, although it can also process other types of data such as images in the case of vision transformers.
A neural network architecture developed at Google that relies on self-attention mechanisms to transform a sequence of input embeddings into a sequence of output embeddings without relying on convolutions or recurrent neural networks. A Transformer can be viewed as a stack of self-attention layers. A Transformer can include any of the following: - an encoder - a decoder - both an encoder and decoder An encoder transforms a sequence of embeddings into a new sequence of the same length. An encoder includes N identical layers, each of which contains two sub-layers. These two sub-layers are applied at each position of the input embedding sequence, transforming each element of the sequence into a new embedding. The first encoder sub-layer aggregates information from across the input sequence. The second encoder sub-layer transforms the aggregated information into an output embedding. A decoder transforms a sequence of input embeddings into a sequence of output embeddings, possibly with a different length. A decoder also includes N identical layers with three sub-layers, two of which are similar to the encoder sub-layers. The third decoder sub-layer takes the output of the encoder and applies the self-attention mechanism to gather information from it. The blog post Transformer: A Novel Neural Network Architecture for Language Understanding provides a good introduction to Transformers. See LLMs: What's a large language model? in Machine Learning Crash Course for more information.
A side-by-side comparison of Convolutional Neural Network and Transformer. Understand how local-pattern learning differs from attention-based sequence and relationship modeling.
A side-by-side comparison of Recurrent Neural Network and Transformer. Understand how the concepts differ, when each term applies, and why the distinction matters for AI governance, evaluation, or system design.