Also known as: BERT Β· Bidirectional Encoder Representations from Transformers (BERT)
Bidirectional Encoder Representations From Transformers (BERT) is a transformer-based language model architecture developed for deep bidirectional language understanding. It is pre-trained on large text corpora and can be adapted to downstream natural language processing tasks such as question answering, classification, and search relevance. BERT's bidirectional design allows it to use both left and right context when representing text.
BERT is a method of pre-training language representations, meaning that we train a general-purpose "language understanding" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP.
Representations from Transformers) A model architecture for text representation. A trained BERT model can act as part of a larger model for text classification or other ML tasks. BERT has the following characteristics: - Uses the Transformer architecture, and therefore relies on self-attention. - Uses the encoder part of the Transformer. The encoder's job is to produce good text representations, rather than to perform a specific task like classification. - Is bidirectional. - Uses masking for unsupervised training. BERT's variants include: - ALBERT, which is an acronym for A L ight BERT. - LaBSE. See Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing for an overview of BERT.