Also known as: BERT · Bidirectional Encoder Representations from Transformers (BERT)
Bidirectional Encoder Representations from Transformers (BERT) — архитектура языковой модели на базе Transformer, разработанная для глубокого двунаправленного понимания языка. Она предварительно обучается на больших текстовых корпусах и может адаптироваться к последующим задачам обработки естественного языка, таким как ответы на вопросы, классификация и релевантность поиска. Двунаправленная конструкция BERT позволяет использовать левый и правый контекст при представлении текста.
BERT is a method of pre-training language representations, meaning that we train a general-purpose "language understanding" model on a large text corpus (like Wikipedia), and then use that model for downstream NLP tasks that we care about (like question answering). BERT outperforms previous methods because it is the first unsupervised, deeply bidirectional system for pre-training NLP.
Representations from Transformers) A model architecture for text representation. A trained BERT model can act as part of a larger model for text classification or other ML tasks. BERT has the following characteristics: - Uses the Transformer architecture, and therefore relies on self-attention. - Uses the encoder part of the Transformer. The encoder's job is to produce good text representations, rather than to perform a specific task like classification. - Is bidirectional. - Uses masking for unsupervised training. BERT's variants include: - ALBERT, which is an acronym for A L ight BERT. - LaBSE. See Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing for an overview of BERT.