Subword token — токен, представляющий часть слова, а иногда и целое слово, используемый языковыми моделями. Subword tokenization позволяет моделям обрабатывать редкие, составные или морфологически сложные слова с использованием повторно применяемых меньших единиц.
In language models, a token that is a substring of a word, which may be the entire word. For example, a word like "itemize" might be broken up into the pieces "item" (a root word) and "ize" (a suffix), each of which is represented by its own token. Splitting uncommon words into such pieces, called subwords, allows language models to operate on the word's more common constituent parts, such as prefixes and suffixes. Conversely, common words like "going" might not be broken up and might be represented by a single token.