Ein Token ist die kleinste Eingabe- oder Ausgabeeinheit, die ein Sprachmodell verarbeitet, etwa ein Wort, Subword, Zeichen, Satzzeichen oder spezielles Symbol. In multimodalen Systemen können Tokens auch atomare Einheiten nichttextlicher Daten wie Bild-Patches repräsentieren. Tokenanzahlen werden häufig genutzt, um Kontextlänge, Modellarbeitslast und Nutzung zu messen.
A token in a language model is the atomic unit that the model is training and making inferences on, namely words, morphemes, and characters. In domains outside of language models, tokens can represent other kinds of atomic units. For example, in computer vision, a token might be a subset of an image. For more information, see List and count tokens.
In a language model, the atomic unit that the model is training on and making predictions on. A token is typically one of the following: - a word---for example, the phrase "dogs like cats" consists of three word tokens: "dogs", "like", and "cats". - a character---for example, the phrase "bike fish" consists of nine character tokens. (Note that the blank space counts as one of the tokens.) - subwords---in which a single word can be a single token or multiple tokens. A subword consists of a root word, a prefix, or a suffix. For example, a language model that uses subwords as tokens might view the word "dogs" as two tokens (the root word "dog" and the plural suffix "s"). That same language model might view the single word "taller" as two subwords (the root word "tall" and the suffix "er"). In domains outside of language models, tokens can represent other kinds of atomic units. For example, in computer vision, a token might be a subset of an image. See Large language models in Machine Learning Crash Course for more information.
The process of splitting a stream of text into separate tokens on detectable boundaries such as white space and hyphens.