A tokenizer is a system or algorithm that converts input data into tokens that a model can process. For multimodal models, tokenization may convert text, images, or other inputs into a unified representational format such as embeddings.
A system or algorithm that translates a sequence of input data into tokens. Most modern foundation models are multimodal. A tokenizer for a multimodal system must translate each input type into the appropriate format. For example, given input data consisting of both text and graphics, the tokenizer might translate input text into subwords and input images into small patches. The tokenizer must then convert all the tokens into a single unified embedding space, which enables the model to "understand" a stream of multimodal input.