noun
Tokenization is the process of converting text into tokens, which are smaller units of meaning like words, parts of words, or even characters. AI models, especially large language models, process text as sequences of tokens rather than raw text.
Most tokenization systems split text into subword units (e.g., ‘running’ → ‘run’, ‘##ning’) to handle rare or unknown words. For example, the GPT-4 model uses a tokenizer that typically splits words into 1.5 tokens per word on average, meaning a 50-word input could become 75 tokens.
Tokenization affects how AI models handle input and output. It limits the length of text a model can process in one go (e.g., 4,096 tokens for GPT-4), which may require truncating or summarizing long texts. It also impacts costs, as some AI services charge per token used.
When you ask an AI to summarize a 1,000-word article, the system first tokenizes the text, which might result in 1,500 tokens, and then processes it in chunks to avoid exceeding the model's token limit.
In other languages
See also: prompt engineering