What is a Tokenization?

In the context of language models, tokenization splits raw text into subword pieces using an algorithm such as byte-pair encoding, so common words become a single token while rarer words split into multiple pieces. This lets a model represent any input, including made-up words or other languages, with a fixed, manageable vocabulary size. Every model call, prompt and response alike, is measured, priced, and context-limited in tokens rather than characters or words, which is why token count directly drives cost and how much conversation history fits in a context window. Tokenization is a separate concept from the security meaning of the word, where sensitive data like a credit card number is replaced with a non-sensitive substitute token that maps back to the real value only through a secured vault. Both senses share the idea of substituting a raw value for a standardized unit, but they solve entirely different problems.