An LLM tokenizer converts text into a sequence of token IDs. A token might represent a whole word, part of a word, punctuation, or a piece of whitespace. The model then looks up a learned vector for each ID before processing the sequence.
Follow one word through the process
Imagine a tokenizer splitting unbelievable into un and believable. It looks up each piece in its vocabulary and returns two integers. Those integers identify entries; a larger ID does not mean a longer word or a more important concept.
This split is illustrative. A different tokenizer might keep the word whole or divide it differently. Even adding a leading space can change the result. To count a real prompt, use the tokenizer for the model that will receive it.
Why not give every word an ID?
A word list cannot anticipate every name, spelling mistake, product code, or newly coined term. Subword tokenizers reuse smaller pieces, so they do not need a separate entry for every possible word. Some tokenizers work with bytes or use byte fallback; others can produce an unknown-token marker. Handling every input is a property of the particular tokenizer, not a promise made by the word “subword.”
The vocabulary and tokenization rules are normally fixed when a model is deployed. Typing a new word does not add a new vocabulary entry. See the BPE example for how one common algorithm learns its pieces.
Why this matters when using a model
Context windows and text API usage are usually measured in tokens. A short-looking identifier can occupy more tokens than a longer, familiar phrase. Languages, code, and punctuation have different token-to-character ratios, so a word count is only a rough budgeting tool.
Token boundaries also help explain why apparently simple spelling tasks can be awkward: a model may receive a word as one unit rather than as individual letters. Tokenization is only part of that explanation; it does not determine whether a particular model will succeed.
Read further: Hugging Face’s tokenizer overview. Compare tokens, words, and characters on the next card.