A word is a unit of language. A character is a unit of written text. A token is a unit chosen by a tokenizer. Sometimes these boundaries line up, but an LLM’s token count is not a word count.
The same text, three counts
Take red fox!:
- As written here, it contains eight characters, including the space and exclamation mark.
- It contains two words in the ordinary reading of the sentence.
- A possible subword tokenization is
red,fox,!: three tokens. This is an example, not the output of every tokenizer.
Notice the space attached to fox. Tokenizers can include whitespace inside a token. Punctuation can be separate, joined to another piece, or split further, depending on the rules and vocabulary.
What each choice costs
Character-based tokenization uses short pieces and tends to produce longer sequences. It can represent unfamiliar words using familiar characters, provided those characters are supported. Bytes offer another approach: a byte vocabulary can cover UTF-8 text, but one visible character may require several bytes.
Word-based tokenization can keep common sentences compact. Its difficulty is the vocabulary: an unseen name may need an unknown-word marker or a separate fallback mechanism. It does not simply disappear from the input.
Subword tokenization keeps many frequent pieces together while splitting less familiar ones. It is widely used in LLMs, but character and byte approaches also exist. “Token” is the general term for any of these units, not a synonym for “subword.”
What should you count in practice?
Use the target model’s tokenizer when estimating context length or billed text usage. If a prompt contains repeated product codes, equations, or a language poorly represented in the tokenizer’s training data, a rule of thumb based on English words can be misleading. Chat formatting and tool definitions may also add tokens beyond the text you typed.
Read further: Hugging Face’s comparison of tokenization methods. Continue with how BPE builds a vocabulary.