A KV cache stores the attention keys and values computed for earlier tokens. During ordinary causal decoding, adding a token does not change those earlier representations, so the model can reuse them instead of processing the entire prefix again.
Prefill first, then decode
During prefill, the model processes the prompt and creates cached keys and values at each attention layer. During decoding, it processes a newly generated token, computes that token’s query, key, and value, and appends the new key and value to the cache.
The new query still compares against the available cached keys. The resulting weights combine the corresponding values. Caching therefore removes repeated work; it does not make long-context attention free. With full attention, this attention work grows linearly with the cached length for each new token. Prefill has a different computation pattern.
Why is this reuse valid? A causal mask prevents an earlier position from depending on later tokens. For the same model and unchanged prefix, its earlier keys and values are still usable. Editing the prefix can invalidate the affected cache entries.
Calculate the memory
For a dense cache with equal-length sequences, the storage is approximately:
2 × layers × batch × cached tokens × KV heads × head dimension × bytes per element
The factor of two accounts for both keys and values. Consider 32 layers, one sequence, 8,192 cached tokens, eight KV heads, a head dimension of 128, and two-byte values. That is 1,073,741,824 bytes, or 1 GiB, for the cache alone. A batch of four such sequences needs about 4 GiB.
This excludes model weights, temporary tensors, allocator overhead, and any reserved capacity. Sliding windows, cache quantization, and other layouts change the calculation. Count KV heads, not query heads: grouped-query attention makes that distinction important.
What the cache does not do
It does not store a database of completed answers or give a model permanent memory. Reusing an identical prompt prefix across requests is another serving optimization; returning a cached answer is different again. See semantic caching vs prefix caching.
Read further: Hugging Face’s cache explanation.