Queries and keys determine the attention weights. Values are the vectors those weights combine. Separating these jobs lets an attention head choose where to get information independently of what information it passes along.
Three views of the same input
For a token’s current hidden vector h, learned projections produce q = hWq, k = hWk, and v = hWv. The projection weights are shared across positions within a layer, but different input vectors produce different queries, keys, and values. Each layer learns its own projections.
“What am I looking for?” is a useful description of a query, but it is an analogy. The query is a numeric vector, not a question written in language. A key is another numeric vector against which it can be compared.
A numerical example
Suppose a query can attend to two positions. After dot products and scaling, its scores are 2 and 1. Softmax turns those into weights of about 0.73 and 0.27.
If the value vectors are [10, 0] and [0, 10], the output of this head at that position is approximately [7.3, 2.7]. The first position contributed more because its key matched the query more strongly. Its value supplied the content that was mixed into the output.
Changing a key can change the mixture proportions. Changing a value while keeping queries and keys fixed changes the content of the mixture without changing those proportions. That is the reason for having separate projections.
Where the mask fits
A causal attention mask excludes future positions before softmax. Their values cannot contribute, regardless of how well their keys might match. In self-attention, Q, K, and V come from the same sequence; in cross-attention, queries can come from one sequence while keys and values come from another.
During token-by-token generation, a new query uses earlier keys and values saved in the KV cache. The previous queries are not needed for this step.
Source: Attention Is All You Need, section 3.2. Continue with self-attention and masking.