Self-attention updates a token’s representation using information from positions in the same sequence. An attention head scores the allowed positions, turns those scores into weights, and combines their value vectors.
Follow the weighted sum
In the card’s illustrative sentence, the position for “sat” assigns a large weight to “cat.” This is a demonstration of the operation, not a measurement from a trained model. The weights describe how much of each value vector goes into the output of one head at one position.
For a smaller example, imagine three value vectors: [2, 0], [0, 4], and [2, 2]. Weights of 0.5, 0.25, and 0.25 produce [1.5, 1.5]. Each coordinate is a weighted sum. The model mixes vectors; it does not select a single word and copy it.
The weights come from comparing queries and keys, scaling the scores, and applying softmax. Before attention dropout, softmax weights across the allowed positions sum to one. Different queries can produce different mixtures from the same set of values.
Which positions are allowed?
The diagram illustrates attention over a whole sentence. In a bidirectional encoder, a position can use tokens on both sides. In a causal decoder, the position for “sat” cannot attend to words that follow it. A mask removes those future positions before the weights are calculated.
This distinction matters during training: the full example may be present in memory, but the mask prevents the model from using the answer it is supposed to predict. Training can process many positions in parallel while still enforcing that rule. Ordinary generation must wait for each sampled token before continuing.
What one head cannot tell you
A high weight is not proof that a word caused the final answer. Multiple heads, later layers, residual connections, and feed-forward networks all affect the result. An attention diagram shows one part of that computation.
Full attention also compares many pairs of positions. That cost motivates the KV cache during decoding and other methods for handling long sequences.
Source: Attention Is All You Need.