Queries, keys and values
Start with a sequence represented by X, a T × d_model matrix. One attention head learns three projections: Q = X W_Q, K = X W_K and V = X W_V. For this head, Q and K have width d_k; V has width d_v. Multiplying Q Kᵀ produces one score for every query/key pair, so its shape is T × T. The output of softmax(Q Kᵀ / √d_k) V has shape T × d_v. A query asks which positions matter for its current representation. A key participates in the matching score. A value supplies the vector that is mixed into the output. These roles describe the computation; they are not three separate copies of a sentence stored in a database. For the following example, use one query and three keys. Choose d_k = 2 and d_v = 2. The query is q = [√2, 0]. The keys are k₁ = [1, 0], k₂ = [0, 1] and k₃ = [1, 1]. The values are v₁ = [1, 0], v₂ = [0, 2] and v₃ = [3, 1]. These independently chosen vectors are an arithmetic example, not activations recorded from a trained language model.