Story of Attention — Part 6: Where Attention Stands Today

Part 6 of 6 in the Story of Attention series.

1. Before the Transformer — how n-gram counting and RNN memory hit their ceilings, and how additive/multiplicative attention first let a decoder look back.
2. Attention All You Need — scaled dot-product attention, multi-head attention, and the four-year search for a way to encode position.
3. KV-Cache Bottleneck: MQA and GQA — sharing and grouping key/value heads to shrink inference memory.
4. Sparse Attention, Then and Now — fixed, windowed, hashed, and learned sparsity patterns, from 2019 to 2025.
5. Linear Attention: Kernels, Decay, and Gating — replacing softmax with decomposable kernels, decay, and gated recurrent state.
6. Where Attention Stands Today (this post) — latent compression, differential attention, and what frontier LLMs actually ship.

---

Every attention mechanism in this series tried to speed up at the expense of some quality of the response. Sparsity gave up completeness. Linear attention's kernel, decay, and gating method brought the cost down to $O(n)$. But it made it harder to find a fact in a long context. DeepSeek's Multi-Head Latent Attention leaves softmax's sharpness untouched and compresses the cache instead. The Differential Transformer leaves the cache untouched and cleans up softmax's own signal instead.

By the end of the previous part, linear attention had given up softmax's sharpness for constant-memory recurrence. It still lagged on exact retrieval. Neither sparsity nor linearity alone solved the problem. So in this part we are going to see what a mechanism without losing much quality would look like.

---

1. Multi-Head Latent Attention — MLA

GQA shrank the cache by storing fewer key/value heads. MLA shrinks it by storing something smaller than keys and values altogether, a latent vector, which is a compressed code that keys and values can be reconstructed from if required. The decompression happens through learned projections, but a small piece of algebra, the heart of this design, means that decompression mostly does not need to happen at all.

The framework runs in two directions. Down-projection compresses each token's hidden state into the latent:

$c_t^{KV} = h_t W^{DKV} \quad \text{where } W^{DKV} \in \mathbb{R}^{d_\text{model} \times d_c}, \; d_c \ll h \cdot d_k$

Only $c_t^{KV}$, of dimension $d_c$, enters the KV cache. Up-projection reconstructs full keys and values when attention needs them:

$k_t^I = c_t^{KV} W^{UK}, \quad v_t^I = c_t^{KV} W^{UV}$

with queries compressed symmetrically:

$c_t^Q = h_t W^{DQ}, \quad q_t^I = c_t^Q W^{UQ}$

At first glance this looks like it just swaps memory for compute, since every cached latent must be decompressed at every step. The absorption trick removes that cost. Let's write out the attention score between a query and a key:

$\text{Score}_i = q_t^I (k_t^I)^T = c_t^Q W_i^{UQ} (W_i^{UK})^T (c_t^{KV})^T$

Key result: $W_i^{UQ}$ and $W_i^{UK}$ are static learned matrices, so the product $W_i^{UQ} (W_i^{UK})^T$ can be pre-multiplied once before inference. The model then matches compressed queries directly against compressed latents. The full keys are never materialized in GPU memory.

There is one case the absorption trick does not survive, and that is RoPE. Rotary position encoding rotates each query and key by an angle that depends on its position, and that rotation does not commute with the pre-multiplied $W_i^{UQ}(W_i^{UK})^T$ matrix from above. The merged matrix would have to depend on both positions $t$ and $j$, which defeats the whole point of computing it once. DeepSeek-V2's fix is to decouple. It splits queries and keys into a compressed part, which gets the full absorption treatment above, and a small separate part that carries all the positional signal through RoPE instead:

$[\mathbf{q}_{t,1}^R;\mathbf{q}_{t,2}^R;\ldots;\mathbf{q}_{t,n_h}^R] = \mathbf{q}_t^R = \text{RoPE}(W^{QR}\mathbf{c}_t^Q), \quad \mathbf{k}_t^R = \text{RoPE}(W^{KR}\mathbf{h}_t)$

The query's RoPE part is computed once across all heads and then sliced per head. The key's RoPE part, $\mathbf{k}_t^R$, is a single vector shared across every head, and it is not decompressed per head the way $\mathbf{k}_{t,i}^C$ is. Concatenate the compressed and RoPE parts before scoring:

$\mathbf{q}_{t,i} = [\mathbf{q}_{t,i}^C;\mathbf{q}_{t,i}^R], \quad \mathbf{k}_{t,i} = [\mathbf{k}_{t,i}^C;\mathbf{k}_t^R]$

$\mathbf{o}_{t,i} = \sum_{j=1}^{t}\text{Softmax}_j\!\left(\frac{\mathbf{q}_{t,i}^T\mathbf{k}_{j,i}}{\sqrt{d_h+d_h^R}}\right)\mathbf{v}_{j,i}^C, \quad \mathbf{u}_t = W^O[\mathbf{o}_{t,1};\mathbf{o}_{t,2};\ldots;\mathbf{o}_{t,n_h}]$

So the attention score splits into an absorbed piece, which is compressed, cheap, and has no RoPE, plus a small RoPE piece computed the ordinary way. That RoPE piece is deliberately kept small, because it is the one part of the score that cannot be pre-folded. This is also why the KV cache ends up holding two things instead of one, the compressed latent $c_t^{KV}$, and the shared RoPE key $\mathbf{k}_t^R$.

For DeepSeek-V2 specifically, $d_c = 4d_h$ and $d_h^R = d_h/2$, so the total cache per token per layer is $d_c + d_h^R = 4.5\,d_h$ elements, regardless of how many heads $n_h$ the model runs. Let's line this up against the other variants from Part 3, in elements per token per layer [1]:

Key result: MLA's cache is pinned to $4.5\,d_h$ regardless of head count $n_h$, the same footprint as GQA run with only 2.25 groups [1], and yet MLA measures stronger than full MHA, not weaker. Grouping sacrifices quality for a smaller cache. MLA's low-rank compression barely sacrifices anything.

Worked example

Let's make the mechanics concrete with a small numeric pass. $d_\text{model} = 4$, $d_c = 2$ (latent dimension), $d_k = 4$ (uncompressed head dimension).

Hidden state: $h_t = [1, 2, 0, 1]$.

Down-projection: $W^{DKV} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \\ 0 & 2 \end{bmatrix}$

$c_t^{KV} = h_t W^{DKV} = [1, 4]$

This 2-element vector is the only data stored in the KV cache for this token, rather than the 4 elements a full key would need.

Up-projection: $W^{UK} = \begin{bmatrix} 2 & 1 & 0 & 1 \\ 0 & 1 & 3 & 0 \end{bmatrix}$

$k_t^I = c_t^{KV} W^{UK} = [1, 4] \begin{bmatrix} 2 & 1 & 0 & 1 \\ 0 & 1 & 3 & 0 \end{bmatrix} = [2, 5, 12, 1]$

Verification via absorption: Assume $q_t^I = [1, 0, 1, 2]$.

Method A (standard): $\text{Score} = q_t^I \cdot (k_t^I)^T = 2 + 0 + 12 + 2 = 16$.

Method B (absorbed): $q_t^I (W^{UK})^T = [4, 3]$, then $[4, 3] \cdot [1, 4]^T = 4 + 12 = 16$.

Both ways you get 16, and the absorbed path never constructs the full 4-dimensional key. The decompression only happens algebraically, it does not sit in memory.

The compressed part is only half the score. Let's finish it by adding the decoupled RoPE piece, with $d_h^R = 2$, half of $d_h = 4$, matching DeepSeek-V2's own ratio. First project the query latent and the hidden state into the RoPE subspace:

$c_t^Q = h_t W^{DQ}$ with $W^{DQ} = \begin{bmatrix}1&1\\1&0\\0&1\\1&1\end{bmatrix} \Rightarrow c_t^Q = [4, 2]$

$h_t W^{KR}$ with $W^{KR} = \begin{bmatrix}1&0\\0&1\\1&0\\0&1\end{bmatrix} \Rightarrow h_t W^{KR} = [1, 3]$

Rotate both by a clean illustrative angle of $90°$ ($[x_1,x_2] \to [-x_2,x_1]$):

$q_t^R = \text{RoPE}([4,2]) = [-2,4], \quad k_t^R = \text{RoPE}([1,3]) = [-3,1]$

Concatenate onto the compressed parts already computed, giving full query $= [1,0,1,2,-2,4]$ and full key $= [2,5,12,1,-3,1]$. The RoPE piece gets no absorption trick, it is computed the ordinary way every time:

$q_t^R \cdot k_t^R = (-2)(-3) + (4)(1) = 10$

(Because this compares token $t$ against itself, offset zero, the rotation cancels exactly and this matches the unrotated dot product $[4,2]\cdot[1,3]=10$. That's RoPE working as intended, not a shortcut: at any nonzero offset the two angles would differ and the rotation would actually change the score.)

The total raw score is $16 + 10 = 26$, and scaled by $\sqrt{d_h + d_h^R} = \sqrt{6}$ this gives ≈ 10.61. The 16 came from cached, pre-absorbed math, and the 10 came from a live two-dimensional rotation that has to run fresh at every step, exactly the split the equations above predicted.

Here is the whole picture, both the architecture and the cache footprint:

_{The DeepSeek-V2 block: DeepSeekMoE's routed and shared experts feed the feed-forward sublayer, while Multi-Head Latent Attention compresses queries, keys, and values through a shared latent before the attention sublayer. Only the latent and the RoPE key component are cached. Redrawn from DeepSeek-AI, 2024 [1].}

Similarly, drawn this time as a direct comparison instead of a block diagram:

_{What each variant caches per token: MHA caches every key and value, GQA and MQA share them across groups, and MLA caches only a single compressed latent, projected back out to keys and values at attention time. Redrawn from DeepSeek-AI, 2024 [1].}

Setting the theory aside, DeepSeek-V2's own reported numbers make the case directly, against DeepSeek 67B, its dense-attention predecessor [1]:

So MLA cuts the cache by over 90% and generates nearly six times faster, and the gain comes from the same place MQA's did back in Part 3, less memory traffic per decoded token. This comparison mixes in DeepSeek-V2's MoE and other architecture changes alongside MLA, not an isolated attention-only ablation, but it is the real headline result the paper reports. MLA ships in DeepSeek-V2 [1] and DeepSeek-V3 [2].

The archetecture does have a few issues. The decoupled RoPE pathway above is one of them, extra machinery the architecture needs only because compression and rotation do not mix well. The latent width $d_c$ is a bottleneck too, if it is too small, distinct keys collapse into indistinguishable latents. And training requires careful initialization to keep the compression stable.

MLA only answers the storage question. A compressed cache says nothing about whether the attention scores themselves deserve their confidence, a head can be perfectly sharp and still confidently wrong, locking onto irrelevant context. The Differential Transformer mechanism proposes solution for that.

---

2. Differential Transformer

Ye et al. [3] start from a problem, if you look closely at a trained Transformer will see that attention heads give weight to tokens that do not matter. When a signal is contaminated by noise, measure it twice and subtract the two measurements, so the noise that is common to both cancels out. The layer is essentially a differential amplifier built out of softmax.

The layer projects the input into two independent Q/K spaces sharing one V, and computes two attention maps:

$A_1 = \text{softmax}\!\left(\frac{Q_1 K_1^T}{\sqrt{d_k}}\right), \quad A_2 = \text{softmax}\!\left(\frac{Q_2 K_2^T}{\sqrt{d_k}}\right)$

A learned scalar $\lambda$ sets the subtraction strength:

$\lambda = \exp(\lambda_{q1} \cdot \lambda_{k1}) - \exp(\lambda_{q2} \cdot \lambda_{k2}) + \lambda_\text{init}$

where $\lambda_{q1}, \lambda_{k1}, \lambda_{q2}, \lambda_{k2}$ are learned vectors and $\lambda_\text{init}$ is a constant anchor. Then:

$A_\text{diff} = A_1 - \lambda A_2$

$\text{Output} = A_\text{diff} \times V$

The first map captures signal plus noise. The second, trained under the subtraction, learns to capture the common-mode noise. And the difference between them is a cleaner signal than either map on its own.

Worked example

$n = 4$, $d_k = 4$, $\lambda_\text{init} = 0.8$.

$Q_1 = \begin{bmatrix}1&0&1&0\\0&2&0&1\\1&1&0&0\\0&0&1&1\end{bmatrix}, \; K_1 = \begin{bmatrix}1&1&0&0\\0&1&1&0\\0&0&2&1\\1&0&0&1\end{bmatrix}, \; Q_2 = \begin{bmatrix}0&1&1&0\\1&0&0&1\\1&0&1&0\\0&1&0&2\end{bmatrix}, \; K_2 = \begin{bmatrix}1&0&0&1\\0&1&1&0\\1&1&0&0\\0&0&1&2\end{bmatrix}, \; V = \begin{bmatrix}2&0&1&1\\1&3&0&0\\0&1&4&2\\1&2&0&5\end{bmatrix}$

With $\lambda_{q1} \cdot \lambda_{k1} = 0.5$ and $\lambda_{q2} \cdot \lambda_{k2} = 0.5$: $\lambda = e^{0.5} - e^{0.5} + 0.8 = 0.8$.

$A_1 \approx \begin{bmatrix} 0.215 & 0.215 & 0.355 & 0.215 \\ 0.311 & 0.311 & 0.189 & 0.189 \\ 0.387 & 0.235 & 0.143 & 0.235 \\ 0.114 & 0.188 & 0.510 & 0.188 \end{bmatrix}, \quad A_2 \approx \begin{bmatrix} 0.143 & 0.387 & 0.235 & 0.235 \\ 0.336 & 0.124 & 0.204 & 0.336 \\ 0.250 & 0.250 & 0.250 & 0.250 \\ 0.203 & 0.123 & 0.123 & 0.551 \end{bmatrix}$

$A_\text{diff} = A_1 - 0.8 A_2 \approx \begin{bmatrix} 0.101 & -0.095 & 0.167 & 0.027 \\ 0.042 & 0.212 & 0.026 & -0.080 \\ 0.187 & 0.035 & -0.057 & 0.035 \\ -0.048 & 0.089 & 0.412 & -0.253 \end{bmatrix}$

Note the negative weights here. Differential attention can actively suppress a distracting token, not just assign it a small positive weight. Softmax alone cannot produce this kind of inhibition, but the subtraction can.

$\text{Output} \approx \begin{bmatrix} 0.134 & -0.064 & 0.768 & 0.570 \\ 0.217 & 0.502 & 0.145 & -0.307 \\ 0.445 & 0.118 & -0.042 & 0.248 \\ -0.260 & 0.174 & 1.600 & -0.490 \end{bmatrix}$

_{Left: two Q/K projections feed the differential attention block, scaled by GroupNorm and (1−λinit) before the output projection. Right: reference implementation of DiffAttn and the multi-head wrapper. Redrawn from Ye et al., 2024 [3].}

The paper's real results back this up directly [3]:

The mechanism is still research-stage. Microsoft Research introduced it, and no major production model has used it yet. The reason is simple arithmetic. Computing two full attention maps per head doubles the cost of the attention step, and this comes at a time when every other mechanism in this series exists to bring that cost down. The dynamic $\lambda$ adds more parameters and more moving parts on top of that. Whether cleaner attention is worth double the cost at 100B+ scale is a question only real training budget can answer, and as of this writing, nobody has spent that budget publicly.

That covers the individual fixes to the 2017 attention layer: compress the cache, sparsify the pattern, linearize the kernel, denoise the score. None of them won on its own. Combining them did.

---

3. Attention Mechanisms in Frontier Models

After a decade of invention, here is what actually shipped:

The conservative consensus: GQA + RoPE. LLaMA 3, Mistral, Qwen, and Gemma, most of the open-weight world sits here, and not out of timidity. GQA gives a 4–8× cache reduction, RoPE gives dependable length behavior, softmax stays unmodified inside, and every kernel and serving stack is already optimized for it. For 128K contexts, this combination is good enough that departing from it needs a real reason.

The compression bet: MLA. DeepSeek pushed cache reduction well past GQA by storing latents instead of heads, at the cost of a more complicated architecture. (The same models pair MLA with a sparsely-activated Mixture-of-Experts feed-forward stack, which is a parameter-routing technique rather than an attention mechanism, but it is the reason DeepSeek-V3 carries 671B parameters while activating only 37B per token.) The implementation is harder, but the throughput gain is large, and the bet is that the implementation cost gets paid once while the throughput keeps paying off. Moonshot's Kimi K2 makes the same bet at 1-trillion-parameter scale, following DeepSeek-V3's MLA design directly.

The hybrid bet: linear and softmax, interleaved. MiniMax and Moonshot's Kimi Linear [11] accept the verdict from Part 5, that linear attention cannot carry retrieval alone, and route around it in depth. Linear or gated-recurrent layers absorb the bulk of the sequence at $O(n)$, while periodic softmax layers supply the sharp retrieval. This is the most architecturally adventurous of the three positions, and the only one currently fielding million-token contexts.

---

The Dissolving Matrix

The $n \times n$ attention matrix that defined the Transformer in 2017 is dissolving. It is not happening all at once. GQA models still compute the full matrix, just with fewer KV heads. MLA does not materialize the full keys. Lightning Attention does not materialize the matrix beyond a single block. Gated DeltaNet holds a fixed-size state and overwrites what's outdated. Each mechanism removes a different part of the original design. One removes the storage. Another removes the pattern. The other removes the kernel. The quadratic object at the center of the architecture is being erased from several directions at once. But the thing it computes, a learned routing of information that depends on content, has survived every one of these.

The matrix is not being replaced by a better matrix. It is being replaced by a division of labor. Sliding-window layers handle locality. Recurrent-state layers handle the bulk of the context. A few full softmax layers handle the retrieval that nothing else has managed to replicate. The 2017 Transformer answered every question with the same operation at every layer. The 2026 frontier model looks more like specialists, where different layers are built for different jobs.

← Part 1: Before the Transformer | ← Part 2: Attention All You Need | ← Part 3: KV-Cache Bottleneck | ← Part 4: Sparse Attention | ← Part 5: Linear Attention

---

References

[1] DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434.

[2] DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437.

[3] Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., & Wei, F. (2024). Differential Transformer. arXiv:2410.05258.

[4] OpenAI (2023). GPT-4 Technical Report. arXiv:2303.08774 (positional scheme not disclosed in the report).

[5] Grattafiori, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783.

[6] Jiang, A. Q., Sablayrolles, A., Mensch, A., et al. (2023). Mistral 7B. arXiv:2310.06825.

[7] Qwen Team (2024). Qwen2.5 Technical Report. arXiv:2412.15115.

[8] Gemma Team (2024). Gemma 2: Improving Open Language Models at a Practical Size. arXiv:2408.00118.

[9] MiniMax (2025). MiniMax-01: Scaling Foundation Models with Lightning Attention. arXiv:2501.08313.

[10] Kimi Team (2025). Kimi K2: Open Agentic Intelligence. arXiv:2507.20534.

[11] Kimi Team (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692.