Story of Attention — Part 5: Linear Attention: Kernels, Decay, and Gating

Part 5 of 6 in the Story of Attention series.

1. Before the Transformer — how n-gram counting and RNN memory hit their ceilings, and how additive/multiplicative attention first let a decoder look back.
2. Attention All You Need — scaled dot-product attention, multi-head attention, and the four-year search for a way to encode position.
3. KV-Cache Bottleneck: MQA and GQA — sharing and grouping key/value heads to shrink inference memory.
4. Sparse Attention, Then and Now — fixed, windowed, hashed, and learned sparsity patterns, from 2019 to 2025.
5. Linear Attention: Kernels, Decay, and Gating (this post) — replacing softmax with decomposable kernels, decay, and gated recurrent state.
6. Where Attention Stands Today — latent compression, differential attention, and what frontier LLMs actually ship.

---

Linear attention introduced by asking what happens if you drop the exponential altogether. Replace $\text{softmax}(QK^T)$ with a decomposable function $\phi(Q)\phi(K)^T$. Matrix multiplication is associative, so you can regroup the computation as $\phi(Q)[\phi(K)^TV]$ instead of $[\phi(Q)\phi(K)^T]V$. The bracketed term is a $d \times d$ matrix, and it stays the same size no matter how long the sequence gets.

In the previous part, models learned to skip most of the attention matrix. But for every pair they kept, softmax still ran its full exponential. This part removes softmax entirely, and asks how much quality can be recovered without it.

---

1. The Linear Attention Family

1a. Kernel Linear Attention

In the year 2020, Katharopoulos et al. [1] proposed an idea in the paper titled "Transformers are RNNs". Replace the softmax kernel with a feature map $\phi: \mathbb{R}^d \to \mathbb{R}^{d'}$ and factor:

$\text{LinearAttention}(Q, K, V)_i = \frac{\phi(q_i)^T \sum_j \phi(k_j) v_j^T}{\phi(q_i)^T \sum_j \phi(k_j)}$

In matrix form:

$\text{LinearAttention}(Q, K, V) = \frac{\phi(Q)\, [\phi(K)^T V]}{\phi(Q)\, [\phi(K)^T \mathbf{1}]}$

Key result: Compute $\phi(K)^T V \in \mathbb{R}^{d' \times d_v}$ first. This is a fixed-size matrix regardless of sequence length. Now, complexity drops from $O(n^2 \cdot d)$ to $O(n \cdot d'^{\,} \cdot d_v)$, linear in $n$.

A common choice for $\phi$ is $\text{elu}(x) + 1$, which keeps outputs non-negative.

The title's claim comes from the causal form. For autoregressive generation the cumulative sums become a recurrent state:

$S_n = S_{n-1} + \phi(k_n) v_n^T, \quad z_n = z_{n-1} + \phi(k_n)$

$\text{output}_n = \frac{\phi(q_n)^T S_n}{\phi(q_n)^T z_n}$

And that is an RNN, $O(1)$ per token with a constant-size state. In Part 1, attention was invented to free the RNN from its fixed-size memory, and here, six years later, attention deliberately becomes an RNN again. The fixed-size bottleneck that Bahdanau removed is brought back on purpose, as the price of linear cost. Now the obvious question becomes, is it worth doing?

Worked example

$n = 4$, $d_k = d_v = 4$, $\phi(x) = \text{elu}(x) + 1$ (for non-negative inputs, $\phi(x) = x + 1$).

$Q = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 2 & 0 & 1 \\ 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 2 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 & 0 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 2 & 1 \\ 1 & 0 & 0 & 1 \end{bmatrix}, \quad V = \begin{bmatrix} 2 & 0 & 1 & 1 \\ 1 & 3 & 0 & 0 \\ 0 & 1 & 4 & 2 \\ 1 & 2 & 0 & 5 \end{bmatrix}$

Since all values are non-negative, $\phi(x) = x + 1$:

$\phi(K)^T V = \begin{bmatrix} 7 & 8 & 6 & 14 \\ 7 & 9 & 6 & 9 \\ 5 & 11 & 13 & 12 \\ 5 & 9 & 9 & 15 \end{bmatrix}$

This $4 \times 4$ intermediate is the important part, because it does not grow with sequence length. For $n = 100{,}000$, this matrix would still be $4 \times 4$.

Output row 1 ($\phi(q_1) = [2, 1, 2, 1]$):

$\text{num}_1 = [2, 1, 2, 1] \cdot \phi(K)^T V = [36,\; 56,\; 53,\; 76]$

$\text{denom}_1 = [2, 1, 2, 1] \cdot \phi(K)^T \mathbf{1} = 38$

$\text{output}_1 = [0.95,\; 1.47,\; 1.39,\; 2.00]$

But going faster can hurt sometimes. For example, imagine the model receives a long, context-heavy input. In that situation, attention becomes very important for figuring out where to focus, so that you do not lose your very important context. No feature map $\phi$ can copy the way the exponential puts almost all its weight on the best match, so linear attention models score worse than softmax on perplexity (PPL, the exponentiated average negative log-likelihood the model assigns to held-out text; lower means the model is less "surprised" by what actually comes next), especially at smaller model sizes. The biggest gaps show up on copying and retrieval tasks, where one key has to clearly beat all the others. The normalization causes its own problem too. When $\phi(q_i)^T z_n$ drifts toward zero, the division becomes unstable.

1b. Performer / FAVOR+

If the problem is that $\phi$ is not softmax, then the simplest possible option one can think of is to build a $\phi$ that approximates softmax. The Performer [2] does this with random feature maps:

$\exp(q^T k) \approx \phi(q)^T \phi(k)$

where $\phi(x) = \frac{\exp(-\|x\|^2/2)}{\sqrt{m}} \begin{bmatrix} \exp(w_1^T x) \\ \vdots \\ \exp(w_m^T x) \end{bmatrix}$ and $w_1, \ldots, w_m \sim \mathcal{N}(0, I_d)$.

The FAVOR+ variant draws the random vectors orthogonally (lower variance) and makes sure the features stay positive (no negative attention weights). With the approximation in hand, the same linear factorization applies: softmax attention at linear cost, in expectation.

In practice, though, the approximation error never fully closed. The paper's own language-modeling experiments run on PG-19 and LM1B, not WikiText-103, and the comparison against a regular Transformer is reported only as a chart rather than an exact table: Performer needs a redrawn, positive-only feature map just to get close to regular-Transformer perplexity on PG-19, and even then a gap remains [2]. And the approximation is weakest exactly where it matters most, for example when the model has to copy an exact word or pull one specific fact out of a long context. Sharp attention distributions correspond to large dot products, and large dot products sit in the poorly approximated tail of the random-feature estimate. So retrieval and copying suffer again. On top of that, the random features make the model's outputs nondeterministic across runs, and production did not adopt it. What the Performer really showed is that softmax's exponential is not an incidental detail of attention. Even a method built specifically to approximate it could not close the gap.

1c. Linformer

Wang et al. [3] reached linear complexity by taking a completely different route. Although often people gets confused with the kernel methods above. Their empirical starting point is that trained attention matrices are approximately low-rank, so most of the $n \times n$ structure is redundant. So instead of changing the kernel, Linformer compresses the sequence axis. It projects the keys and values from length $n$ down to a fixed $k \ll n$ with learned projections $E_i, F_i \in \mathbb{R}^{k \times n}$:

$\text{Linformer}(Q, K, V) = \text{softmax}\!\left(\frac{Q(E_i K)^T}{\sqrt{d_k}}\right) (F_i V)$

Let's follow the dimensions through the computation:

• Project keys: $K_\text{new} = E_i K \in \mathbb{R}^{k \times d_k}$
• Project values: $V_\text{new} = F_i V \in \mathbb{R}^{k \times d_v}$
• Score matrix: $Q K_\text{new}^T \in \mathbb{R}^{n \times k}$ instead of $\mathbb{R}^{n \times n}$

With $k$ constant (e.g., 256), the cost is $O(n \cdot k)$: linear, and softmax survives intact, just over a compressed set of keys.

Worked example

$n = 4$, $d_k = d_v = 4$, $k = 2$.

$Q = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 \end{bmatrix}, \; K = \begin{bmatrix} 2 & 1 & 0 & 0 \\ 0 & 1 & 1 & 0 \\ 0 & 0 & 2 & 1 \\ 1 & 0 & 0 & 2 \end{bmatrix}, \; V = \begin{bmatrix} 1 & 2 & 0 & 1 \\ 0 & 1 & 3 & 0 \\ 2 & 0 & 1 & 1 \\ 0 & 1 & 0 & 2 \end{bmatrix}$

Projection matrix $E_i = F_i \in \mathbb{R}^{2 \times 4}$:

$E_i = \begin{bmatrix} 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 \end{bmatrix}$

Project keys ($K_\text{new} = E_i K$, size $2 \times 4$):

$K_\text{new} = \begin{bmatrix} 2 & 2 & 1 & 0 \\ 1 & 0 & 2 & 3 \end{bmatrix}$

Project values ($V_\text{new} = F_i V$, size $2 \times 4$):

$V_\text{new} = \begin{bmatrix} 1 & 3 & 3 & 1 \\ 2 & 1 & 1 & 3 \end{bmatrix}$

$K$ and $V$ have been compressed from 4 tokens to 2. The score matrix $Q K_\text{new}^T$ is now $4 \times 2$ instead of $4 \times 4$:

$\text{Scores} = Q K_\text{new}^T = \begin{bmatrix} 3 & 3 \\ 2 & 3 \\ 4 & 1 \\ 1 & 5 \end{bmatrix}$

Scale by $\sqrt{d_k} = 2$ and apply row-wise softmax:

$A_\text{lin} \approx \begin{bmatrix} 0.500 & 0.500 \\ 0.378 & 0.622 \\ 0.817 & 0.183 \\ 0.119 & 0.881 \end{bmatrix}$

Output $= A_\text{lin} \times V_\text{new}$:

$\text{Output} \approx \begin{bmatrix} 1.500 & 2.000 & 2.000 & 2.000 \\ 1.622 & 1.756 & 1.756 & 2.244 \\ 1.183 & 2.634 & 2.634 & 1.366 \\ 1.881 & 1.238 & 1.238 & 2.762 \end{bmatrix}$

The output is $4 \times 4$, matching the original dimensions, and no $4 \times 4$ attention grid was ever computed.

_{Linformer's multi-head block: keys and values are projected down to length k before attention runs; queries are not. Redrawn from Wang et al., 2020 [3].}

_{The projection collapses the sequence axis before attention: a k×n matrix (Ei or Fi) applied to K or V shrinks the token dimension from n to k. Redrawn from Wang et al., 2020 [3].}

Linformer is "linear attention" only in the complexity sense. Kernel methods linearize by decomposing the attention kernel, while Linformer linearizes by a low-rank projection of the sequence. The two mechanisms are unrelated, and their failure modes differ accordingly. Linformer's failures are structural. $E_i$ and $F_i$ have shapes fixed at training time, so the model cannot process sequences longer than it was built for, which is an awkward constraint for a long-context method. And the low-rank compression discards fine-grained token distinctions that some tasks need.

What the family so far establishes is that the quadratic matrix can be avoided, at a price in quality. Before we continue, we should look at a 2020 paper that asked an even simpler question. Its negative result ended up telling us what the dot product is actually for.

---

2. Synthesizer

Tay et al. [4] asked why attention weights should come from pairwise comparison at all. The Synthesizer generates the $n \times n$ weight matrix without any query-key dot products, from each token alone, or from nothing at all.

The Dense Synthesizer predicts each token's full row of attention weights from that token's own embedding, through a feed-forward network, with no interaction with other tokens:

$B = \text{ReLU}(X W_1)\, W_2 \quad \in \mathbb{R}^{N \times N}$

$Y = \text{softmax}(B) \times V$

Worked example

$N = 4$, $d = 4$.

$X = \begin{bmatrix} 1 & 0 & 1 & 1 \\ 0 & 1 & 0 & 0 \\ 1 & 1 & 0 & 1 \\ 0 & 0 & 1 & 1 \end{bmatrix}, \; W_1 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 0 \\ 1 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix}, \; W_2 = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 2 & 0 & 0 \\ 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 2 \end{bmatrix}$

After ReLU (all entries already non-negative) and multiplication by $W_2$:

$B = \begin{bmatrix} 4 & 2 & 3 & 2 \\ 0 & 2 & 0 & 0 \\ 2 & 3 & 2 & 2 \\ 2 & 1 & 2 & 2 \end{bmatrix}$

Row-wise softmax:

$A \approx \begin{bmatrix} 0.610 & 0.083 & 0.225 & 0.083 \\ 0.096 & 0.711 & 0.096 & 0.096 \\ 0.175 & 0.475 & 0.175 & 0.175 \\ 0.297 & 0.109 & 0.297 & 0.297 \end{bmatrix}$

The Random Synthesizer goes further and discards content entirely: a static matrix $R \in \mathbb{R}^{N \times N}$, learned by backpropagation, shared across every input the model will ever see:

$Y = \text{softmax}(R) \times V$

No queries, no keys: a fixed routing pattern. Example with $R = \begin{bmatrix} 3 & 1 & 1 & 1 \\ 1 & 2 & 1 & 1 \\ 0 & 1 & 3 & 1 \\ 1 & 1 & 1 & 4 \end{bmatrix}$:

$A = \text{softmax}(R) \approx \begin{bmatrix} 0.711 & 0.096 & 0.096 & 0.096 \\ 0.175 & 0.475 & 0.175 & 0.175 \\ 0.038 & 0.102 & 0.757 & 0.102 \\ 0.043 & 0.043 & 0.043 & 0.870 \end{bmatrix}$

For long sequences the full $R$ is too large, so a factorized variant writes $R_\text{final} = R_1 R_2^T$ with $R_1, R_2 \in \mathbb{R}^{N \times k}$, $k \ll N$. For $N = 4096$, $k = 8$: parameters drop from $4096^2 = 16.8\text{M}$ to $2 \times 4096 \times 8 = 65.5\text{K}$, a 99.6% reduction.

The results are the interesting part of this paper, because of how the picture splits in two. Quality is measured on GLUE (the General Language Understanding Evaluation, a standard suite of language-understanding tasks averaged into one score) and on WMT EN-DE translation [4]:

(The GLUE baseline is T5-Base; the WMT baseline is a vanilla Transformer — different base models, both using standard dot-product self-attention.)

So a routing matrix that ignores the input entirely comes within half a BLEU point of standard attention on translation. On GLUE, though, the gap is real: Dense and Random Synthesizer both land 8 to 11 points below the baseline. The translation result is the surprising part, it means that at least for translation, much of what attention layers do is generic mixing that does not depend on the input.

Although there is a gap in quality, one interesting fact it points out is where that gap shows up. On tasks that require evaluating a relationship between specific tokens, like reading comprehension, entailment, and logical reasoning, the synthesized weights fail, because no fixed or token-local pattern can express something like "this pronoun binds to that noun in this sentence." The authors' own conclusion was that sample-wise pairwise interaction is vital. Because of that, no major model used this mechanism. But it was still valuable, because it showed what the dot product is actually for. The $QK^T$ comparison is the part of attention that actually reads the input, and it cannot be replaced by a fixed pattern.

With pairwise comparison settled, what the linear attention program still needed was a way to recover softmax's other contribution, its bias toward locality and contrast. cosFormer went after that, from inside the kernel framework.

---

3. cosFormer

In 2022, Qin et al. [5] observed that softmax attention has two properties that a plain linear kernel lacks. The weights are non-negative, and there is a concentration effect that favors nearby context. Their design brings both into a decomposable kernel, a ReLU for the non-negativity, and a cosine reweighting that decays the scores smoothly with distance:

$\text{Similarity}_\text{cos}(q_i, k_j) = \phi(q_i)\phi(k_j)^T \cos\!\left(\frac{\pi|i - j|}{2M}\right)$

At first glance there is a problem here. A weight that depends on $|i - j|$ couples every query-key pair, which looks like it breaks decomposability. But the identity $\cos(\alpha - \beta) = \cos\alpha\cos\beta + \sin\alpha\sin\beta$ solves it, because the position indices separate:

$q_i^\text{cos} = \phi(q_i) \cos\!\left(\frac{\pi i}{2M}\right), \quad q_i^\text{sin} = \phi(q_i) \sin\!\left(\frac{\pi i}{2M}\right)$

$\text{Numerator}_i = q_i^\text{cos} \sum_j (k_j^\text{cos})^T v_j + q_i^\text{sin} \sum_j (k_j^\text{sin})^T v_j$

Each summation $\sum_j (k_j)^T v_j$ is again a fixed-size $d_k \times d_v$ matrix, so the total cost stays $O(n \cdot d_k \cdot d_v)$. So the model gets distance awareness while keeping the linear cost.

Worked example

$n = 4$, $d_k = d_v = 4$, $M = 4$.

$Q = \begin{bmatrix} 1 & -1 & 2 & 0 \\ 0 & 3 & -2 & 1 \\ 2 & 1 & 0 & -1 \\ -1 & 0 & 2 & 2 \end{bmatrix}, \quad K = \begin{bmatrix} 1 & 1 & 0 & -2 \\ -1 & 2 & 1 & 0 \\ 0 & 0 & 2 & 1 \\ 2 & -1 & 0 & 1 \end{bmatrix}, \quad V = \begin{bmatrix} 2 & 0 & 1 & 1 \\ 1 & 3 & 0 & 0 \\ 0 & 1 & 4 & 2 \\ 1 & 2 & 0 & 5 \end{bmatrix}$

After ReLU: $\phi(Q) = \begin{bmatrix} 1 & 0 & 2 & 0 \\ 0 & 3 & 0 & 1 \\ 2 & 1 & 0 & 0 \\ 0 & 0 & 2 & 2 \end{bmatrix}$, $\phi(K) = \begin{bmatrix} 1 & 1 & 0 & 0 \\ 0 & 2 & 1 & 0 \\ 0 & 0 & 2 & 1 \\ 2 & 0 & 0 & 1 \end{bmatrix}$

Cosine position weights ($C_{ij} = \cos(\pi|i-j|/8)$):

$C = \begin{bmatrix} 1.000 & 0.924 & 0.707 & 0.383 \\ 0.924 & 1.000 & 0.924 & 0.707 \\ 0.707 & 0.924 & 1.000 & 0.924 \\ 0.383 & 0.707 & 0.924 & 1.000 \end{bmatrix}$

Base scores $\phi(Q)\phi(K)^T$:

$\begin{bmatrix} 1 & 2 & 4 & 2 \\ 3 & 6 & 1 & 1 \\ 3 & 2 & 0 & 4 \\ 0 & 2 & 6 & 2 \end{bmatrix}$

element-wise multiplied by $C$:

$B_\text{cos} = \begin{bmatrix} 1.000 & 1.848 & 2.828 & 0.765 \\ 2.772 & 6.000 & 0.924 & 0.707 \\ 2.121 & 1.848 & 0.000 & 3.696 \\ 0.000 & 1.414 & 5.543 & 2.000 \end{bmatrix}$

Row-wise linear normalization (divide by row sum, not softmax):

$A_\text{cos} \approx \begin{bmatrix} 0.155 & 0.287 & 0.439 & 0.119 \\ 0.266 & 0.577 & 0.089 & 0.068 \\ 0.277 & 0.241 & 0.000 & 0.482 \\ 0.000 & 0.158 & 0.619 & 0.223 \end{bmatrix}$

Output:

$\text{Output} \approx \begin{bmatrix} 0.716 & 1.537 & 1.912 & 1.628 \\ 1.178 & 1.955 & 0.622 & 0.784 \\ 1.277 & 1.688 & 0.277 & 2.688 \\ 0.381 & 1.539 & 2.475 & 2.354 \end{bmatrix}$

On the Long Range Arena (LRA) benchmark, a suite of synthetic and real tasks (up to 16K tokens) built specifically to test how well an architecture holds up as sequence length grows, cosFormer beat the vanilla Transformer, and it also came out ahead on WikiText-103 language-modeling perplexity [5]:

Direct adoption of this mechanism was limited, but the idea carried forward. A decay function standing in for softmax's concentration reappears throughout what follows, most prominently in RetNet. cosFormer's own constraints are the fixed shape of the decay, one schedule that is not learnable and not per-head, and the softness of linear normalization, which divides by a row sum rather than exponentiating and so it does not produce truly peaked distributions. So sharp retrieval remains out of reach.

RetNet took the decay idea and tried to fix two things. It provides the local bias, and it also unlocks a second, recurrent representation of the same computation.

---

4. RetNet — Retentive Network

In 2023, Sun et al. [6] set an explicit target they called the "impossible triangle", which is parallel training, cheap inference, and competitive quality. Historically an architecture could have any two of these. The Transformer has the first and the third, and RNNs have the second. RetNet's claim was all three, and it comes from one design decision, drop softmax and weight the past positions with a causal geometric decay $\gamma^{i-j}$.

Parallel form (training):

$\text{Retention}(Q, K, V) = (QK^T \odot D) V$

where $D \in \mathbb{R}^{n \times n}$ is the causal decay matrix: $D_{ij} = \gamma^{i-j}$ if $i \geq j$, else $0$.

Recurrent form (inference):

$S_n = \gamma S_{n-1} + k_n^T v_n$

$\text{Retention}(q_n) = q_n S_n$

The state $S_n \in \mathbb{R}^{d_k \times d_v}$ is fixed-size; each decoding step costs $O(1)$ regardless of context length.

Key result: Both forms produce identical outputs. The parallel form exploits GPU parallelism during training, and the recurrent form gives constant-time inference. Softmax is what stood in the way, because as a row-wise nonlinearity over all the scores it cannot be pushed through the recurrence. Once it is removed, the same computation can be written in both shapes.

Worked example

$n = 4$, $d_k = d_v = 4$, $\gamma = 0.5$.

$Q = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 4 & 0 & 0 \\ 1 & 0 & 0 & 1 \\ 0 & 2 & 0 & 2 \end{bmatrix}, \; K = \begin{bmatrix} 1 & 1 & 0 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 0 & 1 & 0 \\ 0 & 0 & 1 & 1 \end{bmatrix}, \; V = \begin{bmatrix} 2 & 0 & 1 & 1 \\ 0 & 4 & 0 & 0 \\ 1 & 0 & 2 & 0 \\ 0 & 1 & 1 & 0 \end{bmatrix}$

Decay matrix ($\gamma = 0.5$):

$D = \begin{bmatrix} 1.0 & 0 & 0 & 0 \\ 0.5 & 1.0 & 0 & 0 \\ 0.25 & 0.5 & 1.0 & 0 \\ 0.125 & 0.25 & 0.5 & 1.0 \end{bmatrix}$

Parallel path: $QK^T = \begin{bmatrix} 1 & 0 & 2 & 1 \\ 4 & 4 & 0 & 0 \\ 1 & 1 & 1 & 1 \\ 2 & 4 & 0 & 2 \end{bmatrix}$

Element-wise: $A_\text{ret} = QK^T \odot D = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 2 & 4 & 0 & 0 \\ 0.25 & 0.5 & 1 & 0 \\ 0.25 & 1 & 0 & 2 \end{bmatrix}$

There is no softmax normalization here, the geometric decay bounds the values instead.

$\text{Output}_\text{parallel} = A_\text{ret} \times V = \begin{bmatrix} 2 & 0 & 1 & 1 \\ 4 & 16 & 2 & 2 \\ 1.5 & 2 & 2.25 & 0.25 \\ 0.5 & 6 & 2.25 & 0.25 \end{bmatrix}$

Recurrent verification (Token 3): $q_3 = [1, 0, 0, 1]$

$S_1 = k_1^T v_1 = \begin{bmatrix} 2 & 0 & 1 & 1 \\ 2 & 0 & 1 & 1 \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{bmatrix}$

$S_2 = 0.5 S_1 + k_2^T v_2 = \begin{bmatrix} 1 & 0 & 0.5 & 0.5 \\ 1 & 4 & 0.5 & 0.5 \\ 0 & 0 & 0 & 0 \\ 0 & 4 & 0 & 0 \end{bmatrix}$

$S_3 = 0.5 S_2 + k_3^T v_3 = \begin{bmatrix} 1.5 & 0 & 2.25 & 0.25 \\ 0.5 & 2 & 0.25 & 0.25 \\ 1.0 & 0 & 2.0 & 0 \\ 0 & 2 & 0 & 0 \end{bmatrix}$

$\text{Output}_3 = q_3 S_3 = [1, 0, 0, 1] \times S_3 = [1.5,\; 2.0,\; 2.25,\; 0.25]$

This is exactly row 3 of the parallel output, so the two forms really are the same computation.

_{RetNet's dual form: the same computation written as a parallel matrix product for training (left) and as a recurrent state update with decay γ for inference (right). Redrawn from Sun et al., 2023 [6].}

The paper's actual finding is a crossover, not a clean win at every size: RetNet only starts to outperform Transformer on perplexity once the model size passes 2B parameters [6]. At 6.7B, where the paper also runs a head-to-head zero-shot comparison, RetNet leads on every task tested:

As a standalone architecture, RetNet found little adoption, but its ideas spread more than the model did. Parallel training paired with recurrent inference became the template for the mechanisms that close this post. The decay-weighted state reappears in Lightning Attention's inter-block recurrence, and in a gated form in DeltaNet.

RetNet's weaknesses come from the same design decision. The decay $\gamma^{i-j}$ is fixed and monotonic, so "older" means "less relevant" by construction. But a document where the crucial definition sits on page one breaks that assumption immediately. Removing softmax also removes the in-head nonlinearity, which flattens what a single head can express. And $QK^T \odot D$ maps poorly onto standard dense GEMM operations, so realizing the theoretical speed takes custom kernels.

The most important structural problem is that RetNet decides what to forget by age. The next mechanism asks whether the state has to forget at all, or whether it can keep accumulating, chunk by chunk, indefinitely.

---

5. Infini-attention

Munkhdalai et al. [7] answer with a two-tier design. The text is processed in fixed-size chunks. Within a chunk, the model runs standard dot-product attention at full resolution, softmax included. Across chunks, it maintains a compressive memory, a fixed-size $d \times d$ matrix that absorbs every chunk that came before. A learned gate then combines the local and the global signals.

The persistent state is a memory matrix $M_t \in \mathbb{R}^{d \times d}$ with normalization vector $z_t \in \mathbb{R}^d$:

Memory update (after processing chunk $t$):

$M_t = M_{t-1} + \sigma(K)^T V$

$z_t = z_{t-1} + \sum_i \sigma(K_i)^T$

where $\sigma(\cdot)$ is a non-negative activation (ELU + 1).

Global retrieval (for the current chunk's queries):

$A_\text{global} = \frac{\sigma(Q) M_{t-1}}{\sigma(Q) z_{t-1}^T}$

Combining:

$Y = \text{sigmoid}(\beta) \odot A_\text{local} + (1 - \text{sigmoid}(\beta)) \odot A_\text{global}$

with $\beta \in \mathbb{R}^d$ a learned gating vector. The memory update here is exactly the state accumulation of kernel linear attention from section 1a. So the architecture is softmax attention for the recent past and linear attention for everything else, joined together chunk by chunk.

Worked example

Chunk size $N = 4$, $d = 4$. Prior memory state:

$M_{t-1} = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \end{bmatrix}, \quad z_{t-1} = [2, 2, 2, 2]$

Current queries $Q = \begin{bmatrix} 1 & 0 & 1 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 1 & 0 & 0 \\ 0 & 0 & 1 & 1 \end{bmatrix}$

Since $Q$'s entries are already non-negative, $\sigma(Q) = Q + 1$:

$\sigma(Q) = \begin{bmatrix} 2 & 1 & 2 & 1 \\ 1 & 2 & 1 & 2 \\ 2 & 2 & 1 & 1 \\ 1 & 1 & 2 & 2 \end{bmatrix}$

Global retrieval numerator ($\sigma(Q) \times M_{t-1}$):

$= \begin{bmatrix} 4 & 2 & 4 & 2 \\ 2 & 4 & 2 & 4 \\ 3 & 3 & 3 & 3 \\ 3 & 3 & 3 & 3 \end{bmatrix}$

Denominator ($\sigma(Q) \times z_{t-1}^T = [12, 12, 12, 12]$).

$A_\text{global} = \begin{bmatrix} 0.333 & 0.167 & 0.333 & 0.167 \\ 0.167 & 0.333 & 0.167 & 0.333 \\ 0.25 & 0.25 & 0.25 & 0.25 \\ 0.25 & 0.25 & 0.25 & 0.25 \end{bmatrix}$

The memory stays $d \times d = 4 \times 4$ no matter how many chunks have passed through it, so we get constant memory for unbounded context.

The headline results were aimed directly at the long-context frontier [7]:

The strong number hides an asymmetry. Near the end of a 1M-token input, the local chunk still reaches the passkey directly, so the compressive memory barely has to do any work. Near the start or the middle, the local window is long gone and the compressive memory is the only thing being tested, and that is exactly where the accuracy collapses.

Infini-attention remains research-stage, and no production LLM has adopted it. The reasons are clear from the design. Compressing hundreds of thousands of tokens into one $d \times d$ matrix produces contextual blurring. The memory holds themes well but loses precise details, so finding one specific fact in a long document gets harder as the document grows. The linear associative retrieval lacks softmax's sharpness and smooths over precise details. And the whole system depends on the gate $\beta$ learning to balance local and global information well. If it is trained badly, the gate ends up ignoring one of the two pathways.

None of this makes the core idea wrong. A compressive recurrent state really can extend a linear kernel to unbounded context, and Infini-attention did establish that much. What it was not built for is raw throughput. MiniMax's Lightning Attention takes the same recurrent-state idea and builds it for that.

---

6. MiniMax Lightning Attention

Lightning Attention [8] restructures RetNet's decay mechanism around one performance question, which parts of the computation can run in parallel and which must be sequential? Its answer splits the sequence into blocks of size $B$, and from here $i$ indexes a block (the $i$-th chunk of $B$ tokens), not a single token as it did earlier in this post. Within a block, the model computes a causally masked, decay-weighted $B \times B$ score matrix in parallel, the same $QK^T \odot D$ structure RetNet uses, just confined to one block at a time. Between blocks, it passes a recurrent state $S_i \in \mathbb{R}^{d_k \times d_v}$ forward, the same running state as before, now updated once per block instead of once per token:

$O_i = (Q_i K_i^T \odot M) V_i + \Lambda Q_i S_{i-1}$

$S_i = \lambda^B S_{i-1} + (\lambda^B \Lambda^{-1} K_i)^T V_i$

where $M_{st} = \lambda^{s-t}$ for $s \geq t$ and $0$ otherwise is the intra-block causal-decay mask, and $\Lambda = \text{diag}(\lambda, \lambda^2, \ldots, \lambda^B)$ corrects for each token's position inside the block. Like RetNet, there is no softmax and no normalizing denominator here, the decay term keeps the values bounded on its own. The decay $\lambda$ plays the role of RetNet's $\gamma$, applied twice over, once per token inside a block through $\Lambda$, and once per whole block between blocks through $\lambda^B$. The structure is a compromise with the hardware. Pure recurrence is too sequential for GPUs, a full parallel form is quadratic, and the blockwise split keeps most of the parallelism at linear cost.

At its release, this was the mechanism carrying the longest production context on record. MiniMax-01, a 456B-parameter MoE model, supports 4 million tokens [8, 9]. But there is one architectural detail that matters most for this series. MiniMax-01 does not run on Lightning Attention alone. Its layers interleave Lightning Attention with standard softmax attention, one softmax layer for every seven Lightning Attention layers [9]. This is a production-scale admission, built into the architecture itself, that linear attention cannot yet carry retrieval quality on its own. The same pattern comes back throughout Part 6, linear layers for the bulk of the context, and softmax layers for the sharpness.

But one problem remains, and Lightning Attention shares it with RetNet, Infini-attention, and the original kernel formulation. The state only ever adds. New key-value pairs pile on top of old ones, and over a long enough sequence, unrelated pieces of information overlap and blur inside the fixed-size state. Decay fades everything at the same rate, whether it is old or new, and accumulation never fades anything at all. Neither one can replace an old fact with a fresh one.

---

7. Gated DeltaNet / Delta Attention

Yang et al. [10] fix this blurring, using a rule that is older than deep learning. $S_t \in \mathbb{R}^{d_k \times d_v}$ is the same running state matrix from kernel linear attention and RetNet, just indexed by token $t$ instead of $n$. Instead of only ever accumulating new key-value pairs, treat each new pair as a correction to a prediction the state is already making. Define a loss for the current token, how far the state's current guess is from the truth:

$L_t(S) = \tfrac{1}{2}\|S_{t-1}k_t - v_t\|^2$

$S_{t-1}k_t$ is what the state predicts for key $k_t$ right now, so $L_t$ is small when the state already knows this key's value and large when it doesn't. Take one step of gradient descent on $L_t$ with respect to $S$, step size $\beta_t$, and the update falls straight out:

$S_t = S_{t-1} - \beta_t \nabla_S L_t\big|_{S_{t-1}} = S_{t-1} + \beta_t (v_t - S_{t-1} k_t) k_t^T$

Key result: the delta rule is not a hand-designed heuristic, it is exactly one step of online gradient descent on the per-token loss $L_t(S) = \tfrac12\|S_{t-1}k_t - v_t\|^2$, with $\beta_t$ playing the role of the learning rate.

The term $(v_t - S_{t-1} k_t)$ is the delta, the new value minus the predicted value. If the state already stores the right value, the delta is zero and nothing changes. If the key's value has changed, the old value gets overwritten, not discarded. This is the classical delta rule of content-addressable memory, brought into the linear-attention state.

Worked example: take $d = 2$ and a one key $k_t = [1, 0]^T$, so $S_{t-1}k_t$ just reads off the first column of the state. Say $S_{t-1} = \begin{bmatrix} 3 & 0 \\ 1 & 0 \end{bmatrix}$, so the state currently returns $[3,1]^T$ for this key, and a new value $v_t = [5,2]^T$ arrives. The delta is $[5,2]^T - [3,1]^T = [2,1]^T$. With $\beta_t = 1$, $S_t = S_{t-1} + [2,1]^T k_t^T = \begin{bmatrix} 5 & 0 \\ 2 & 0 \end{bmatrix}$. Querying the new state with the same key gives $S_t k_t = [5,2]^T$, exactly $v_t$, and the second column, the memory for every other key, was never touched.

Gated DeltaNet, the full version, adds a scalar forget gate on top:

$S_t = \alpha_t S_{t-1} + \beta_t (v_t - \alpha_t S_{t-1} k_t) k_t^T$

with input-dependent scalar $\alpha_t \in (0, 1)$ and update strength $\beta_t \in [0, 1]$. The output is $o_t = S_t q_t$.

Expand the plain delta update (the $\alpha_t = 1$ case) and it splits into an erase term and a write term:

$S_t = (I - \beta_t k_t k_t^T) S_{t-1} + \beta_t v_t k_t^T$

The first term erases whatever the state held along $k_t$'s direction, the second writes the new value back in. That is the sense in which the delta rule really overwrites rather than accumulates. But it only overwrites when the same key comes back around. A key that simply goes stale, one that's never queried again, is never touched by this term, nothing in the rule decays it on its own. That gap is exactly what the gate closes: $\alpha_t$ shrinks the whole state before the delta step runs, so memory fades even for keys the delta term never revisits. The state now has the two operations any working memory needs, a targeted overwrite from the delta rule and a graduated forgetting from the gate, where RetNet had only uniform decay.

Kimi Linear's Kimi Delta Attention (KDA) [13] extends Gated DeltaNet with a finer-grained, channel-wise gate in place of the per-head scalar gate above, a distinct variant rather than a straight reuse.

On the adoption side, Kimi Linear [13] runs a layerwise hybrid of KDA and Multi-Head Latent Attention. Qwen3-Next also shipped Gated DeltaNet itself, in a 3:1 hybrid layout that interleaves Gated DeltaNet layers for linear-attention efficiency with full (Gated) attention layers for high-fidelity reasoning [14].

The delta rule's cost is built into its definition. Computing $S_{t-1} k_t$ at each step is a state-key product that scales with the state size, and for large states it becomes the bottleneck. The mechanism is also very new (2025), and its training stability at frontier scale is still being established.

---

What comes next

In this part, every mechanism its linear cost by giving something up. Kernel methods lose the sharpness of the exponential. cosFormer and RetNet failed due to its fixed decay mechanism. Infini-attention and Lightning Attention blur exact retrieval into a compressed state. And Gated DeltaNet's overwrite rule slows the blurring without eliminating it. The pattern across scales has been consistent, linear attention lags softmax on perplexity, and it lags even more on pinpointing one fact in a long context. As of this writing, no frontier model runs on linear attention alone, every production deployment interleaves it with real softmax layers.

The next part covers how the field actually resolved this impasse, from two directions at once. DeepSeek's Multi-Head Latent Attention keeps softmax's full sharpness and compresses the cache instead of the computation. The Differential Transformer keeps softmax and improves its signal-to-noise ratio directly. And the closing comparison table shows what the frontier models of 2025–2026 actually ship, mechanism by mechanism.

Continue to Part 6: Where Attention Stands Today →

---

References

[1] Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML 2020. arXiv:2006.16236.

[2] Choromanski, K., Likhosherstov, V., Dohan, D., et al. (2020). Rethinking Attention with Performers. ICLR 2021. arXiv:2009.14794.

[3] Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020). Linformer: Self-Attention with Linear Complexity. arXiv:2006.04768.

[4] Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020). Synthesizer: Rethinking Self-Attention in Transformer Models. ICML 2021. arXiv:2005.00743.

[5] Qin, Z., Sun, W., Deng, H., Li, D., Wei, Y., Lv, B., Yan, J., Kong, L., & Zhong, Y. (2022). cosFormer: Rethinking Softmax in Attention. ICLR 2022. arXiv:2202.08791.

[6] Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., & Wei, F. (2023). Retentive Network: A Successor to Transformer for Large Language Models. arXiv:2307.08621.

[7] Munkhdalai, T., Faruqui, M., & Gopal, S. (2024). Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. arXiv:2404.07143.

[8] Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., & Zhong, Y. (2024). Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models. arXiv:2401.04658.

[9] MiniMax (2025). MiniMax-01: Scaling Foundation Models with Lightning Attention. arXiv:2501.08313.

[10] Yang, S., Kautz, J., & Hatamizadeh, A. (2024). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464.

[11] Gu, A., & Dao, T. (2023). Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.

[12] Dao, T., & Gu, A. (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality. ICML 2024. arXiv:2405.21060.

[13] Kimi Team (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692.

[14] Qwen Team (2025). Qwen3-Next-80B-A3B-Instruct (Model Card). Hugging Face. https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct.