Distribution Matching Is Not Enough: Two Failure Modes in Latent Text Drifting

The Drifting paper (Deng et al., 2026, arXiv:2602.04770) introduced a one-step generative framework built on a kernel attraction-repulsion field over the data distribution. The training loss drives this field to zero; at equilibrium, the generator maps noise to data in a single forward pass. The headline result is state-of-the-art image generation on ImageNet at one network function evaluation, beating multi-step diffusion baselines.

We tried to adapt the method to text. The natural pipeline is two stages: an autoencoder from tokens to a continuous latent space, and a generator that maps noise into that latent space using a kernel-based drifting loss, composed at inference with the frozen autoencoder's decoder. The autoencoder converged easily. The generator went through two attempts — the first stabilized only after a round of debugging, the second was stable from the start. Decoded text in both was punctuation, hyphens, and repeated stop words.

What follows is a technical postmortem. The failure decomposes into two structurally distinct mechanisms — a trainable kernel that trivializes its own loss, and an aggregate matching metric saturated by mode collapse — both sitting on top of a deeper issue: the unpaired distribution-matching objectives we used leave a low-energy basin near the manifold mean unconstrained, and our loss had no pairing constraint to rule it out.

---

The setup

Formally, given a target distribution $p$ and a generator distribution $q_\theta = f_\theta \,\#\, \mathcal{N}(0, I)$, the drifting field is

$$V_{p, q}(x) \;=\; \frac{1}{Z_p Z_q}\, \mathbb{E}_{y^+ \sim p,\; y^- \sim q}\!\left[\, k(x, y^+)\, k(x, y^-)\, (y^+ - y^-) \,\right]$$

with kernel $k(\cdot, \cdot)$ (multi-scale RBF). The training loss

$$\mathcal{L}(\theta) \;=\; \mathbb{E}_{\varepsilon \sim \mathcal{N}}\!\left[\, \big\|\, f_\theta(\varepsilon) - \mathrm{sg}\!\big(\, f_\theta(\varepsilon) + V_{p, q_\theta}(f_\theta(\varepsilon)) \,\big) \,\big\|^2 \,\right]$$

drives $V \to 0$, with $V \equiv 0$ at $p = q_\theta$ under the paper's anti-symmetry condition (swapping $p$ and $q$ flips the sign of the field, so $V_{p,p} \equiv 0$). For text, $f_\theta : \mathbb{R}^{128 \times 256} \to \mathbb{R}^{128 \times 256}$ takes a noise sample $\varepsilon \sim \mathcal{N}(0, I)$ and produces a latent; decoded text comes from passing that latent through the frozen Stage 1 decoder.

Stage 1. A frozen-BERT-encoder + projection + transformer-decoder autoencoder mapping 128-token sequences to an $\mathbb{R}^{128 \times 256}$ latent. It converged easily — 99.89% per-token reconstruction on held-out OpenWebText, 88.44% exact-match at 128 tokens, latent statistics tight ($\mathbb{E}[z] \approx 0$, $\mathrm{std}(z) \approx 0.47$). Stage 1 was not the bottleneck.

Stage 2. A DiT-style generator trained with a hierarchical kernel-based drifting loss to map noise into the same latent space. The drifting loss does not factor naturally over a 128-token sequence, so we used the obvious decomposition into terms operating at different scales of representation:

$$\mathcal{L}_{\text{S2}}(\theta, \psi, \xi) \;=\; \alpha\, \mathcal{L}_{\text{tok}} \;+\; \beta\, \mathcal{L}_{\text{win}} \;+\; \gamma\, \mathcal{L}_{\text{seq}} \;+\; \delta\, \mathcal{L}_{\text{anchor}} \;+\; \eta\, \mathcal{L}_{\text{disc}}$$

The first three terms ($\mathcal{L}_{\text{tok}}, \mathcal{L}_{\text{win}}, \mathcal{L}_{\text{seq}}$) are kernel-based drifting losses operating in a feature space $\phi_\psi : \mathbb{R}^{256} \to \mathbb{R}^{32}$ — originally learnable, jointly trained with the generator parameters $\theta$ — at three scales: per-position (token), 8-token sliding window, and full-sequence attention pool. The anchor matches batch-level moments — means, variances, average norms, per-position deltas — directly in raw latent space. The discriminator $D_\xi$ is a Transformer reading the full 128-position sequence in raw latent space, providing an adversarial signal once activated.

We will also need a simple raw-space diagnostic. Define the drift between two batches of latents $\mathcal{Z}_a, \mathcal{Z}_b \subset \mathbb{R}^{128 \times 256}$ as

$$d(\mathcal{Z}_a, \mathcal{Z}_b) \;=\; \mathbb{E}_l\!\left[\, \Big\|\, \frac{1}{|\mathcal{Z}_a|}\sum_{z \in \mathcal{Z}_a} z_l \;-\; \frac{1}{|\mathcal{Z}_b|}\sum_{z \in \mathcal{Z}_b} z_l \,\Big\| \,\right]$$

— the L2 distance between per-position batch means, averaged over positions $l$. This is a first-moment statistic. We track $d(\mathcal{Z}_{\text{gen}}, \mathcal{Z}_{\text{real}})$ as the gen-vs-real drift, and a floor $d_{\text{floor}}$ obtained by splitting a real batch in half and computing the same quantity between the halves. Both failure modes interact with this metric directly.

The promise of the Stage 2 stack: at convergence, $q_\theta$ matches $p$ at multiple scales of representation, and a sequence-level critic catches anything the kernels miss. The actual outcome unfolded as two distinct failure modes in sequence.

Failure 1: the trainable kernel becomes a mirror

The first thing the training run did was drive $\mathcal{L}_{\text{tok}}, \mathcal{L}_{\text{win}}, \mathcal{L}_{\text{seq}} \to 0$ within a few thousand steps, while $d(\mathcal{Z}_{\text{gen}}, \mathcal{Z}_{\text{real}})$ sat between 0.5 and 1.8 against $d_{\text{floor}} \approx 0.65$. The kernel losses were satisfied; the underlying samples were not aligned. When the discriminator activated on raw latents, it found near-perfect separability, and the adversarial gradient exploded. Three instruments, three different stories.

The mechanism is not subtle. Consider the per-position term

$$\mathcal{L}_{\text{tok}}(\theta, \psi) \;=\; \mathbb{E}\!\left[\, \big\|\, \phi_\psi(z_{\text{gen}}) - \mathrm{sg}\!\big(\, \phi_\psi(z_{\text{gen}}) + V^\phi(\phi_\psi(z_{\text{gen}})) \,\big) \,\big\|^2 \,\right]$$

with $V^\phi$ the drifting field computed on $\phi$-features. Joint optimization in $(\theta, \psi)$ has access to two qualitatively different descent directions:

1. Match in raw space. Move $\theta$ so that $z_{\text{gen}} \to z_{\text{real}}$ in $\mathbb{R}^{256}$; $\phi$-feature alignment follows.
2. Match in feature space. Move $\psi$ so that $\phi_\psi(z_{\text{gen}}) \to \phi_\psi(z_{\text{real}})$ for any $z_{\text{gen}}, z_{\text{real}}$, regardless of raw-space distance.

Both directions reduce $\mathcal{L}_{\text{tok}}$. The optimizer has no preference between them. Direction (2) is cheaper — $\psi$ has thousands of parameters dedicated to nothing else, while $\theta$ has the much harder job of producing real-data-looking samples from noise. The cheap descent direction wins. In the limit, $\phi_\psi$ stops being a discriminative projection and becomes a mirror: it maps any input into a region of feature space where the kernel cannot tell samples apart. The proxy loss has been satisfied. The underlying distributions remain mismatched. Anything downstream that reads raw latents — including the frozen decoder — produces garbage.

First recovery: freezing the kernel

The fix to failure mode 1 is structural: $\phi$ must be data-independent. A Johnson–Lindenstrauss-style frozen random projection $\phi(z) = Wz$ with $W \in \mathbb{R}^{256 \times 32}$ orthogonal-columned and fixed satisfies the distance-preservation guarantee

$$(1 - \epsilon)\, \|z_a - z_b\|^2 \;\le\; \|\phi(z_a) - \phi(z_b)\|^2 \;\le\; (1 + \epsilon)\, \|z_a - z_b\|^2$$

with high probability, for $d_\phi = O(\epsilon^{-2} \log n)$. Kernel matching in this feature space implies (approximate) matching in raw space. A trainable $\phi$ does not satisfy this; a frozen random $\phi$ does. Lesson: anything you train alongside the generator and use to grade the generator will be optimized to grade leniently. Freeze the judge, or train it on a structurally separated objective.

The freeze alone wasn't enough to recover the run. Three additional changes turned out to be needed.

Raw latent manifold anchor. Direct pressure on the generator's outputs to look statistically like real latents in raw space, independent of any kernel: a four-term anchor matching means, standard deviations, average norms, and per-position deltas, weighted at parity with the per-position kernel term in the total loss. The generator now had a reason to stay on the manifold the decoder was trained against. Lesson: distributional supervision should live in the same space the decoder reads from, not in a transformed proxy space.

Gated, ramped discriminator. Instead of activating at a fixed step count, the critic waited until the drift ratio $d(\mathcal{Z}_{\text{gen}}, \mathcal{Z}_{\text{real}})/d_{\text{floor}}$ dropped below 2. Once active, the adversarial weight ramped in across thousands of steps. Activating an adversarial signal at a fixed step had caused immediate collapse: the critic activated when the generator was still off-manifold, found a trivial decision boundary, and the adversarial gradient exploded. Lesson: gate adversarial signals on real-space readiness, not on training-step elapsed.

Hinge loss with R1 penalty. Binary cross-entropy (BCE) has the form $-\log \sigma(D)$ for the generator, with loss value unbounded as $D \to -\infty$ — every confidently-fake sample makes a large contribution to the total loss. Hinge has the form $-D$, linear in $D$. With spectral normalization on the input/output linears and an R1 input-gradient penalty on real samples (a regularizer that bounds the discriminator's gradient norm on real inputs; Mescheder et al. 2018), the gradient is doubly stabilized. Lesson: swap the loss; do not work around BCE's pathologies with weight schedules.

After all four changes — the freeze plus the three above — the run looked beautiful. Gen-vs-real drift settled to 0.48–0.55, slightly below $d_{\text{floor}} \approx 0.65$. The discriminator stayed in equilibrium with the generator. The adversarial weight stabilized around 0.02. Gradient norms were normal. The loss curves were boring in the way that good training is boring.

Failure 2: stable, but bad

Decoded samples at 10,000 steps were the same shape of garbage as before — high-frequency tokens, no semantic structure, no grammar. By every visible metric the run was healthy. The decoded outputs said the model had learned nothing.

The clue was sitting in the drift number we had been celebrating. The metric $d$ defined earlier is a first-moment statistic — only mean alignment, nothing higher-order — and it has a specific gameability. If the generator's outputs are concentrated in a small region around some point $\mu^*$ (in the strict limit, $z \equiv \mu^*$), then

$$\frac{1}{|\mathcal{Z}_{\text{gen}}|} \sum_{z \in \mathcal{Z}_{\text{gen}}} z_l \;\approx\; \mu^*_l$$

regardless of the spread within $\mathcal{Z}_{\text{gen}}$. Combined with the raw-space anchor — which actively pushes the generator's per-dim mean toward $\mathbb{E}_p[z]$ — this gives $\mu^* \approx \mathbb{E}_p[z]$ and $d(\mathcal{Z}_{\text{gen}}, \mathcal{Z}_{\text{real}}) \to 0$ even though the generator's variance has collapsed. The metric does not measure variance, pairwise distance, or any higher-order distributional property. Mode collapse to a small region near the population mean trivially saturates it.

The 0.48 < 0.65 reading — generator beating the natural noise floor of the data — is the signature of exactly this collapse. Two halves of a real batch differ from each other by 0.65 because the batch-mean estimator has finite-sample variability over the 256-sample halves; a degenerate generator with collapsed variance has a much tighter batch-mean estimator and aligns with $\mu^* \approx \mathbb{E}_p[z]$ effortlessly. The frozen decoder, fed latents stripped of their per-sample variability, produces a token distribution dominated by the highest-frequency tokens of the corpus — function words, punctuation, the trivial fillers a language model defaults to when its input carries almost no information. Which is what we observed.

The instrument was being satisfied by exactly the failure mode it should have been built to flag.

The deeper learning: no pairing constraint

Both failures share a structural cause. Each term in $\mathcal{L}_{\text{S2}}$ has the form

$$\mathcal{L}_*(\theta) \;=\; \mathcal{D}\!\left(\, \pi_*[q_\theta] \;\big\|\; \pi_*[p] \,\right)$$

for some statistical divergence $\mathcal{D}$ (kernel-based MMD-like for the drifting terms, MSE on moments for the anchor, Wasserstein-like for the hinge adversarial term) and some projection $\pi_*$ of the distributions onto a representation. None of these terms involves a coupling between specific noise inputs and specific token sequences. They constrain marginals (or marginals of projections); they do not constrain joint targets.

Concretely: the Stage 2 objective at full convergence guarantees, in the best case, $q_\theta = p$ as distributions over $\mathbb{R}^{128 \times 256}$. It does not guarantee that any particular $\varepsilon \sim \mathcal{N}$ produces a $z_\theta(\varepsilon)$ whose decoded text is meaningful. A constant generator $f_\theta(\varepsilon) \equiv \mu^*$ has $q_\theta = \delta_{\mu^*}$, which is a terrible distribution-matcher of $p$ in the strict sense — but the finite-sample, low-moment, projection-based matching tests we built lack the resolution to distinguish $\delta_{\mu^*}$ from $p$. What we observed was the empirical signature of that resolution gap.

The pairing constraint that would rule out the collapse has the form

$$z_\theta(\varepsilon^{(i)}) \;\approx\; z_{\text{real}}^{(i)} \quad \text{for an explicit assignment } \varepsilon^{(i)} \mapsto z_{\text{real}}^{(i)}.$$

Stage 1 produces a cache of real latents, one per training example. The unpaired drifting objective never references this assignment; it treats the generated and real samples as independent draws from $q_\theta$ and $p$ respectively. The signal for which generated latent should match which real latent is simply absent from the loss.

This is a classical failure pattern in GANs and other unpaired distribution-matching generators. We are not claiming a discovery. What we are noting is that the pattern shows up cleanly and stubbornly in this specific adaptation, and that the standard suite of training-time instruments will not warn you about it. They will, in practice, do something worse: they will reassure you while it happens.

Future directions

The path forward replaces unpaired drifting with paired rectified flow (a linear-interpolation form of flow matching; Liu et al. 2022, Lipman et al. 2023). Each training example becomes a pair $(\text{tokens}_i, z_{\text{real}}^{(i)})$ from the Stage 1 cache, defining a straight noise-to-data path

$$x_t = (1 - t)\, \varepsilon + t\, z_{\text{real}}^{(i)}, \qquad v^* = z_{\text{real}}^{(i)} - \varepsilon, \qquad t \sim \mathcal{U}(0, 1).$$

The model predicts the velocity $v_\theta(x_t, t)$, and the recovered clean latent is

$$z_\theta = x_t + (1 - t)\, v_\theta(x_t, t).$$

The total loss adds a cross-entropy term on the frozen decoder's output:

$$\mathcal{L}_{\text{teacher}} \;=\; \lambda_v\, \|v_\theta - v^*\|^2 \;+\; \lambda_z\, \|z_\theta - z_{\text{real}}^{(i)}\|^2 \;+\; \lambda_{\text{ce}}\, \mathrm{CE}\!\big(\, p_{\text{dec}}(\,\cdot \mid z_\theta),\, \text{tokens}_i \,\big) \;+\; \cdots$$

(plus structural latent matching at multiple segment lengths and a short differentiable rollout). Two properties of this objective do what the unpaired drifting loss could not.

First, it pairs. Every $\varepsilon$ has an explicit target $z_{\text{real}}^{(i)}$. The constant-generator escape route is closed by construction.

Second, the cross-entropy term is the only loss component that propagates a cross-position signal. The decoder is a Transformer with full sequence self-attention; its prediction at position $l$ depends on every $z_\theta^{(l')}$, $l' \in [1, L]$, via attention. A generated latent that is locally plausible at each position but globally incoherent will receive different gradients at different positions, depending on the joint sequence of generated latents. The flow MSE, the latent MSE, the structural latent terms, and the manifold anchor are all per-element or per-window factorized.

This is not a magical property of CE; it is a property of routing the loss through a model with global attention over the sequence. The decoder is already trained, frozen, and known-good — and provides exactly the joint signal the unpaired drifting loss lacks. Beyond the teacher itself, the next step is distillation to a one-step student that retains this cross-position signal. An earlier version of the teacher — without rollout supervision and without structural latent terms — ran stably for 41,000 steps and still produced word salad, which is why the current version has both. Whether the strengthened teacher actually decodes coherent text at scale, and whether it distills cleanly to a single step, are the open empirical questions.

The lesson we are carrying out of this is structural rather than tactical. Unpaired distribution matching, in any of its kernel-based, moment-based, or adversarial forms, is too weak a signal for sequence generation: the matching constraints leave a low-energy basin near the manifold mean that satisfies the loss while producing nothing. The failure shows up cleanly as mode collapse to that mean. The standard suite of training-time instruments — kernel losses, anchor losses, drift estimates, discriminator equilibrium — will not warn you about it. The fix lives in the loss function rather than in the optimization, and it specifically requires either a pairing constraint, a sequence-level joint term routed through a model with global attention, or both.

What broke our intuition was not the optimizer or the architecture or the engineering. It was that the loss we had chosen to optimize was satisfied at a point where the model was useless, and every instrument we had pointed at it agreed. The instruments were faithful to the loss. The loss was unfaithful to the goal.

---

Related reading

• Probing Latent Directions in Video Diffusion Models
• Diffusion vs Autoregression: How Language Models Think