Today, we bring you a summary of our series about transformer alternatives.

TheSequence is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe

For the better part of a decade, the entire field has been a giant, spectacularly funded wrapper around a single operation: self-attention. The Transformer didn’t win because it was the most elegant or the most brain-like design. It won because it had the best scaling story and it won the hardware lottery. Every token looks at every other token, the whole thing maps cleanly onto a GPU grid, and you train it all at once. Add data, parameters, compute, context — and the loss curve cooperates. That smoothness is rare. Most clever ideas in deep learning never become industrial. This one did.

But the tax was always there in plain sight. Self-attention buys you something genuinely valuable — perfect, lossless recall over the entire context, with every token able to address every other token directly, and a training pass that parallelizes across the whole sequence at once. That’s the benefit, and it’s a real one. The cost is that attention scales quadratically with sequence length, and autoregressive decoding drags around a KV-cache that grows linearly with every token you’ve already seen. When you’re pushing past a million tokens, or watching a 70B model’s cache eat 40GB of VRAM, O(n²) compute and O(n) memory stop being footnotes and become the actual bill. So the interesting question was never “are Transformers good?” They’re spectacular. The question is whether they’re the final architecture or just the first truly scalable one — soon to be absorbed into something richer.

That was the thesis we set out to test, and the cleanest way to read the eight issues is as four families, each making a different bet against attention.

The first family is recurrent and linear-recurrent models — the RNN comeback and xLSTM. Their pitch is constant memory: instead of a cache that grows forever, they carry a fixed-size hidden state and pay O(n) compute over a sequence rather than O(n²). The classic objection was that RNNs train serially and can’t saturate a GPU, but the modern variants reformulate the recurrence so it parallelizes during training while staying cheap at inference. The benefit is brutally efficient generation; the open challenge is whether a fixed-size state can hold enough to match attention’s exact recall on long-range, retrieval-heavy tasks.

The second family is state space models — the SSM/Mamba line, the most serious challenger of the bunch. SSMs treat a sequence as a continuous linear dynamical system, which gives them a near-magical dual form: a parallelizable convolution for training and a recurrent scan for inference. They get linear scaling and long-context handling almost for free. The trade-off is expressivity — pure SSMs can struggle with precise in-context copying and lookup, which is exactly why the strongest results today are hybrids that interleave a few attention layers among many SSM layers.

The third family is text diffusion — generation that abandons left-to-right decoding entirely, refining a whole sequence in parallel over a handful of denoising steps. The benefit is non-autoregressive speed and bidirectional context at generation time; the challenge is matching the raw quality and controllability of autoregressive models, which LLaDA, Gemini Diffusion, and Mercury are now pushing on hard.

The fourth family is liquid and continuous-time models, which throw out the parallel-lookup mental model altogether in favor of dynamics that evolve continuously in time, aiming for far smaller, more adaptive networks. The benefit is parameter efficiency and a different inductive bias; the challenge is scaling that story to frontier sizes.

None of these has dethroned attention. But the monoculture is over, and the most likely future is explicitly hybrid: attention where exact recall earns its quadratic cost, something linear-time everywhere else.

TheSequence is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Subscribe

Here is the full series, in order:

• #846 — Beyond Transformer: A New Series — The kickoff, framing the palpable vibe shift on arXiv toward post-attention architectures and the decade we’ve spent as a wrapper around self-attention. It lays out the plan to map every major viable alternative to the Transformer.
• #850 — The Unexpected Comeback of RNNs — The case for recurrent networks as the alternative most people overlooked, revisiting why linear-time recurrence is attractive again. It positions modern RNN variants as a serious challenger rather than a relic.
• #854 — Return of the King: Unrolling the xLSTM Architecture — Traces the lineage from the 1990s LSTM through the 2017 Transformer pivot into xLSTM, the modernized revival of Hochreiter and Schmidhuber’s design. It explains how reworked gating and scaling let xLSTM compete with attention-based models.
• #858 — How State Space Models Went from Curiosity to Serious Transformer Competitor — Charts the rise of SSMs as the O(n²) attention bottleneck becomes a real constraint at million-token contexts and large KV-caches. It argues state space models have quietly matured into a genuine rival to the dominant paradigm.