\n
![](\"https:\/\/images.ctfassets.net\/jdtwqhzvc2n1\/4w8pJoJCpKW8g1eJxxgy3f\/c8d0f3a8431956228510e551a2b474f1\/aRNZNKxpXqqt3S_iXlQfh_71e40940f61b4bf89d6b5f5cbeafd63e.png?w=300&q=30\")
<\/p>\n
\n<\/p>\n
When the transformer architecture was introduced in 2017 in the now seminal Google paper "Attention Is All You Need," it became an instant cornerstone of modern artificial intelligence. <\/p>\n
Every major large language model (LLM) \u2014 from OpenAI's GPT series to Anthropic's Claude, Google's Gemini, and Meta's Llama \u2014 has been built on some variation of its central mechanism: attention<\/b>, the mathematical operation that allows a model to look back across its entire input and decide what information matters most.<\/p>\n
Eight years later, the same mechanism that defined AI\u2019s golden age is now showing its limits. Attention is powerful, but it is also expensive \u2014 its computational and memory costs scale quadratically with context length, creating an increasingly unsustainable bottleneck for both research and industry. As models aim to reason across documents, codebases, or video streams lasting hours or days, attention becomes the architecture\u2019s Achilles\u2019 heel.<\/p>\n
On October 28, 2025, the little-known AI startup Manifest AI introduced a radical alternative. Their new model, Brumby-14B-Base<\/b>, is a retrained variant of Qwen3-14B-Base<\/b>, one of the leading open-source transformer models.<\/p>\n
But while many variants of Qwen have been trained already, Brumby-14B-Base is novel in that it abandons attention altogether. <\/p>\n
Instead, Brumby replaces those layers with a novel mechanism called Power Retention<\/b>\u2014a recurrent, hardware-efficient architecture that stores and updates information over arbitrarily long contexts without the exponential memory growth of attention.<\/p>\n
Trained at a stated cost of just $4,000, the 14-billion-parameter Brumby model performs on par with established transformer models like Qwen3-14B and GLM-4.5-Air, achieving near-state-of-the-art accuracy on a range of reasoning and comprehension benchmarks.<\/p>\n
The core of Manifest AI\u2019s innovation lies in what they call the Power Retention layer. <\/p>\n
In a traditional transformer, every token computes a set of queries (Q), keys (K), and values (V), then performs a matrix operation that measures the similarity between every token and every other token\u2014essentially a full pairwise comparison across the sequence. <\/p>\n
This is what gives attention its flexibility, but also what makes it so costly: processing a sequence twice as long takes roughly four times the compute and memory.<\/p>\n
Power Retention keeps the same inputs (Q, K, V), but replaces the global similarity operation with a recurrent state update. <\/p>\n
Each layer maintains a memory matrix S, which is updated at each time step according to the incoming key, value, and a learned gating signal. <\/p>\n
The process looks more like an RNN (Recurrent Neural Network) than a transformer: instead of recomputing attention over the entire context, the model continuously compresses past information into a fixed-size latent state.<\/p>\n
This means the computational cost of Power Retention does not grow with context length<\/b>. Whether the model is processing 1,000 or 1,000,000 tokens, the per-token cost remains constant.<\/b> <\/p>\n
That property alone\u2014constant-time per-token computation\u2014marks a profound departure from transformer behavior.<\/p>\n
At the same time, Power Retention preserves the expressive power that made attention successful. Because the recurrence involves tensor powers of the input (hence the name \u201cpower retention\u201d), it can represent higher-order dependencies between past and present tokens. <\/p>\n
The result is an architecture that can theoretically retain long-term dependencies indefinitely, while remaining as efficient as an RNN and as expressive as a transformer.<\/p>\n
Perhaps the most striking aspect of Brumby-14B\u2019s training process is its efficiency. Manifest AI trained the model for only 60 hours on 32 Nvidia H100 GPUs, at a cost of roughly $4,000 \u2014 less than 2% of what a conventional model of this scale would cost to train from scratch.<\/p>\n
However, since it relied on a transformer-based model, it's safe to say that this advance alone will not end the transformer AI-era.<\/p>\n
As Jacob Buckman, founder of Manifest AI, clarified in an email to VentureBeat: \u201cThe ability to train for $4,000 is indeed only possible when leveraging an existing transformer model,\u201d he said. \u201cBrumby could not be trained from scratch for that price.\u201d<\/p>\n
Still, Buckman emphasized the significance of that result: \u201cThe reason this is important is that the ability to build on the weights of the previous generation of model architectures is a critical accelerant for the adoption of a new modeling paradigm.\u201d <\/p>\n
He argues this demonstrates how attention-free systems can catch up to transformer performance \u201cfor orders-of-magnitude less\u201d investment.<\/p>\n
In the loss curves released by Manifest AI, Brumby\u2019s training loss quickly converges to that of the Qwen3 baseline within 3,000 training steps, even as the architecture diverges significantly from its transformer origins. <\/p>\n
Although Brumby-14B-Base began life as Qwen3-14B-Base, it did not remain identical for long. Manifest AI fundamentally altered Qwen3\u2019s architecture by removing its attention layers\u2014the mathematical engine that defines how a transformer model processes information\u2014and replacing them with their new \u201cpower retention\u201d mechanism. This change restructured the model\u2019s internal wiring, effectively giving it a new brain while preserving much of its prior knowledge.<\/p>\n
Because of that architectural swap, the existing Qwen3 weights no longer fit perfectly. They were trained to operate within a transformer\u2019s attention dynamics, not the new retention-based system. As a result, the Brumby model initially \u201cforgot\u201d how to apply some of its learned knowledge effectively. The retraining process\u2014about 3,000 steps<\/b> of additional learning\u2014served to recalibrate those weights, aligning them with the power retention framework without having to start from zero.<\/p>\n
A helpful way to think about this is to imagine taking a world-class pianist and handing them a guitar. They already understand rhythm, harmony, and melody, but their hands must learn entirely new patterns to produce the same music. Similarly, Brumby had to relearn how to use its existing knowledge through a new computational instrument. Those 3,000 training steps were, in effect, its crash course in guitar lessons.<\/p>\n
By the end of this short retraining phase, Brumby had regained its full performance, reaching the same accuracy as the original Qwen3 model. That quick recovery is what makes the result so significant: it shows that an attention-free system can inherit and adapt the capabilities of a transformer model with only a fraction of the training time and cost.<\/p>\n
The benchmark progression plots show a similar trend: the model rapidly approaches its target accuracy on core evaluations like GSM8K, HellaSwag, and MMLU after only a few thousand steps, matching or even slightly surpassing Qwen3 on several tasks.<\/p>\n
Across standard evaluation tasks, Brumby-14B-Base<\/b> consistently performs at or near parity with transformer baselines of comparable scale.<\/p>\n Task<\/b><\/p>\n<\/td>\n Brumby-14B<\/b><\/p>\n<\/td>\n Qwen3-14B<\/b><\/p>\n<\/td>\n GLM-4.5-Air<\/b><\/p>\n<\/td>\n Nemotron Nano (12B)<\/b><\/p>\n<\/td>\n<\/tr>\n ARC<\/p>\n<\/td>\n 0.89<\/p>\n<\/td>\n 0.94<\/p>\n<\/td>\n 0.92<\/p>\n<\/td>\n 0.93<\/p>\n<\/td>\n<\/tr>\n GSM8K<\/p>\n<\/td>\n 0.88<\/p>\n<\/td>\n 0.84<\/p>\n<\/td>\n 0.83<\/p>\n<\/td>\n 0.84<\/p>\n<\/td>\n<\/tr>\n GSM8K (Platinum)<\/p>\n<\/td>\n 0.87<\/p>\n<\/td>\n 0.88<\/p>\n<\/td>\n 0.85<\/p>\n<\/td>\n 0.87<\/p>\n<\/td>\n<\/tr>\n HellaSwag<\/p>\n<\/td>\n 0.77<\/p>\n<\/td>\n 0.81<\/p>\n<\/td>\n 0.85<\/p>\n<\/td>\n 0.82<\/p>\n<\/td>\n<\/tr>\n MATH<\/p>\n<\/td>\n 0.62<\/p>\n<\/td>\n 0.54<\/p>\n<\/td>\n 0.47<\/p>\n<\/td>\n 0.26<\/p>\n<\/td>\n<\/tr>\n MBPP<\/p>\n<\/td>\n 0.57<\/p>\n<\/td>\n 0.75<\/p>\n<\/td>\n 0.73<\/p>\n<\/td>\n 0.71<\/p>\n<\/td>\n<\/tr>\n MMLU<\/p>\n<\/td>\n 0.71<\/p>\n<\/td>\n 0.78<\/p>\n<\/td>\n 0.77<\/p>\n<\/td>\n 0.78<\/p>\n<\/td>\n<\/tr>\n MMLU (Pro)<\/p>\n<\/td>\n 0.36<\/p>\n<\/td>\n 0.55<\/p>\n<\/td>\n 0.51<\/p>\n<\/td>\n 0.53<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n While it lags slightly behind transformers on knowledge-heavy evaluations like MMLU-Pro<\/b>, it matches or outperforms them on mathematical reasoning<\/b> and long-context reasoning<\/b> tasks\u2014precisely where attention architectures tend to falter. This pattern reinforces the idea that recurrent or retention-based systems may hold a structural advantage for reasoning over extended temporal or logical dependencies.<\/p>\n Brumby\u2019s power retention design offers another major advantage: hardware efficiency.<\/p>\n Because the state update involves only local matrix operations, inference can be implemented with linear complexity in sequence length. <\/p>\n Manifest AI reports that their fastest kernels, developed through their in-house CUDA framework Vidrial, can deliver hundreds-fold speedups over attention on very long contexts.<\/p>\n Buckman said the alpha-stage Power Retention kernels \u201cachieve typical hardware utilization of 80\u201385%, which is higher than FlashAttention2\u2019s 70\u201375% or Mamba\u2019s 50\u201360%.\u201d <\/p>\n (Mamba is another emerging \u201cpost-transformer\u201d architecture developed by Carnegie Mellon scientists back in 2023 that, like Power Retention, seeks to eliminate the computational bottleneck of attention. It replaces attention with a state-space<\/i> mechanism that processes sequences linearly \u2014 updating an internal state over time rather than comparing every token to every other one. This makes it far more efficient for long inputs, though it typically achieves lower hardware utilization than Power Retention in early tests.)<\/p>\n Both Power Retention and Mamba, he added, \u201cexpend meaningfully fewer total FLOPs than FlashAttention2 on long contexts, as well as far less memory.\u201d <\/p>\n According to Buckman, the reported 100\u00d7 speedup comes from this combined improvement in utilization and computational efficiency, though he noted that \u201cwe have not yet stress-tested it on production-scale workloads.\u201d<\/p>\n Perhaps no statistic in the Brumby release generated more attention than the training cost.<\/p>\n A 14-billion-parameter model, trained for $4,000, represents a two-order-of-magnitude reduction in the cost of foundation model development.<\/p>\n Buckman confirmed that the low cost reflects a broader scaling pattern. \u201cFar from diminishing returns, we have found that ease of retraining improves with scale,\u201d he said. \u201cThe number of steps required to successfully retrain a model decreases with its parameter count.\u201d <\/p>\n Manifest has not yet validated the cost of retraining models at 700B parameters, but Buckman projected a range of $10,000\u2013$20,000 for models of that magnitude\u2014still far below transformer training budgets.<\/p>\n He also reiterated that this approach could democratize large-scale experimentation by allowing smaller research groups or companies to retrain or repurpose existing transformer checkpoints without prohibitive compute costs.<\/p>\n According to Buckman, converting an existing transformer into a Power Retention model is designed to be simple. <\/p>\n \u201cIt is straightforward for any company that is already retraining, post-training, or fine-tuning open-source models,\u201d he said. \u201cSimply pip install retention, change one line of your architecture code, and resume training where you left off.\u201d<\/p>\n He added that after only a small number of GPU-hours, the model typically recovers its original performance\u2014at which point it gains the efficiency benefits of the attention-free design. <\/p>\n \u201cThe resulting architecture will permit far faster long-context training and inference than previously,\u201d Buckman noted.<\/p>\n On infrastructure, Buckman said the main Brumby kernels are written in Triton, compatible with both NVIDIA and AMD accelerators. Specialized CUDA kernels are also available through the team\u2019s in-house Vidrial framework. Integration with vLLM and other inference engines remains a work in progress: \u201cWe have not yet integrated Power Retention into inference engines, but doing so is a major ongoing initiative at Manifest.\u201d<\/p>\n As for distributed inference, Buckman dismissed concerns about instability: \u201cWe have not found this difficulty to be exacerbated in any way by our recurrent-state architecture. In fact, context-parallel training and GPU partitioning for multi-user inference both become significantly cleaner technically when using our approach.\u201d<\/p>\n Beyond the engineering details, Buckman also described Manifest\u2019s broader mission. \u201cOur mission is to train a neural network to model all human output,\u201d he said. <\/p>\n The team\u2019s goal, he explained, is to move beyond modeling \u201cartifacts of intelligence\u201d toward modeling \u201cthe intelligent processes that generated them.\u201d This shift, he argued, requires \u201cfundamentally rethinking\u201d how models are designed and trained\u2014work that Power Retention represents only the beginning of.<\/p>\n The Brumby-14B release, he said, is \u201cone step forward in a long march\u201d toward architectures that can model thought processes continuously and efficiently.<\/p>\n The launch of Brumby-14B sparked immediate discussion on X (formerly Twitter), where researchers debated the framing of Manifest AI\u2019s announcement. <\/p>\n Some, including Meta researcher Ariel (@redtachyon), argued that the \u201c$4,000 foundation model\u201d tagline was misleading, since the training involved reusing pretrained transformer weights rather than training from scratch.<\/p>\n \u201cThey shuffled around the weights of Qwen, fine-tuned it a bit, and called it \u2018training a foundation model for $4k,\u2019\u201d Ariel wrote.<\/p>\n Buckman responded publicly, clarifying that the initial tweet had been part of a longer thread explaining the retraining approach. \u201cIt\u2019s not like I was being deceptive about it,\u201d he wrote. \u201cI broke it up into separate tweets, and now everyone is mad about the first one.\u201d<\/p>\n In a follow-up email, Buckman took a measured view of the controversy. \u201cThe end of the transformer era is not yet here,\u201d he reiterated, \u201cbut the march has begun.\u201d <\/p>\n He also acknowledged that the $4,000 claim, though technically accurate in context, had drawn attention precisely because it challenged expectations about what it costs to experiment at frontier scale.<\/p>\n The release of Brumby-14B-Base is more than an engineering milestone; it is a proof of concept that the transformer\u2019s dominance may finally face credible competition. <\/p>\n By replacing attention with power retention, Manifest AI has demonstrated that performance parity with state-of-the-art transformers is possible at a fraction of the computational cost\u2014and that the long-context bottleneck can be broken without exotic hardware.<\/p>\n The broader implications are twofold. First, the economics of training and serving large models could shift dramatically, lowering the barrier to entry for open research and smaller organizations. <\/p>\n Second, the architectural diversity of AI models may expand again, reigniting theoretical and empirical exploration after half a decade of transformer monoculture.<\/p>\n As Buckman put it: \u201cThe end of the transformer era is not yet here. Our release is just one step forward in a long march toward the future.\u201d<\/p>\n\n\n
\n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n Hardware Efficiency and Inference Performance<\/b><\/h3>\n
Training and Scaling Economics<\/b><\/h3>\n
Integration and Deployment<\/b><\/h3>\n
Mission and Long-Term Vision<\/b><\/h3>\n
Public Debate and Industry Reception<\/b><\/h3>\n
Conclusion: A Crack in the Transformer\u2019s Wall?<\/b><\/h3>\n