Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders.<\/em> Subscribe Now<\/em><\/p>\n\n\n\n Researchers at KAIST AI and Mila have introduced a new Transformer architecture that makes large language models (LLMs) more memory- and compute-efficient. The architecture, called Mixture-of-Recursions (MoR), significantly improves model accuracy and delivers higher throughput compared with vanilla transformers, even when constrained by the same parameter count and compute budget.<\/p>\n\n\n\n The impressive capabilities of today\u2019s LLMs are directly tied to their ever-increasing size. But as these models scale, their memory footprints and computational requirements often become untenable, making both training and deployment challenging for organizations outside of hyperscale data centers. This has led to a search for more efficient designs.<\/p>\n\n\n\n Efforts to improve LLM efficiency have focused mainly on two methods: parameter sharing and adaptive computation. Parameter sharing techniques reduce the total number of unique parameters by reusing weights across different parts of the model, thereby reducing the overall computational complexity. For example, \u201clayer tying\u201d is a technique that reuses a model\u2019s weights across several layers. Adaptive computation methods adjust models so that they only use as much inference resources as they need. For example, \u201cearly exiting\u201d dynamically allocates compute by allowing the model to stop processing \u201csimpler\u201d tokens early in the network.<\/p>\n\n\n\n However, creating an architecture that effectively unifies both parameter efficiency and adaptive computation remains elusive.<\/p>\n\n\n\n The AI Impact Series Returns to San Francisco – August 5<\/strong><\/p>\n\n\n\n The next phase of AI is here – are you ready? Join leaders from Block, GSK, and SAP for an exclusive look at how autonomous agents are reshaping enterprise workflows – from real-time decision-making to end-to-end automation.<\/p>\n\n\n\n Secure your spot now – space is limited: https:\/\/bit.ly\/3GuuPLF<\/p>\n\n\n\n Mixture-of-Recursions is a framework that combines parameter sharing with adaptive computation to tackle the high computational demands of LLMs. It builds on the concept of Recursive Transformers, models that repeatedly apply a set of shared layers multiple times. Instead of a deep stack of unique layers, a Recursive Transformer partitions the model into a few \u201crecursion blocks,\u201d each with a shared pool of parameters. This design allows for more computation without increasing the model\u2019s size.<\/p>\n\n\n\n MoR enhances this recursive approach with two key components. The first is a lightweight router that intelligently assigns a specific recursion depth to each token. This concept is similar to the routing mechanism in Mixture-of-Experts (MoE) models, where a router directs tokens to specialized expert networks. In MoR, however, the \u201cexperts\u201d are the different recursion depths, allowing the model to choose how much computation to apply to each token dynamically. It decides how many times a shared block of layers should be applied based on a token\u2019s complexity, or its required \u201cdepth of thinking.\u201d This directs computation only where it is most needed, avoiding wasted cycles on easy-to-process parts of the input.<\/p>\n\n\n The second component is a more efficient key-value (KV) caching strategy. KV caching is a standard technique that stores information from previous tokens to speed up generation, but it becomes a memory bottleneck in recursive models. MoR introduces a \u201crecursion-wise\u201d KV caching mechanism that selectively stores and retrieves key-value pairs only for the tokens that are still active at a given recursion step. This targeted caching reduces memory traffic and improves throughput without needing complex, post-training modifications.<\/p>\n\n\n\n As the researchers state in their paper, \u201cIn essence, MoR enables models to efficiently adjust their thinking depth on a per-token basis, unifying parameter efficiency with adaptive computation.\u201d<\/p>\n\n\n\n To test their framework, the researchers trained MoR models ranging from 135 million to 1.7 billion parameters and compared them against vanilla and standard recursive baseline models on validation loss and few-shot accuracy benchmarks.<\/p>\n\n\n\n The results demonstrate significant gains. When given an equal training compute budget, an MoR model achieved higher average few-shot accuracy (43.1% vs. 42.3%) than a vanilla baseline despite using nearly 50% fewer parameters. When trained on the same amount of data, the MoR model reduced training time by 19% and cut peak memory usage by 25% compared to the vanilla model.<\/p>\n\n\n\n The MoR architecture also proves to be scalable. While it slightly underperformed the vanilla model at the smallest 135M parameter scale, the gap closed rapidly as the model size increased. For models with over 360M parameters, MoR matched or exceeded the performance of standard Transformers, especially on lower compute budgets. Furthermore, MoR\u2019s design dramatically boosts inference throughput. One MoR configuration achieved a 2.06x speedup over the vanilla baseline. For a company operating at scale, this could translate into significant operational cost savings.<\/p>\n\n\n\n Sangmin Bae, co-author of the paper and a PhD student at KAIST, broke down the practical impact in an email to VentureBeat. \u201cWhile it\u2019s difficult to provide exact numbers, at a high level, reducing model parameter size and KV cache footprint means we can perform inference on many more samples simultaneously,\u201d he said. \u201cThis translates to an increased number of tokens processed at once, and handling longer context windows becomes feasible.\u201d<\/p>\n\n\n\n While the paper\u2019s results come from models trained from scratch, a key question for enterprises is how to adopt MoR without massive upfront investment. According to Bae, \u201cuptraining\u201d existing open-source models is a \u201cdefinitely more cost-effective approach.\u201d He noted that while training a new model is straightforward, an \u201cuptraining approach could be more suitable and efficient until the scalability of MoR itself is fully validated.\u201d<\/p>\n\n\n\n Adopting MoR also introduces new architectural \u201cknobs\u201d for developers, allowing them to fine-tune the balance between performance and efficiency. This trade-off will depend entirely on the application\u2019s needs.<\/p>\n\n\n\n \u201cFor simpler tasks or scenarios, it may be beneficial to use models with more recursion steps, offering greater flexibility, and vice versa,\u201d Bae explained. He stressed that the \u201coptimal settings will highly depend on the specific deployment setting,\u201d encouraging teams to explore the trade-offs based on the paper\u2019s findings.<\/p>\n\n\n\n Looking ahead, the MoR framework is \u201cmodality-agnostic,\u201d meaning its adaptive computation principles are not limited to text. This opens the door to significant efficiency gains in processing video, audio, and other complex data types.<\/p>\n\n\n\n \u201cWe\u2019re very excited about its potential extension to multi-modality scenarios where efficiency gains are crucial,\u201d Bae said.<\/p>\n\n\n\n By dynamically adjusting the processing depth for each segment of a video or audio stream, MoR could unlock even greater cost savings and performance improvements, bringing the power of large-scale AI to a wider range of enterprise applications. As the paper concludes, MoR offers \u201can effective path towards achieving large-model capabilities with significantly reduced computational and memory overhead.\u201d<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>The scaling challenges of LLMs<\/h2>\n\n\n\n
\n\n\n\n
\n<\/div>How Mixture-of-Recursions works<\/h2>\n\n\n\n
![\"Mixture-of-recursion](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/07\/image_500670.png?w=356\")
![\"Different](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/07\/image_6c5840.png?w=800\")
MoR in action<\/h2>\n\n\n\n
A practical path for enterprise adoption<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n