Join the event trusted by enterprise leaders for nearly two decades. VB Transform brings together the people building real enterprise AI strategy.\u00a0Learn more<\/em><\/p>\n\n\n\n Apple\u2019s machine-learning group set off a rhetorical firestorm earlier this month with its release of \u201cThe Illusion of Thinking,\u201d a 53-page research paper arguing that so-called large reasoning models (LRMs) or reasoning large language models (reasoning LLMs) such as OpenAI\u2019s \u201co\u201d series and Google\u2019s Gemini-2.5 Pro and Flash Thinking don\u2019t actually engage in independent \u201cthinking\u201d or \u201creasoning\u201d from generalized first principles learned from their training data.<\/p>\n\n\n\n Instead, the authors contend, these reasoning LLMs are actually performing a kind of \u201cpattern matching\u201d and their apparent reasoning ability seems to fall apart once a task becomes too complex, suggesting that their architecture and performance is not a viable path to improving generative AI to the point that it is artificial generalized intelligence (AGI), which OpenAI defines as a model that outperforms humans at most economically valuable work, or superintelligence, AI even smarter than human beings can comprehend.<\/p>\n\n\n\n Unsurprisingly, the paper immediately circulated widely among the machine learning community on X and many readers\u2019 initial reactions were to declare that Apple had effectively disproven much of the hype around this class of AI: \u201cApple just proved AI \u2018reasoning\u2019\u00a0models like Claude, DeepSeek-R1, and o3-mini don\u2019t actually reason at all,\u201d declared Ruben Hassid, creator of EasyGen, an LLM-driven LinkedIn post auto writing tool. \u201cThey just memorize patterns really well.\u201d<\/p>\n\n\n\n But now today, a new paper has emerged, the cheekily titled \u201cThe Illusion of The Illusion of Thinking\u201d \u2014 importantly, co-authored by a reasoning LLM itself, Claude Opus 4 and Alex Lawsen, a human being and independent AI researcher and technical writer \u2014 that includes many criticisms from the larger ML community about the paper and effectively argues that the methodologies and experimental designs the Apple Research team used in their initial work are fundamentally flawed.<\/p>\n\n\n\n While we here at VentureBeat are not ML researchers ourselves and not prepared to say the Apple Researchers are wrong, the debate has certainly been a lively one and the issue about the capabilities of LRMs or reasoner LLMs compared to human thinking seems far from settled.<\/p>\n\n\n\n Using four classic planning problems \u2014 Tower of Hanoi, Blocks World, River Crossing and Checkers Jumping \u2014 Apple\u2019s researchers designed a battery of tasks that forced reasoning models to plan multiple moves ahead and generate complete solutions. <\/p>\n\n\n\n These games were chosen for their long history in cognitive science and AI research and their ability to scale in complexity as more steps or constraints are added. Each puzzle required the models to not just produce a correct final answer, but to explain their thinking along the way using chain-of-thought prompting.<\/p>\n\n\n\n As the puzzles increased in difficulty, the researchers observed a consistent drop in accuracy across multiple leading reasoning models. In the most complex tasks, performance plunged to zero. Notably, the length of the models\u2019 internal reasoning traces\u2014measured by the number of tokens spent thinking through the problem\u2014also began to shrink. Apple\u2019s researchers interpreted this as a sign that the models were abandoning problem-solving altogether once the tasks became too hard, essentially \u201cgiving up.\u201d<\/p>\n\n\n\n The timing of the paper\u2019s release, just ahead of Apple\u2019s annual Worldwide Developers Conference (WWDC), added to the impact. It quickly went viral across X, where many interpreted the findings as a high-profile admission that current-generation LLMs are still glorified autocomplete engines, not general-purpose thinkers. This framing, while controversial, drove much of the initial discussion and debate that followed.<\/p>\n\n\n\n Among the most vocal critics of the Apple paper was ML researcher and X user @scaling01 (aka \u201cLisan al Gaib\u201d), who posted multiple threads dissecting the methodology. <\/p>\n\n\n\n In one widely shared post, Lisan argued that the Apple team conflated token budget failures with reasoning failures, noting that \u201call models will have 0 accuracy with more than 13 disks simply because they cannot output that much!\u201d <\/p>\n\n\n\n For puzzles like Tower of Hanoi, he emphasized, the output size grows exponentially, while the LLM context windows remain fixed, writing \u201cjust because Tower of Hanoi requires exponentially more steps than the other ones, that only require quadratically or linearly more steps, doesn\u2019t mean Tower of Hanoi is more difficult\u201d and convincingly showed that models like Claude 3 Sonnet and DeepSeek-R1 often produced algorithmically correct strategies in plain text or code\u2014yet were still marked wrong.<\/p>\n\n\n\n Another post highlighted that even breaking the task down into smaller, decomposed steps worsened model performance\u2014not because the models failed to understand, but because they lacked memory of previous moves and strategy. <\/p>\n\n\n\n \u201cThe LLM needs the history and a grand strategy,\u201d he wrote, suggesting the real problem was context-window size rather than reasoning.<\/p>\n\n\n\n I raised another important grain of salt myself on X: Apple never benchmarked the model performance against human performance on the same tasks. \u201cAm I missing it, or did you not compare LRMs to human perf[ormance] on [the] same tasks?? If not, how do you know this same drop-off in perf doesn\u2019t happen to people, too?\u201d I asked the researchers directly in a thread tagging the paper\u2019s authors. I also emailed them about this and many other questions, but they have yet to respond.<\/p>\n\n\n\n Others echoed that sentiment, noting that human problem solvers also falter on long, multistep logic puzzles, especially without pen-and-paper tools or memory aids. Without that baseline, Apple\u2019s claim of a fundamental \u201creasoning collapse\u201d feels ungrounded.<\/p>\n\n\n\n Several researchers also questioned the binary framing of the paper\u2019s title and thesis\u2014drawing a hard line between \u201cpattern matching\u201d and \u201creasoning.\u201d <\/p>\n\n\n\n Alexander Doria aka Pierre-Carl Langlais, an LLM trainer at energy efficient French AI startup Pleias, said the framing misses the nuance<\/em>, arguing that models might be learning partial heuristics rather than simply matching patterns. <\/p>\n\n\n\n Ok I guess I have to go through that Apple paper. <\/p> My main issue is the framing which is super binary: “Are these models capable of generalizable reasoning, or are they leveraging different forms of pattern matching?” Or what if they only caught genuine yet partial heuristics. pic.twitter.com\/GZE3eG7WlM<\/a><\/p>\u2014 Alexander Doria (@Dorialexander) June 8, 2025<\/a><\/blockquote>\n<\/div><\/figure>\n\n\n\n Ethan Mollick, the AI focused professor at University of Pennsylvania\u2019s Wharton School of Business, called the idea that LLMs are \u201chitting a wall\u201d premature, likening it to similar claims about \u201cmodel collapse\u201d that didn\u2019t pan out.<\/p>\n\n\n\n Meanwhile, critics like @arithmoquine were more cynical, suggesting that Apple\u2014behind the curve on LLMs compared to rivals like OpenAI and Google\u2014might be trying to lower expectations,\u201d coming up with research on \u201chow it\u2019s all fake and gay and doesn\u2019t matter anyway\u201d they quipped, pointing out Apple\u2019s reputation with now poorly performing AI products like Siri.<\/p>\n\n\n\n In short, while Apple\u2019s study triggered a meaningful conversation about evaluation rigor, it also exposed a deep rift over how much trust to place in metrics when the test itself might be flawed.<\/p>\n\n\n\n In other words, the models may have understood the puzzles but ran out of \u201cpaper\u201d to write the full solution.<\/p>\n\n\n\n \u201cToken limits, not logic, froze the models,\u201d wrote Carnegie Mellon researcher Rohan Paul in a widely shared thread summarizing the follow-up tests.<\/p>\n\n\n\n Yet not everyone is ready to clear LRMs of the charge. Some observers point out that Apple\u2019s study still revealed three performance regimes \u2014 simple tasks where added reasoning hurts, mid-range puzzles where it helps, and high-complexity cases where both standard and \u201cthinking\u201d models crater.<\/p>\n\n\n\n Others view the debate as corporate positioning, noting that Apple\u2019s own on-device \u201cApple Intelligence\u201d models trail rivals on many public leaderboards.<\/p>\n\n\n\n In response to Apple\u2019s claims, a new paper titled \u201cThe Illusion of the Illusion of Thinking\u201d was released on arXiv by independent researcher and technical writer Alex Lawsen of the nonprofit Open Philanthropy, in collaboration with Anthropic\u2019s Claude Opus 4. <\/p>\n\n\n\n The paper directly challenges the original study\u2019s conclusion that LLMs fail due to an inherent inability to reason at scale. Instead, the rebuttal presents evidence that the observed performance collapse was largely a by-product of the test setup\u2014not a true limit of reasoning capability.<\/p>\n\n\n\n Lawsen and Claude demonstrate that many of the failures in the Apple study stem from token limitations. For example, in tasks like Tower of Hanoi, the models must print exponentially many steps \u2014 over 32,000 moves for just 15 disks \u2014 leading them to hit output ceilings. <\/p>\n\n\n\n The rebuttal points out that Apple\u2019s evaluation script penalized these token-overflow outputs as incorrect, even when the models followed a correct solution strategy internally.<\/p>\n\n\n\n The authors also highlight several questionable task constructions in the Apple benchmarks. Some of the River Crossing puzzles, they note, are mathematically unsolvable as posed, and yet model outputs for these cases were still scored. This further calls into question the conclusion that accuracy failures represent cognitive limits rather than structural flaws in the experiments.<\/p>\n\n\n\n To test their theory, Lawsen and Claude ran new experiments allowing models to give compressed, programmatic answers. When asked to output a Lua function that could generate the Tower of Hanoi solution\u2014rather than writing every step line-by-line\u2014models suddenly succeeded on far more complex problems. This shift in format eliminated the collapse entirely, suggesting that the models didn\u2019t fail to reason. They simply failed to conform to an artificial and overly strict rubric.<\/p>\n\n\n\n The back-and-forth underscores a growing consensus: evaluation design is now as important as model design.<\/p>\n\n\n\n Requiring LRMs to enumerate every step may test their printers more than their planners, while compressed formats, programmatic answers or external scratchpads give a cleaner read on actual reasoning ability.<\/p>\n\n\n\n The episode also highlights practical limits developers face as they ship agentic systems\u2014context windows, output budgets and task formulation can make or break user-visible performance.<\/p>\n\n\n\n For enterprise technical decision makers building applications atop reasoning LLMs, this debate is more than academic. It raises critical questions about where, when, and how to trust these models in production workflows\u2014especially when tasks involve long planning chains or require precise step-by-step output.<\/p>\n\n\n\n If a model appears to \u201cfail\u201d on a complex prompt, the problem may not lie in its reasoning ability, but in how the task is framed, how much output is required, or how much memory the model has access to. This is particularly relevant for industries building tools like copilots, autonomous agents, or decision-support systems, where both interpretability and task complexity can be high.<\/p>\n\n\n\n Understanding the constraints of context windows, token budgets, and the scoring rubrics used in evaluation is essential for reliable system design. Developers may need to consider hybrid solutions that externalize memory, chunk reasoning steps, or use compressed outputs like functions or code instead of full verbal explanations.<\/p>\n\n\n\n Most importantly, the paper\u2019s controversy is a reminder that benchmarking and real-world application are not the same. Enterprise teams should be cautious of over-relying on synthetic benchmarks that don\u2019t reflect practical use cases\u2014or that inadvertently constrain the model\u2019s ability to demonstrate what it knows.<\/p>\n\n\n\n Ultimately, the big takeaway for ML researchers is that before proclaiming an AI milestone\u2014or obituary\u2014make sure the test itself isn\u2019t putting the system in a box too small to think inside.<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>ACT NOW: Come discuss the latest LLM advances and research at VB Transform on June 24-25 in SF \u2014 limited tickets available<\/em>. REGISTER NOW<\/h2>\n\n\n\n
How the Apple Research study was designed \u2014 and what it found<\/h2>\n\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/06\/Gs7EI0SXsAAVpXk.jpg?w=800\")
<\/figure>\n\n\n\nCritics take aim on X<\/h2>\n\n\n\n
A measurement artifact, or a ceiling?<\/h2>\n\n\n\n
The rebuttal: \u201cThe Illusion of the Illusion of Thinking\u201d<\/h2>\n\n\n\n
Why it matters for enterprise decision-makers<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div><\/script>\r\n