Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More<\/em><\/p>\n\n\n\n Large language models (LLMs) are increasingly capable of complex reasoning through \u201cinference-time scaling,\u201d a set of techniques that allocate more computational resources during inference to generate answers. However, a new study from Microsoft Research reveals that the effectiveness of these scaling methods isn\u2019t universal. Performance boosts vary significantly across different models, tasks and problem complexities.<\/p>\n\n\n\n The core finding is that simply throwing more compute at a problem during inference doesn\u2019t guarantee better or more efficient results. The findings can help enterprises better understand cost volatility and model reliability as they look to integrate advanced AI reasoning into their applications.<\/p>\n\n\n\n The Microsoft Research team conducted an extensive empirical analysis across nine state-of-the-art foundation models. This included both \u201cconventional\u201d models like GPT-4o, Claude 3.5 Sonnet, Gemini 2.0 Pro and Llama 3.1 405B, as well as models specifically fine-tuned for enhanced reasoning through inference-time scaling. This included OpenAI\u2019s o1 and o3-mini, Anthropic\u2019s Claude 3.7 Sonnet, Google\u2019s Gemini 2 Flash Thinking, and DeepSeek R1.<\/p>\n\n\n\n They evaluated these models using three distinct inference-time scaling approaches:<\/p>\n\n\n\n These approaches were tested on eight challenging benchmark datasets covering a wide range of tasks that benefit from step-by-step problem-solving: math and STEM reasoning (AIME, Omni-MATH, GPQA), calendar planning (BA-Calendar), NP-hard problems (3SAT, TSP), navigation (Maze) and spatial reasoning (SpatialMap).<\/p>\n\n\n\n Several benchmarks included problems with varying difficulty levels, allowing for a more nuanced understanding of how scaling behaves as problems become harder.<\/p>\n\n\n\n \u201cThe availability of difficulty tags for Omni-MATH, TSP, 3SAT, and BA-Calendar enables us to analyze how accuracy and token usage scale with difficulty in inference-time scaling, which is a perspective that is still underexplored,\u201d the researchers wrote in the paper detailing their findings.<\/p>\n\n\n\n The researchers evaluated the Pareto frontier of LLM reasoning by analyzing both accuracy and the computational cost (i.e., the number of tokens generated). This helps identify how efficiently models achieve their results.\u00a0<\/p>\n\n\n\n They also introduced the \u201cconventional-to-reasoning gap\u201d measure, which compares the best possible performance of a conventional model (using an ideal \u201cbest-of-N\u201d selection) against the average performance of a reasoning model, estimating the potential gains achievable through better training or verification techniques.<\/p>\n\n\n\n The study provided several crucial insights that challenge common assumptions about inference-time scaling:<\/p>\n\n\n\n Benefits vary significantly:<\/strong> While models tuned for reasoning generally outperform conventional ones on these tasks, the degree of improvement varies greatly depending on the specific domain and task. Gains often diminish as problem complexity increases. For instance, performance improvements seen on math problems didn\u2019t always translate equally to scientific reasoning or planning tasks.<\/p>\n\n\n\n Token inefficiency is rife:<\/strong> The researchers observed high variability in token consumption, even between models achieving similar accuracy. For example, on the AIME 2025 math benchmark, DeepSeek-R1 used over five times more tokens than Claude 3.7 Sonnet for roughly comparable average accuracy.\u00a0<\/p>\n\n\n\n More tokens do not lead to higher accuracy:<\/strong>\u00a0Contrary to the intuitive idea that longer reasoning chains mean better reasoning, the study found this isn\u2019t always true.<\/span> \u201cSurprisingly, we also observe that longer generations relative to the same model can sometimes be an indicator of models struggling, rather than improved reflection,\u201d the paper states. \u201cSimilarly, when comparing different reasoning models, higher token usage is not always associated with better accuracy. These findings motivate the need for more purposeful and cost-effective scaling approaches.\u201d<\/p>\n\n\n\n Cost nondeterminism:<\/strong> Perhaps most concerning for enterprise users, repeated queries to the same model for the same problem can result in highly variable token usage. This means the cost of running a query can fluctuate significantly, even when the model consistently provides the correct answer.\u00a0<\/p>\n\n\n The potential in verification mechanisms:<\/strong> Scaling performance consistently improved across all models and benchmarks when simulated with a \u201cperfect verifier\u201d (using the best-of-N results).\u00a0<\/p>\n\n\n\n Conventional models sometimes match reasoning models:<\/strong> By significantly increasing inference calls (up to 50x more in some experiments), conventional models like GPT-4o could sometimes approach the performance levels of dedicated reasoning models, particularly on less complex tasks. However, these gains diminished rapidly in highly complex settings, indicating that brute-force scaling has its limits.<\/p>\n\n\n\n These findings carry significant weight for developers and enterprise adopters of LLMs. The issue of \u201ccost nondeterminism\u201d is particularly stark and makes budgeting difficult. As the researchers point out, \u201cIdeally, developers and users would prefer models for which the standard deviation on token usage per instance is low for cost predictability.\u201d<\/p>\n\n\n\n \u201cThe profiling we do in [the study] could be useful for developers as a tool to pick which models are less volatile for the same prompt or for different prompts,\u201d Besmira Nushi, senior principal research manager at Microsoft Research, told VentureBeat. \u201cIdeally, one would want to pick a model that has low standard deviation for correct inputs.\u201d\u00a0<\/p>\n\n\n\n The study also provides good insights into the correlation between a model\u2019s accuracy and response length. For example, the following diagram shows that math queries above ~11,000 token length have a very slim chance of being correct, and those generations should either be stopped at that point or restarted with some sequential feedback. However, Nushi points out that models allowing these post hoc mitigations also have a cleaner separation between correct and incorrect samples.<\/p>\n\n\n\n \u201cUltimately, it is also the responsibility of model builders to think about reducing accuracy and cost non-determinism, and we expect a lot of this to happen as the methods get more mature,\u201d Nushi said. \u201cAlongside cost nondeterminism, accuracy nondeterminism also applies.\u201d<\/p>\n\n\n\n Another important finding is the consistent performance boost from perfect verifiers, which highlights a critical area for future work: building robust and broadly applicable verification mechanisms.\u00a0<\/p>\n\n\n\n \u201cThe availability of stronger verifiers can have different types of impact,\u201d Nushi said, such as improving foundational training methods for reasoning. \u201cIf used efficiently, these can also shorten the reasoning traces.\u201d<\/p>\n\n\n\n Strong verifiers can also become a central part of enterprise agentic AI solutions. Many enterprise stakeholders already have such verifiers in place, which may need to be repurposed for more agentic solutions, such as SAT solvers, logistic validity checkers, etc.\u00a0<\/p>\n\n\n\n \u201cThe questions for the future are how such existing techniques can be combined with AI-driven interfaces and what is the language that connects the two,\u201d Nushi said. \u201cThe necessity of connecting the two comes from the fact that users will not always formulate their queries in a formal way, they will want to use a natural language interface and expect the solutions in a similar format or in a final action (e.g. propose a meeting invite).\u201d<\/p>\n Daily insights on business use cases with VB Daily<\/strong><\/p>\n If you want to impress your boss, VB Daily has you covered. We give you the inside scoop on what companies are doing with generative AI, from regulatory shifts to practical deployments, so you can share insights for maximum ROI.<\/p>\n \n\t\t\t\t\tThanks for subscribing. Check out more VB newsletters here.\n\t\t\t\t<\/p>\n An error occured.<\/p>\n<\/p><\/div>\n
\n<\/div>Putting scaling methods to the test<\/h2>\n\n\n\n
\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/04\/image_70ff62.png?w=483\")
<\/figure><\/div>\n\n\n![\"Inference-time](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/04\/image_ebef09.png?w=800\")
More compute isn\u2019t always the answer<\/h2>\n\n\n\n
![\"variance](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/04\/image_909607.png?w=800\")
![\"GPT-4o](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/04\/image_fe40bb.png?w=800\")
Implications for the enterprise<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/04\/image_84c14a.png?w=800\")
![\"\"](\"https:\/\/venturebeat.com\/wp-content\/uploads\/2025\/04\/image001.png\")
<\/figure>\n\n\n\n
![\"\"\/](\"https:\/\/venturebeat.com\/wp-content\/themes\/vb-news\/brand\/img\/vb-daily-phone.png\")
\n\t\t\t\t<\/div>\n<\/p><\/div>\n<\/div>\t\t\t<\/div>\r\n