Need smarter insights in your inbox? Join our weekly newsletters to get solely what issues to enterprise AI, information, and safety leaders. Subscribe Now
A complete new research has revealed that open-source synthetic intelligence fashions devour considerably extra computing assets than their closed-source rivals when performing equivalent duties, doubtlessly undermining their value benefits and reshaping how enterprises consider AI deployment methods.
The analysis, carried out by AI agency Nous Analysis, discovered that open-weight fashions use between 1.5 to 4 occasions extra tokens — the essential items of AI computation — than closed fashions like these from OpenAI and Anthropic. For easy information questions, the hole widened dramatically, with some open fashions utilizing as much as 10 occasions extra tokens.
Measuring Pondering Effectivity in Reasoning Fashions: The Lacking Benchmarkhttps://t.co/b1e1rJx6vZ
We measured token utilization throughout reasoning fashions: open fashions output 1.5-4x extra tokens than closed fashions on equivalent duties, however with large variance relying on process sort (as much as… pic.twitter.com/LY1083won8
— Nous Analysis (@NousResearch) August 14, 2025
“Open weight fashions use 1.5–4× extra tokens than closed ones (as much as 10× for easy information questions), making them generally dearer per question regardless of decrease per‑token prices,” the researchers wrote of their report printed Wednesday.
The findings problem a prevailing assumption within the AI business that open-source fashions provide clear financial benefits over proprietary options. Whereas open-source fashions usually value much less per token to run, the research suggests this benefit could be “simply offset in the event that they require extra tokens to purpose a few given downside.”
AI Scaling Hits Its Limits
Energy caps, rising token prices, and inference delays are reshaping enterprise AI. Be a part of our unique salon to find how prime groups are:
- Turning power right into a strategic benefit
- Architecting environment friendly inference for actual throughput positive factors
- Unlocking aggressive ROI with sustainable AI methods
Safe your spot to remain forward: https://bit.ly/4mwGngO
The actual value of AI: Why ‘cheaper’ fashions might break your funds
The analysis examined 19 totally different AI fashions throughout three classes of duties: primary information questions, mathematical issues, and logic puzzles. The workforce measured “token effectivity” — what number of computational items fashions use relative to the complexity of their options—a metric that has obtained little systematic research regardless of its important value implications.
“Token effectivity is a crucial metric for a number of sensible causes,” the researchers famous. “Whereas internet hosting open weight fashions could also be cheaper, this value benefit might be simply offset in the event that they require extra tokens to purpose a few given downside.”
The inefficiency is especially pronounced for Massive Reasoning Fashions (LRMs), which use prolonged “chains of thought” to unravel complicated issues. These fashions, designed to suppose by means of issues step-by-step, can devour hundreds of tokens pondering easy questions that ought to require minimal computation.
For primary information questions like “What’s the capital of Australia?” the research discovered that reasoning fashions spend “a whole lot of tokens pondering easy information questions” that might be answered in a single phrase.
Which AI fashions really ship bang on your buck
The analysis revealed stark variations between mannequin suppliers. OpenAI’s fashions, notably its o4-mini and newly launched open-source gpt-oss variants, demonstrated distinctive token effectivity, particularly for mathematical issues. The research discovered OpenAI fashions “stand out for excessive token effectivity in math issues,” utilizing as much as thrice fewer tokens than different industrial fashions.
Amongst open-source choices, Nvidia’s llama-3.3-nemotron-super-49b-v1 emerged as “probably the most token environment friendly open weight mannequin throughout all domains,” whereas newer fashions from corporations like Magistral confirmed “exceptionally excessive token utilization” as outliers.
The effectivity hole different considerably by process sort. Whereas open fashions used roughly twice as many tokens for mathematical and logic issues, the distinction ballooned for easy information questions the place environment friendly reasoning must be pointless.
What enterprise leaders must find out about AI computing prices
The findings have instant implications for enterprise AI adoption, the place computing prices can scale quickly with utilization. Firms evaluating AI fashions usually deal with accuracy benchmarks and per-token pricing, however might overlook the whole computational necessities for real-world duties.
“The higher token effectivity of closed weight fashions usually compensates for the upper API pricing of these fashions,” the researchers discovered when analyzing complete inference prices.
The research additionally revealed that closed-source mannequin suppliers look like actively optimizing for effectivity. “Closed weight fashions have been iteratively optimized to make use of fewer tokens to cut back inference value,” whereas open-source fashions have “elevated their token utilization for newer variations, presumably reflecting a precedence towards higher reasoning efficiency.”
How researchers cracked the code on AI effectivity measurement
The analysis workforce confronted distinctive challenges in measuring effectivity throughout totally different mannequin architectures. Many closed-source fashions don’t reveal their uncooked reasoning processes, as a substitute offering compressed summaries of their inner computations to forestall rivals from copying their methods.
To deal with this, researchers used completion tokens — the whole computational items billed for every question — as a proxy for reasoning effort. They found that “most up-to-date closed supply fashions is not going to share their uncooked reasoning traces” and as a substitute “use smaller language fashions to transcribe the chain of thought into summaries or compressed representations.”
The research’s methodology included testing with modified variations of well-known issues to attenuate the affect of memorized options, similar to altering variables in mathematical competitors issues from the American Invitational Arithmetic Examination (AIME).
The way forward for AI effectivity: What’s coming subsequent
The researchers recommend that token effectivity ought to grow to be a main optimization goal alongside accuracy for future mannequin improvement. “A extra densified CoT will even enable for extra environment friendly context utilization and should counter context degradation throughout difficult reasoning duties,” they wrote.
The discharge of OpenAI’s open-source gpt-oss fashions, which reveal state-of-the-art effectivity with “freely accessible CoT,” may function a reference level for optimizing different open-source fashions.
The entire analysis dataset and analysis code are out there on GitHub, permitting different researchers to validate and prolong the findings. Because the AI business races towards extra highly effective reasoning capabilities, this research means that the true competitors is probably not about who can construct the neatest AI — however who can construct probably the most environment friendly one.
In any case, in a world the place each token counts, probably the most wasteful fashions might discover themselves priced out of the market, no matter how properly they will suppose.
