Part 3 of a series on AI benchmarking: the agentic benchmark-to-reality gap, in numbers. What a 90-tool-call, hours-long enterprise task reveals that a clean leaderboard score never will
Let’s recap where this series has gotten us. In Part 1, we looked at why the benchmark scores themselves are shakier than they look: saturation compresses the useful range of established tests, and contamination, in both its classic form and its new “the agent just searched for the answer” form, quietly inflates the numbers. In Part 2, we went further and showed that even when a score isn’t inflated by contamination, it can still be the product of an evaluation that’s flat-out broken or gamed, as the UC Berkeley team demonstrated by hacking eight major AI agent benchmarks to near-perfect scores without solving a single task.
So here’s Part 3’s premise, and it’s the one that matters most if you’re actually trying to deploy an AI agent to do real work: suppose none of that happened. Suppose the benchmark is clean, well-run, properly isolated, no contamination, no gaming, the score is completely legitimate. Does that score tell you how the agent will perform when you point it at your business?
The honest answer, according to several of the most rigorous agentic AI studies published in the last year, is: not really. And the reasons why are some of the most useful, most concrete numbers in this entire series.
The “Agentic Disconnect”
Start with a study that should give pause to anyone picking a model based on a benchmark chart: the Kamiwaza Agentic Merit Index (KAMI), built from 170,000 test items and 5.5 billion tokens across 35 model configurations, specifically designed to test enterprise-relevant agentic tasks like extracting and analyzing data from text, CSVs, and databases.
Here’s the finding the researchers themselves call the “agentic disconnect.” When newer open models like Qwen3 launched, the accompanying benchmarks (including popular agentic tool-use tests) showed something striking: even small Qwen3 models, down to 4 billion parameters, appeared to outperform the much larger previous-generation Qwen2.5 72B on agentic benchmarks. A reasonable person reading those charts would conclude the smaller, newer model is simply better, and probably cheaper to run too. Why wouldn’t you switch?
Then KAMI ran its own mundane, enterprise-style agentic tasks. The result: not a single Qwen3 configuration, including the large 235-billion-parameter mixture-of-experts model, actually outperformed the older Qwen2.5 72B at the tasks KAMI tested. The benchmark rankings that suggested an upgrade were, for this kind of work, pointing in the wrong direction entirely.
This is the practical version of everything Part 1 and Part 2 warned about, landing on an actual purchasing decision. If your team picked a model based on “the newer one wins on the agentic tool-use benchmark,” and your actual workload looks like KAMI’s enterprise data-extraction tasks rather than the benchmark’s tasks, you may have just downgraded your agent while believing you upgraded it.
There’s a second, smaller finding in the same study worth keeping in your back pocket: reasoning toggles can radically change both accuracy and cost in ways that don’t track intuitively. One smaller model scored 37.8% with reasoning disabled, using about 900 tokens per conversation. Turn reasoning on, and accuracy rose to 50.5%, but at the cost of roughly 14 times more tokens per conversation. Whether that trade is worth it depends entirely on whether your use case is cost-sensitive or accuracy-sensitive, which is exactly the kind of judgment a single leaderboard number can’t make for you.
What “Good” Actually Looks Like on Enterprise Tasks
If the agentic disconnect makes you want to throw out benchmark rankings entirely and just test everything yourself, the next study explains why that’s also harder than it sounds, and gives you a sense of how far even the best current agents are from “solved.”
AgentArch evaluated 18 distinct agentic architecture configurations (different combinations of orchestration strategy, ReAct-style versus function-calling prompting, memory design, and “thinking tool” integration) across state-of-the-art models on enterprise tasks of varying complexity.
The headline number is sobering: across all 18 configurations and all the frontier models tested, the highest-scoring setup achieved a maximum of 35.3% success on the more complex enterprise task. On the simpler task, the best configuration reached 70.8%. Even granting that “complex” is doing a lot of work in that sentence, a 35.3% ceiling, the best result among 18 carefully-designed architectures, is a long way from the 90%+ numbers that dominate the benchmark conversation in Part 1.
The other finding worth sitting with: there was no single best architecture. Different models performed best with different orchestration and memory setups. The “one-size-fits-all” assumption, pick the best model, wire it up the obvious way, ship it, doesn’t hold. Getting from “this model scores well on a benchmark” to “this agent performs well on our task” involves a second, largely invisible layer of architectural decisions that the benchmark score says nothing about.
What Real Work Actually Looks Like
Part of why benchmarks struggle to predict production performance is that most benchmark tasks are short. A question, an answer, done. Real agentic work isn’t like that, and AgencyBench is useful precisely because it was built to look like real work rather than a benchmark task.
AgencyBench evaluates six core agentic capabilities across 32 real-world scenarios, comprising 138 individual tasks, each with specific deliverables and rubrics rather than a single right answer. The numbers describing what these scenarios actually require are the point: an average of 90 tool calls, roughly 1 million tokens, and hours of execution time, per scenario, to complete.
Ninety tool calls is not “ask a question, get an answer.” That’s ninety opportunities for something to go slightly wrong: a file read that returns unexpected formatting, an API call that times out, a search that returns a near-miss result the agent has to notice is wrong. Each of those is individually small. Across ninety steps and an hour of execution, small error rates compound into large failure rates in a way that a benchmark consisting of one-shot questions structurally cannot capture.
The results reflect that gap. Closed-source models significantly outperformed open-source models, 48.4% versus 32.1%, on these long-horizon scenarios. And the choice of agentic scaffold (the framework wrapping the model: how it manages memory, calls tools, and structures its reasoning) mattered enough that proprietary models performed best within their own native ecosystems, while open-source models had different optimization peaks depending on the framework around them. Once again: the model is not the product. The model plus the scaffold plus the task shape is the product, and benchmarks that test the model in isolation don’t capture the other two.
The Number That Should Worry You Most: Consistency
Of everything in this series, this is the statistic I’d put in front of anyone about to sign off on an agent deployment.
A 2025 study proposing a framework called CLEAR (Cost, Latency, Efficacy, Assurance, Reliability) for evaluating enterprise agentic AI made a simple but devastating observation about how most agent benchmarks are run: once. A single run, a single score, reported as “the” result.
The researchers tested what happens when you run the same agent on the same task repeatedly. On a single run, the agents they tested succeeded about 60% of the time. Across eight consecutive runs of the identical task, the consistency rate dropped to 25%.
Read that again. Not “the agent fails 40% of the time, consistently.” The agent succeeds on roughly 6 out of 10 single attempts, but if you need it to reliably get the same task right repeatedly, the rate at which it does so across eight tries falls to 1 in 4. A benchmark score from a single run, the kind of number that ends up on a vendor’s slide, captures the 60%. It tells you almost nothing about the 25%, and the 25% is what production actually looks like: the same kind of task, run over and over, day after day, by a system you’re not personally watching.
This is, in a sense, the agentic version of the saturation and contamination problems from Part 1. A single-run benchmark score isn’t lying exactly, but it’s measuring something narrower than what it appears to measure, “can this agent do this task,” when the real question for deployment is “can this agent do this task reliably, the fortieth time, the same way it did the first time.”
The Cost Variable Nobody Puts on the Leaderboard
The same CLEAR study found something else that rarely makes it into benchmark comparisons at all: cost.
Across the benchmarks and agents the researchers analyzed, they found up to 50x cost variation between agents achieving similar accuracy on the same tasks. Two agents, comparable scores, and one costs fifty times more to run than the other. When the researchers optimized purely for accuracy, the resulting agents were 4.4 to 10.8 times more expensive than cost-aware alternatives that achieved comparable performance.
If a benchmark leaderboard reports only accuracy, and two agents are within a point of each other (which, per Part 1, you should already be treating as noise), the leaderboard is silent on the possibility that one of those two options costs ten times more to run at scale. For a one-off demo, that’s irrelevant. For a process running continuously across an organization, that’s the difference between a project that pays for itself and one that quietly becomes a budget problem six months in.
The researchers’ proposed fix, the CLEAR framework itself, scores agents across cost, latency, efficacy, assurance (things like security and policy compliance), and reliability, not accuracy alone. When they validated this against expert human judgment of which agents would actually succeed in production (a panel of 15 evaluators), the CLEAR score correlated with production success at 0.83. Accuracy alone correlated at 0.41. That’s roughly the difference between a measurement that’s actually useful and one that’s barely better than a coin flip.
Putting the Series Together
Here’s the throughline across all three parts, stated plainly.
Part 1: the benchmark score itself is noisier and more contaminated than its precision suggests, especially once agents can search for their own answer keys.
Part 2: even a score that isn’t contaminated can be the product of a benchmark that’s gameable or simply broken, and the incentive structure around leaderboards (funding, press, procurement) rewards exactly this kind of gaming, whether deliberate or emergent.
Part 3: even a score that’s clean, ungamed, and accurately measured can fail to predict real-world performance, because real work involves long tool-call chains where small errors compound, because the best architecture isn’t universal, because “the model that wins the benchmark” can lose on your actual workload, and because a single run tells you almost nothing about whether the agent will still be right on the fortieth try.
If you’re evaluating an AI agent for real deployment, here’s what to actually ask for, in order of how rarely vendors volunteer it: run-to-run consistency across at least a handful of repeated trials on the same task, not a single best-case run. Cost-per-task at the volume you’d actually run it, not accuracy in isolation. Performance on tasks that resemble your actual workflow in shape, long chains of tool calls with realistic friction, not short one-shot questions. And, if at all possible, your own small-scale pilot on your own data, because as KAMI’s “agentic disconnect” shows, the benchmark that predicts performance on someone else’s workload might predict nothing about yours.
None of this means AI agents aren’t useful, or that the underlying technology isn’t improving fast, because it clearly is. It means the leaderboard, in all three of the ways this series has covered, was never the whole story. The number on the chart is a starting point for a conversation, not the end of one. Ask the next question. The vendors who can answer it are the ones worth talking to.
Sources: