How IT leaders can measure and improve agentic AI value and cost

Token prices have never been lower. Enterprise AI bills have never been higher. 

Over the past year, the blended price enterprises pay per token has fallen by roughly 75% as model providers race each other toward commodity pricing.¹ Ask any CIO who signed off on an agentic AI program what happened to the bill, and the answer is that it tripled.¹ The math appears to defy logic, but it doesn’t. It just requires a new frame.

A conventional chatbot exchange is a single round-trip, one prompt and one response. An AI agent handling a sales-qualification workflow, a security-triage task, or a code review does not make a single call. It makes dozens, reasoning through problems in steps, retrying on failure, calling external tools, reinjecting context on every turn, and handing off to human review when confidence drops. 

Agentic workflows routinely consume 5 to 30 times more tokens per task than a single chatbot interaction. Multiply that across enterprise volume and a 75% price reduction triples the bill, without anyone making an obviously bad decision.

The scale of this hidden cost is striking. A 2026 audit of 30 production engineering teams found that resent context alone accounted for 62% of the agent inference bill.² Research from the Stanford Digital Economy Lab shows where the money goes. Input, not output, drives the cost of agentic work, because an agent rereads its accumulated context before nearly every action. In that study, agentic coding tasks burned up to 1,000 times more tokens than a simple code-chat exchange, and the cost of the same task varied by as much as 30 times from one run to the next.³ Every reasoning step retrieves the knowledge base again, and every call retokenizes the system prompt. Prompt caching can cut that re-sent cost by up to 90%.⁴ The discount must be engineered and then defended. Prompts have to be structured so the repeated prefix stays stable, and a single changed token silently turns cached reads back into full-price ones. Teams without telemetry never learn that caching is their largest lever, and never notice when the discount quietly stops landing.

The old ROI model assumed predictable cost, deterministic output, and linear value accumulation. None of that holds for agents. Replacing it takes new metrics, new measurement infrastructure, and a different idea of what “cost” and “value” mean at the task level. It also demands a different posture from IT leadership about what gets tracked and who is accountable for it.

Most organizations cannot measure agentic AI at all, and that gap sits with IT. 

A 2026 Elastic survey of enterprise IT organizations found that 85% plan to implement LLM observability, yet only 8% have done so. That is not an intent problem. It is an infrastructure problem. Vendor billing dashboards provide messages and credits, aggregate counts that cannot attribute cost to individual prompts, tool calls, retries, or escalations.

Without first-party telemetry at the trace, call, and model level, the four metrics I describe below remain estimates rather than measurements. A CIO may know their AI program costs came in higher than projected, but without trace-level data, they cannot determine whether the driver is retrieval inefficiency, model selection, retry loops, governance failures, or all four.

Invest in first-party telemetry that tells you why costs fluctuate before you invest in outcome dashboards. You need to know the cost of each task, the model it used, and the tokens it burned, not just a monthly total. Without that task-level view, the four metrics below are guesswork.

These metrics form a diagnostic system, not just a reporting layer. A low score on any metric points to where a problem is brewing.

Gartner^Ⓡ analyst Rita Sallam introduced a metrics framework specifically designed for agentic AI.⁵ The four metrics shift the unit of value from the token to the completed task.

These terms are not yet industry-standard shorthand, so each appears once with its Gartner label and then in common language afterward.

 Gartner terminology: Agent cost per completed task

The universal efficiency unit. Cost per completed task normalizes cost across consumption-based, fixed-conversation, and outcome-based pricing models, which matters when you are comparing vendors. Critically, it penalizes retry failures. A cheaper model that requires three attempts costs more per completed task than a slightly more expensive model that succeeds on the first pass. Artificial Analysis, a leading AI benchmarking service, now reports cost per task as a headline metric alongside its intelligence score, for exactly this reason.²

Gartner terminology: Context memory optimization score

Average input tokens consumed per agent per completed task. Lower is better. A persistently high context efficiency score, even after retrieval architecture improvements, is a leading indicator of poor data quality, knowledge graph immaturity, or inefficient reinjection patterns. It is the direct readout of retrieval quality.

Gartner terminology: Effective context utilization

The formula is task success rate multiplied by the accuracy score divided by the cost per completed task. Higher is better. Quality-adjusted efficiency catches the trap of optimizing cost in isolation because an agent that is cheap but frequently inaccurate scores poorly. It forces quality and cost into the same frame, where they belong.

 Gartner terminology: Agent value multiple

This is the metric that connects technical performance to board-level conversation. By adding cost savings to incremental revenue and incremental margin and then dividing all of that by total cost, you get your business value multiple. Note that inference spend alone is not total cost. You need to include the full lifecycle cost, covering development, integration, data management, security, and governance.

A low quality-adjusted efficiency score combined with a high context efficiency score (high input tokens per task) signals a retrieval problem, not a model problem. A business value multiple that plateaus while cost per task climbs shows that workflow embedding has stalled, meaning the agent is deployed but not integrated into how work actually flows.

So how do you improve your performance across the metrics we discussed above? I’ve found that each metric has one or more levers that, when put into practice, help increase the value and decrease the cost of agentic AI, improving the metrics.

Poor retrieval is the most direct driver of cost inflation in agentic AI. Context rot, the degradation of model focus when context windows are flooded with low-signal information, causes agents to compensate with brute-force token consumption, driving context efficiency scores up and quality-adjusted efficiency down. Gartner predicts that by 2027, organizations prioritizing semantics in Al-ready data will increase their agentic Al accuracy by up to 80% and reduce cost by up to 60%.⁶

You want to aim for real-time data sync, chain-of-thought context validation, and multisource retrieval through model context protocol (MCP) architectures to produce a step-change in reliability and cost. Open source frameworks such as LangChain can connect to retrieval and tool infrastructure through these open standards, enabling enterprise teams to build on a shared, interoperable foundation.

The practical goal here is to reduce what the agent must consume to answer correctly, favoring precision over volume. Hybrid semantic-keyword retrieval, semantic chunking, and structured tool definitions all reduce context bloat without sacrificing coverage. Each percentage point of reduction in context per task is a direct reduction in cost per completed task and an improvement in quality-adjusted efficiency.

AI pilot purgatory is expensive. Organizations run isolated experiments that generate impressive demos and flat ROI. Workflow transformation is all about moving from point applications disconnected from actual workflows to AI assistants integrated into tools employees use daily and, ultimately, agents that initiate, execute, and complete multistep tasks autonomously across systems.

The critical discipline at this stage is AI FinOps, which means engaging finance early, building clear ownership of benefit tracking, and creating processes that convert time saved into financial outcomes. An agent that saves 25% of an analyst’s time only improves the business value multiple if that time translates to a measurable result, such as more accounts processed, faster incident resolution, or a higher conversion rate. Time saved is a leading indicator, but financial outcome is the metric.

If your business value multiple plateaus while cost per task climbs, the diagnosis is almost always shallow embedding, where the agent assists a workflow but does not replace manual steps or generate additional throughput.

Multimodel cost routing is the most underutilized lever in enterprise AI, and it operates at the step level, not the program level.

Within a single provider, model cost can vary roughly 5 to 30 times (think a lightweight model like Claude Haiku versus a frontier model like Claude Opus), and across providers the cheapest capable open source models can cost around 100 times less per token than the most expensive frontier models.⁷ Most workflow steps do not require the most capable model. They require the minimum reliable model for that step’s task risk and context requirements.

Cost routing means choosing the cheapest reliable model per step, given task risk, context length, latency tolerance, and failure cost. 

That means you can reserve high-capability models for synthesis, judgment, and high-stakes decisions, and use the lowest-cost reliable model for everything else. The routing decision must also account for retry cost, since a cheaper model that fails more often can be more expensive per completed task than the model one tier up. The evidence is strong. Academic research from LMSYS achieved an 85% cost reduction while holding quality at 95% of GPT-4,⁸ and AWS Bedrock Intelligent Prompt Routing reports 56%–64% cost savings in its own evaluations without a drop in response quality.⁹

Matching model to step requirements rather than over-provisioning capability lowers cost per task without sacrificing quality.

Governance failures in agentic AI are performance failures, not just compliance failures. An agent with over-broad permissions creates security exposure and compliance liability. An agent with over-narrow permissions creates task failures and escalation loops. Both inflate cost per completed task, one through remediation cost and the other through retry and human handoff cost.

Gartner predicts that by 2027, 60% of organizations will fail due to incohesive data governance frameworks.¹⁰ The maturity path moves from manual security and acceptable-use policies to automated personally identifiable information (PII) redaction in retrieval augmented generation (RAG) pipelines and cross-functional AI governance councils, to permissions-aware agents that dynamically filter their outputs based on the requesting user’s role and access level, with reasoning guardrails and adversarial testing embedded in the development process.

The diagnostic metric is cost per task variance, the fluctuation of cost per task over time. Well-governed agents behave predictably and their cost per task is stable. Poorly governed agents generate unpredictable escalation and remediation events, and variance climbs accordingly. 

Consider an agent blocked from a data source that retries across a dozen tool calls before escalating to a human. That is both a governance failure and a cost event, one that registers as a spike in cost-per-task variance well before it surfaces in a compliance report. Risk-adjusted business value multiple must include governance failure costs, covering breach remediation, compliance penalties, and human intervention at the boundary of agent authority. Governance is not a tax on the AI program. It is a cost driver that, when managed well, reduces variance and improves the predictability of business-value outcomes.

The final lever is organizational. Technical efficiency does not automatically produce financial outcomes. AI FinOps closes that gap. It requires finance involvement at project inception, explicit ownership of benefit tracking, and a disciplined process for turning agent output into cost reduction or new revenue.

You’ll start this improvement process with gains treated as soft benefits, typically less than 10% of working time saved, with no financial translation. As business effectiveness improves, time savings reach 25% and begin to convert to cost savings. And ultimately, AI initiatives generate incremental revenue and contribute to margin. Business value multiple becomes a genuine executive metric, not a cost-avoidance story but a growth argument.

Business value multiple is most revealing when it distinguishes an organizational problem (benefit attribution, finance ownership, measurement process) from a technology problem (model performance, retrieval quality, workflow embedding depth).

AI maturity is nonlinear. An organization can be highly mature in workflow transformation while still operating at a lower level of maturity in governance, creating blind spots that surface as liabilities the moment agents are granted more autonomy. That is why all five levers require independent assessment. Strong performance on one lever does not compensate for weakness in any other.

Boards and finance teams are already asking the right questions. What is our AI actually costing us? Where is the value going? Why are these initiatives stalling? The four metrics answer those questions, but only for organizations that have built the observability infrastructure to compute them from first-party data rather than vendor estimates.

Only about 17% of organizations have deployed AI agents today but more than 60% expect to within two years and an additional 42% plan to adopt them in the next 12 months, according to the Gartner^Ⓡ 2026 CIO survey.¹¹ The organizations that reach production with defensible ROI will be those that built the measurement infrastructure before scaling up, applied cost routing at the step level, embedded AI into real workflows rather than alongside them, and held governance to the same performance standard as retrieval or model selection.

The rest will continue watching their bills triple while token prices fall.

The principles in this article are platform-agnostic, but the implementation choices are not. Here is how we believe we have applied this framework in practice. 

• Performance data. We use Elastic LLM observability — built into the Elasticsearch Platform — to capture per-model token consumption, cost, latency, and error rates across major providers (OpenAI, Azure OpenAI, Amazon Bedrock, Google Vertex, Anthropic). It converts that raw telemetry into per-task dollar cost, the first-party data we need to compute the four metrics from real observations rather than billing summaries. In one published walkthrough, that instrumentation caught a 2.4 times jump in cost after a model swap and reported it as a concrete number rather than a vague sense that costs had risen.
• Multimodel cost routing. The same telemetry that tracks cost per trace enables model selection analysis, identifying which model choices at which workflow steps are driving cost per completed task up or down. This creates the observability foundation for step-level cost routing decisions.
• Governance signals. Guardrail and AI-safety signals in Elastic LLM observability provide the data inputs for risk-adjusted business value multiple, surfacing policy blocks, refusals, and escalation events as cost signals rather than one-off incidents.
• Context engineering. Elastic’s acquisition of Jina AI added embedding and reranking that put only the most relevant context in front of the model. In our published benchmark, that cut context size by more than 40% while holding retrieval coverage steady, which lowers the context efficiency score and the cost per task. Because Elasticsearch speaks the open MCP standard, context engineering works with whatever agent framework our team already runs, from LangChain and LangGraph to Nous Research’s Hermes, rather than locking us into one.

Looking further out, we are building toward a shared, governed context layer that sits between enterprise data and any AI agent. Instead of rescanning raw data on every run, agents would retrieve precomputed, source-linked context in a single call, reaching better answers at far lower token cost from a knowledge base that gets stronger the more teams use it. It comes back to the principle that runs through this article. Less context, but better context, lowers cost while keeping accuracy high.

First-party telemetry is the common thread and the prerequisite for everything else. You cannot route models you cannot measure. You cannot attribute value to a workflow you cannot trace. Build the measurement layer first.

This metric-and-lever framework gives IT leaders a structured lens for the questions boards and finance teams are already asking. What is our AI actually costing us? Where is the value going? Why are these initiatives stalling?

In our view, the four metrics turn that vague sense into numbers a leader can defend. Cost per completed task, context efficiency, quality-adjusted efficiency, and business value multiple each measure something a monthly bill hides, and a weak score on any of them points to the lever that will move it. Track them over time and the ROI question stops being a guess. You can see where the money goes, prove where the value lands, and pull the right lever when an initiative stalls.

1. Cockroach Labs, “The Bill Arrives: How to Manage Agentic AI Costs at Scale,” 2026.
2. Artificial Analysis, “Artificial Analysis Benchmarking Methodology.”
3. Stanford Digital Economy Lab, “How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks,” 2026.
4. Anthropic, “Prompt caching with Claude,” 2025.
5. Gartner, Inc. "With Al Agents, You Need a New Way to Calculate Cost and Value," Rita Sallam. 2026.
6. Gartner, Inc. “Gartner Says Lack of Semantics Causes Inaccurate AI Agents and Wasted Spending,” 2026.
7. Anthropic, “Claude Platform Docs: Pricing.”
8. LMSYS, “RouteLLM: Learning to Route LLMs with Preference Data,” 2025.
9. AWS, “Use Amazon Bedrock Intelligent Prompt Routing for cost and latency benefits,” 2025.
10.  Gartner, Inc. “Adopt a Data Governance Approach That Enables Business Outcomes.”
11. Gartner, Inc. “What the 2026 Hype Cycle for Agentic AI Reveals,” Rajesh Kandaswamy, 2026.

GARTNER is a trademark of Gartner, Inc. and/or its affiliates.