How IT leaders can measure the ROI of agentic AI initiatives

Traditional ROI models, designed for predictable software implementations, don’t translate well to agentic AI. Unlike traditional tools, agentic AI systems are nondeterministic — they don’t just follow scripts, but reason and adapt. This variability impacts both costs, such as token consumption and reasoning steps, as well as value, such as quality, reliability, and scalability of outputs.

For example, an agent might take three steps to solve a customer query one day and ten steps the next, depending on the complexity. This unpredictability makes traditional cost-per-seat or cost-per-license models inadequate. To truly measure ROI, IT leaders must adopt a financial framework that captures both the variable nature of costs and the nuanced value generated by agentic AI.

To successfully measure ROI, organizations need to gather the right information upfront. These questions are essential to creating a foundation for assessing agentic AI value:

1. How do upfront and ongoing costs differ in agentic AI projects? Upfront costs include licensing, pilot development, and data cleaning, while ongoing costs like token consumption and monitoring are more variable. Understanding these differences is crucial to calculating total cost of ownership (TCO).

2. What is your "human baseline"? Before deployment, you should document metrics like process times, error rates, and cost-per-task for the tasks the AI will handle. Without this baseline, we would be making our best guess when measuring improvements.

3. What are the challenges in quantifying agentic AI benefits? Intangible values — like reduced employee burnout or improved scalability — can be difficult to measure. Same with quantifying reliability and quality improvements. Clear metrics and benchmarks are needed to make these benefits tangible.

4. How are you protecting your ROI? It's impossible to eliminate risk, but how do you get to acceptable levels of risk? As we venture deeper into embedding agentic AI into our workflows, we're thinking about a “least privilege, least function, and least exposure” approach. Proper architecture and controls are their own conversation, but should be highlighted here as a critical prerequisite to ROI realization.

When assessing ROI, it’s critical to use metrics tailored to the unique dynamics of agentic AI. Here are the core areas to evaluate:

• Agent value multiple (AVM): Add cost savings, incremental revenue, and margin improvements, then divide by total cost. This quantifies the efficiency of your investment.

• Cost avoidance: Measure costs you didn’t incur, such as avoiding new hires due to increased efficiency. By shifting work to agents you will likely be able to consolidate SaaS vendors and right-size license costs.

• Agent cost per completed task (ACCT): This normalizes costs by calculating the total expense required for a successfully completed task, no matter how complex.

• Context memory optimization score (CMOS): This metric tracks how many input tokens your agent needs to complete a task. Lower token usage means lower costs. By optimizing your design and using knowledge graphs, you can manage token consumption and avoid unexpected expenses.

• Success rate: Evaluate the percentage of workflows resolved without human intervention. High success rates translate into greater autonomy and efficiency.

• Effective context utilization (ECU): A composite metric combining task success rate and accuracy relative to cost, ensuring agents operate efficiently and reliably.

• User acceptance: For internal agents, track whether employees are actually using the solution or reverting to manual methods.

To calculate the total cost for your calculations, you must look beyond the more visible costs of agentic AI. According to Gartner, these are the hidden agentic AI costs you may be overlooking:

• Application development and ongoing maintenance: Customizing AI solutions to align with specific business needs often requires substantial development time and expertise. Beyond initial deployment, organizations must account for regular updates, performance tuning, and maintenance to ensure continued alignment with evolving business objectives and technological standards.

• Integration with existing enterprise systems: It goes without saying that it's an uphill battle to integrate new tech, like agentic AI, into your current tech stack. Complexities may arise while connecting your new AI models with legacy systems, workflows, or other business-critical platforms. This work often requires additional tools, time, and significant expertise to achieve operational readiness.

• Security and governance frameworks to mitigate risks and accelerate innovation: These frameworks are essential to enable your organization to accelerate innovation by creating a secure foundation for deploying AI confidently without compromising data privacy or violating regulatory standards. They require time and money to build.

• Data management for accuracy at a viable cost: Agentic AI systems require high-quality, relevant data, but managing that data effectively comes with significant costs. From ensuring data accuracy and consistency to managing storage and retrieval, organizations need solutions that optimize these processes while maintaining affordability. Note that the importance of this investment cannot be overstated. Avoid reasoning debt — messy data that causes agents to think too hard and consume too many tokens — and ensure that you’re making the appropriate investments in your data architecture. 

• Business change and transformation required to fully realize the business value: Fully realizing the value of AI requires investment in business transformation efforts. You can put all the resources into building it, but will they use it? This hidden cost includes employee training, fostering a culture of AI adoption, and aligning teams to new operational processes. Underestimating the scale of change management can delay adoption and result in underutilized AI solutions — which ultimately leads to less ROI for your agentic AI.

By carefully evaluating and budgeting for these overlooked costs, IT leaders can establish a clear path toward achieving robust, scalable, and cost-effective agentic AI solutions that deliver measurable business value.

Don't rely on back-of-the-napkin math. Here are a few ways to predict agentic AI costs:

• Simulation tools: Use pilots to simulate token consumption across complex and simple queries. This helps forecast variability in compute costs.

• FinOps for AI: Partner with your finance team to develop processes for cost-benefit analysis and reporting. Having support from a team who can help show the value of your AI agents is essential. 

Agentic AI will require organizations (and finance teams) to think differently about the traditionally fixed IT budget. AI compute costs need to be weighed directly against the efficiency gains produced by agent outcomes, and both of these aspects will vary. Yes the agent consumed 50k tokens, but it also closed 5k tickets without human intervention — so, how much did it really cost? The nondeterministic, variable nature of agentic AI will require us to model more variability into traditionally fixed cost projections.

Agentic AI is poised to transform how organizations operate, but success depends on careful planning and measurement. How do you assess the value generated by agentic AI initiatives? By answering key questions, setting baselines, using next-gen metrics, and accounting for hidden costs, IT leaders can move beyond pilots and ensure agentic AI delivers sustainable business value.

Ready to learn more about measuring agentic AI ROI? Get the Gartner report.