7 lessons for IT leaders on using observability to monitor AI applications

Teams under pressure to show AI progress should not treat telemetry as a phase-two concern. By the time leadership asks for a return figure, the usage history that would have answered that question needs to already have been captured. If it hasn’t been captured, there may not be a way to reconstruct it. Data can age out. Sampling can discard what was never flagged as important.

According to the Elastic Landscape of Observability report, while 93% of organizations report financial and business impact to leadership in some form, only 19% do so regularly as part of an established process.³ The rest report occasionally or only when asked, which means the figure gets assembled as a one-off under deadline from whatever is available.

Pilots get funded on a projected return and need to be launched with a definition of success, so there is a threshold to measure against. Decide what success looks like before the first user arrives, and instrument for it in the same sprint. ROI should be a query, not an anecdote.

Uptime and latency were sufficient metrics when point A to point B produced the same result every time. However, a generative application can be fully available, comfortably fast, and totally wrong. Teams that catch quality problems track a second set of signals alongside the traditional ones, such as token consumption because it is the cost meter; retrieval quality because a grounded answer depends on what was retrieved; and use-case classification because knowing what people actually ask is the fastest way to quickly course correct.

These signals matter most after launch. Launch instrumentation tells you if your app is running okay. But as real users interact with the application, they surface a different and equally important question: Is this version better than the last? Nothing in that instrumentation can provide an answer.⁴ Swap a model, rewrite a prompt, or change the retrieval strategy, and you need a system that tells you whether you made things better or only made them different. That is a comparison, and a comparison needs a record of what came before. And, a test suite only answers part of it, which are the cases you thought to capture. The questions users actually ask drift constantly, and the curated set needs to constantly evolve from live telemetry. The baseline has to exist before you know you need it.

When a team creates an AI application by hand, someone has to name every field it emits like the token counts, the model, the tool calls, and the retrieval steps. Every dashboard, alert, and cost report that follows is keyed to those invented names, and the debt comes due at the moment of change. If an agent framework is swapped, a second model provider is added. Or if a new company is acquired, the reporting layer has to be rebuilt.

OpenTelemetry's semantic conventions for generative AI exist to prevent this. They define standard names for the operations that matter, including model calls, agent invocations, tool executions, and retrievals. Adoption is moving in one direction: OpenTelemetry in production nearly doubled year over year, and 60% of organizations now use a vendor distribution, which is up from 44% according to the Landscape of Observability report.

No generative AI span, metric, or attribute in that specification is marked stable yet, and attributes have already been renamed between versions.⁵ However, using a moving standard still beats using a private one. A rename is a migration; a private vocabulary is a rebuild.

At some point someone will put your AI cost report next to the provider invoice and ask why they disagree. The answer is usually that the application worked out its own numbers instead of recording the ones the provider already sent back.

Three versions of this are common: 

• Token counts estimated with a tokenizer library when the provider's response carries the exact figure it will bill 

• Cost computed from prices hardcoded in application code, which go stale with every model release 

• Duplicate instrumentation emitting two records of the same call, quietly doubling every downstream count

Estimation also fails in ways a dashboard cannot show. Tokenizers differ by provider, so the same prompt counts differently against different models. Reasoning models are billed for intermediate work the caller never receives, so a count taken from the visible response omits part of what is charged and omits most on the calls that cost the most.⁶

In an audit of a reference environment our field team built this year, we found and removed all three, including token estimates that disagreed with the provider's actual figures and duplicated spans that inflated call counts. The correction was not more code; it was less. Let the instrumentation report what it already knows, then compute derived values like cost at query time against a price table so that a price change is a data update rather than a deployment.

Observability spend is a top priority for leadership in 2026: 97% of organizations have already hit unexpected costs or overages, and 67% say it happens regularly. AI workloads make that worse because a request that once produced a handful of spans can now produce dozens.

Two decisions cause most of the damage, and both are made below the level where anyone reviews them. 

1. Tagging metrics with values that grow with usage: A conversation identifier is the classic case. Every new conversation then adds to what the monitoring platform has to store and index, so the observability bill starts tracking adoption instead of infrastructure. The application succeeds and the monitoring gets more expensive. 

2. The remedy teams reach for once the bill arrives: This keeps a flat percentage of everything. A uniform sample discards failures and successes in the same proportion, and the failures are the entire reason the telemetry exists.

Neither problem needs exotic tooling. Schema and sampling policy are architectural decisions with a budget attached, not defaults inherited from a configuration file.

In our experience, most quality failures in an AI application are not model failures. They are data and retrieval failures that surface as model failures.

An application grounded in your own content answers from documents it pulled at query time. When the answer is wrong, the only visible artifact is the model's output, so that is what gets investigated. Teams rewrite the prompt, then escalate to a larger and more expensive model, and the answer improves slightly or not at all because the model was working from what it was given. We have seen this in our own testing. In the same reference environment, an answer failed a quality review because the documents retrieved to support it covered three of the six policy areas the question asked about. No model swap recovers a document that was never retrieved.

None of that is visible unless the application records it. What was retrieved, how it ranked, and how much of the answer it actually supported are properties of the request rather than the model, and they exist only at the moment of the call. Capture them and quality debugging changes character. You can tell a model problem from a ranking problem from a content problem, and only the first is solved by changing models.

The old rule still holds — garbage in; garbage out — now delivered as a fluent and confident hallucination. We spent a year building a knowledge-centric support process with engineers writing knowledge articles as part of case closures. That foundation made the Support Assistant truly valuable. Better data raised the ceiling. Measuring the retrieval is what lets us watch the ceiling move.

A complete cost picture answers what the AI spent. It says nothing about whether the AI was right. These are different disciplines, and the second is the one most organizations defer indefinitely. Machine learning engineer Hamel Husain, who consults with AI product teams on evaluation, argues that unsuccessful AI products almost always share one root cause: "a failure to create robust evaluation systems."⁶ Systematic evaluation of groundedness, faithfulness, and tool use is not a quarterly exercise. It belongs wired into the same trace data that carries performance and spend, so a quality score points back at the run that produced it.

Cost visibility on its own can also mislead because the number most dashboards show has the wrong denominator. Price per call is a unit cost. What the business actually buys is a resolved request. A weaker model lowers the first figure and can raise the second because work that fails on the first attempt comes back as a retry, a follow-up question, or a longer chain of tool calls, and each of those bills again. The savings show up immediately and in the obvious place. The offsetting spend shows up later in a different line and often in a different team's budget. So, the two savings don’t seem connected. 

The remedy is a change of denominator. Report cost per completed task — total spend divided by the tasks that actually succeeded — and the cheaper model that needs three attempts stops looking cheap. Keep the quality verdict on the same trace as the cost, so a finance review and a quality review are reading the same record. There is a compact version of that pairing for anyone who wants a single number: task success rate multiplied by accuracy, then divided by cost per completed task. For the Support Assistant, success meant answering a question without a ticket being opened.

An assistant that answers badly wastes a moment. An agent that reasons badly acts on it across multiple steps with tool calls at a cost that accrues per task rather than per question. The usage history that was never captured becomes an action history that cannot be explained. The baseline that was never recorded becomes judgment drift that cannot be seen. 

The operational question shifts from what the model said to what the agent did and why. Everything in this piece has been preparation for that question, and the preparation is a single decision made before the next AI initiative ships; its value will be measurable from the first event. 

An organization that cannot measure what an assistant returned has no basis for handing an agent the authority to act.

Dive deeper into how leading organizations are tackling the most pressing challenges in AI model lifecycle management in our on-demand webinar, The new competitive advantage: Using observability to monitor AI applications.