Machine learning, generative AI, and LLMs in observability

For site reliability engineers (SREs), AI promises to accelerate discovery, diagnosis, and insight. However, only good data gets great results.

Today’s production environments aren’t defined by a handful of services, predictable traffic patterns, or static infrastructure. Modern systems are cloud-native, distributed, and continuously changing. Kubernetes orchestrates fleets of ephemeral workloads. Microservices communicate across dynamic networks. Code deployments happen multiple times a day. And in an explosive number of new apps, code doesn’t go through a full review cycle anymore. Increasingly, AI-powered applications introduce non-deterministic behavior into production systems.

So, observability is at an inflection point.

While applications continue to generate large amounts of telemetry data — logs, metrics, distributed traces, and profiles — traditional monitoring cannot give your team a holistic picture of your environment, especially with AI.

Familiar pain points for SRE teams include:

1. Too many alerts, not enough clarity
2. Too much data, too little context
3. Faster incidents, slower understanding

Observability must evolve from displaying data to interpreting and acting on it. AI is leading this shift.

The arrival of generative AI (GenAI) — powered by large language models (LLMs) — is creating a simpler path forward. GenAI and AI assistants are changing how engineers interact with observability altogether. Instead of fixed/static dashboards and queries, SREs increasingly use natural language to investigate incidents and understand system behavior.

Now, agentic AI is garnering attention from both developers and executives. Agentic AI represents a shift from reactive operations to proactive, adaptive reliability engineering. AI agents can observe systems, reason across telemetry, and take action through workflows and remediation under human-defined guardrails. 

However, to get the most out of GenAI and agentic AI, teams need to focus on the foundation of this whole operation: data.

Before SRE teams can successfully apply machine learning (ML), GenAI, or LLM-powered assistants to observability, they must solve a more fundamental problem: how telemetry data is collected, structured, stored, and connected.

A gold mine for effective and efficient AI, telemetry data is:

• Reliable: Telemetry reflects real system behavior.
• Context-rich: Logs, traces, and metadata are necessary for causal inferences.
• Real-time: Data arrives continuously.
• Deep: Long-term telemetry storage enables trend analysis and prediction.

As LLMs enable GenAI to process natural language, logs have resurfaced in importance. Once considered unwieldy due to their unstructured nature and verbosity, they are now invaluable diagnostic tools for the truth. Logs are naturally the primary source of context that GenAI tools can analyze and interpret to identify and explain issues. 

This is why your data foundation matters. Every layer of the modern observability stack emits valuable data. That includes application logs, infrastructure metrics, distributed traces, events, and business signals.

For traditional monitoring, this explosion of telemetry data, and unstructured data in particular, creates noise that results in: 

• Slower incident investigations and prolonged downtime
• Ballooning storage costs with diminishing returns
• Significant human effort spent searching instead of understanding
• Operational drag that slows innovation and delivery

However, GenAI is not a band-aid for an already-broken observability stack. In fact, it amplifies any weaknesses in your system. Without consistent context, reliable correlations, and economically sustainable data retention, even the most advanced AI models can produce misleading insights.

The three advances paving the way for AI-powered modern observability: cost-effective, performant storage; OpenTelemetry (OTel); and GenAI.

For years, log management forced painful tradeoffs. Teams either retained rich data at high cost or aggressively sampled and stripped context to save money, often creating blind spots during incidents.

The signals traditionally labelled “noise” and discarded — ahem, logs — are necessary to understanding system behavior patterns or predicting failures. They make up the web of rich, contextual data that ML and AI systems thrive on. 

Now, retention is no longer a luxury. Cloud object storage, compression, and indexing make petabytes of data easily searchable at low cost — especially when deployed with an intelligent data tiering and lifecycle strategy.

Modern storage architectures, particularly those leveraging object storage and advanced compression techniques like Zstandard, can achieve remarkable cost-to-value ratios. The secret is organizing related data together and moving it to cheaper storage tiers quickly. This approach lets you have your cake and eat it too — full fidelity data retention without breaking the bank.

AI requires history and depth. Pattern recognition, anomaly detection, and predictive analysis all depend on long-term data. Cost-effective storage makes it practical to analyze real-time and historical telemetry side by side.

As for logs, they are an indispensable fuel for reliable analytics and AI-driven observability. With cost-effective storage, you can afford modern observability built on high-fidelity telemetry data.

If AI is the brain of modern observability, OTel is the nervous system. It provides the shared structure and context that allows observability insights to emerge.

OTel defines open, vendor-neutral standards for collecting logs, metrics, and traces across languages, platforms, and environments. More importantly, it enforces consistency — the single most important requirement for scalable AI-driven analysis.

Why OTel changes everything

• Vendor neutrality: Instrumentation is portable. You own the data you generate and will work with any OTel-compliant observability tooling. This is essential for long-term AI strategies that will evolve over time.

• Standardized semantic conventions: Telemetry signals share consistent metadata such as service names, resource attributes, deployment versions, and environment identifiers. This shared language makes context and correlation possible at scale, no matter how messy your environment gets.

• Auto-instrumentation: OTel can automatically capture rich logs and traces for many common frameworks, dramatically lowering friction and deployment effort.

For logs in particular, this is transformative. Logs are easily enriched with trace IDs, span IDs, and resource attributes, evolving from noisy text files to first-class, correlatable signals. 

Next-generation signals: Wide events

Effective modern correlation turns raw data into a navigable narrative. Thanks to OTel’s shared semantic conventions, the automatic correlation of a log from one service to the same event’s metrics and traces is now possible.

This contextual linkage is emerging as a new type of signal: the wide event. Wide events are rich, context-aware records that span services, infrastructure, and time. Instead of fragmented signals, teams get a coherent narrative of what happened and why for investigations.

Wide events preserve the context required for ML and LLMs to analyze and reason accurately across systems. Work continues on an open standard for wide events in OTel, but expect even more effective investigations from GenAI in the future.

Several observability workflows are already well served by AIOps capabilities:

• Anomaly detection: Adaptive baselines detect sudden or unexpected changes in latency, throughput, or error rates.

• Classification and summarization: Massive volumes of data (including unstructured data types) can be automatically grouped, categorized, and summarized, dramatically reducing analysis time.

• Event and symptom correlation: Clustering related alerts, metrics anomalies, and events cuts down alert noise and accelerates root cause identification.

• Pattern-based root cause surfacing: For well-understood failure modes, AIOps can automatically identify and surface likely root causes.

AIOps is already the standard in the observability space, democratizing access to analytics and insights while significantly improving engineers’ ability to identify and troubleshoot issues at scale.

By structuring data, reducing noise, and embedding intelligence into observability workflows, AIOps tells you something went wrong and prepares organizations for more advanced capabilities from GenAI: LLM-powered assistants and agentic AI. For SRE managers, mastering AIOps is the first step in building AI-native operations. It is where observability stops being reactive and begins to scale with the systems it monitors. ML and AIOps are not basic. Rather, they’re the base layer of modern observability.

What is retrieval augmented generation (RAG) for observability?

The critical architectural concept that connects internal knowledge to the LLM for AI assistants is retrieval augmented generation (RAG). LLMs by themselves do not know how your systems are built, what your runbooks say, or how past incidents were resolved. RAG grounds AI responses in your data by retrieving relevant internal information and supplying it as context.

RAG is a technique that consists of supplementing LLMs with proprietary data. RAG gives SREs the ability to combine telemetry data, historical incident data, runbooks, and internal organizational knowledge bases into a unified context for GenAI reasoning and resolution.

For observability, this means responses can reference actual system state and historical patterns, including similar past incidents, documented remediation steps, and known service dependencies. RAG gives observability data a real-time knowledge source that AI can reason over, enabling trusted responses tailored to your environment. The result is an assistant that generates organization-specific, situation-aware answers combined with knowledge available in public domains.

Automated documentation

Effective incident response extends beyond technical remediation. Stakeholders need timely updates, clear explanations, and confidence that issues are being handled.

AI assistants can generate summaries and structured incident communications on demand. Take the prompt: “Draft an incident update with root cause, impact, and next steps.” An AI assistant can assemble a coherent message, complete with relevant links and timelines, and deliver it via email or chat integrations.

These communications can later serve as the foundation for post-incident reviews, saving hours of SRE work.

Beyond faster incident resolution, AI assistants provide a broad impact, including:

• Improved and more effective SLOs
• Faster understanding and validation of issues
• Lower cognitive load on-call 
• Better support for junior engineers
• Preservation and reuse of institutional knowledge

With effective AI assistants, incident response shifts from a source of constant stress to a repeatable, scalable capability.

LLMs are remarkable at synthesizing, generating language, and understanding issues. But alone, they do not meet the demands of operational reliability for observability teams.

Dreaming of a proactive and autonomous system? That dream is closer than ever as GenAI evolves into its next iteration: agentic AI. A quickly evolving discipline, agentic AI has birthed AI agents: digital teammates that can reason and act more autonomously, leveraging other models, data sources, and a knowledge base.  

AI agents represent a new generation of observability, one that leverages GenAI, agentic architectures, and automated workflows. They not only identify issues but also independently resolve them intelligently and at scale.

Traditional AI systems respond to prompts; agentic AI systems behave as active participants in observability. An AI agent continuously observes telemetry, reasons across context, investigates, and takes action toward defined operational goals.

What are AI agents in observability?

AI agents are powered by agentic technology and LLMs — even sometimes referred to as LLM agents. Where LLMs enable conversational communication between agent and user, agentic technology enables reasoning, planning, decision-making, and action. 

How are AI agents different from AI assistants?

AI agents:

• Maintain internal state and reasoning loops
• Plan and decompose tasks into subtasks
• Coordinate across multiple specialized components
• Act autonomously under defined policies

An AI agent works like an autonomous team member with logic: it knows the goal, evaluates the environment, and takes steps toward resolution.

In observability, a basic, effective agentic workflow looks like this:

1. Detect anomalies or signals in telemetry streams.
2. Diagnose probable root causes and correlations.
3. Decide on action plans based on confidence and policy.
4. Act by triggering workflows, remediations, or alerts.

Imagine a scenario where a GenAI agent correlates a latency spike with a recent deployment. It searches and identifies the regression across recent logs and traces, post-deployment. And it decides on a series of actions, including recommending a rollback for operator approval. 

Human SREs are still extremely important. They are needed to validate the root cause of an issue and manually make the decision to roll back the code.

MCP is an open source protocol that standardizes agent access to tools. In practice, AI agents can query observability databases, trigger workflows, retrieve telemetry, and generate analytics while enforcing access control and logging every action.

Today, agent engineers and developers rely on a patchwork of brittle, high-friction workarounds to connect agents to internal tools. Custom metadata formats, inconsistent interfaces, and poor tool discoverability limit interoperability, making it costly and error-prone to adapt agents to new systems, often requiring extensive retraining or reprogramming. 

MCP creates a predictable and reliable ecosystem for developers by introducing clear, shared standards across the agent-tool boundary. It ensures that AI is not a rogue agent with unfettered privileges, but a tool-aware process that respects enterprise governance and security constraints.

As agentic AI is embraced within observability, the age-old question still stands: buy or build? Vendor-built agents may be suitable for basic tasks, but they don’t always translate well to the unique combination of technologies, tools, and workflows in each application environment. 
AI agent builders provide an alternative. They use a natural language chat interface to help create custom agents and tools for your environment’s needs and use cases. In this framework, SREs are not replaceable, but rather an integral part of the agentic AI evolution.

As organizations embed LLMs and GenAI into production workflows, they need observability for AI, not just with AI.

AI systems themselves become mission-critical workloads for observability and other operational teams. They introduce new failure modes, cost dynamics, and security risks that traditional observability was never designed to handle.

To operate AI safely at scale, SRE teams need LLM observability.