The first time most teams learn that their OpenAI project is close to a rate limit is when production traffic starts getting throttled with HTTP 429 responses. OpenAI enforces rate limits at the project level, not at the organization level, so a single noisy workload in one project can saturate that project's RPM or TPM ceiling while the rest of the organization still has plenty of room. Without OpenAI rate limit monitoring that compares configured limits against actual consumption, headroom is invisible until it runs out.

OpenAI API monitoring in Elastic: what's new

We're pleased to announce updates to the Elastic OpenAI integration. On top of the existing token usage and audit log coverage, the integration now polls OpenAI's List project rate limits Admin API per project and rolls the results up into both per-project and org-wide views. A new openai.rate_limits dataset feeds two new dashboard panels and a prebuilt threshold alert rule, so teams can see how close each project and model is to being throttled before users experience production impact.

What the integration polls: Usage, Audit Logs and Rate Limits APIs

The Elastic OpenAI integration is built for teams running applications on the OpenAI API platform. The people accountable for it are the developers shipping those services, the platform and SRE teams keeping them running, and the finance and FinOps owners answering "how much is our software consuming, and are we within our capacity envelopes?"

The integration collects from three OpenAI Admin API surfaces:

• Usage API for usage counts across tokens, characters, seconds, sessions, bytes, and images, with project, model, user, and API key attribution where that Usage API surface provides it.
• Audit Logs API for organization audit events such as API key creation, project changes, and user activity.
• Rate Limits API for configured RPM, TPM, and IPM ceilings per project and per model, plus other limit dimensions where available; the new headroom views compare the per-minute request, token, and image limits against actual consumption.

Because everything is pulled from the Admin API at the organization level, platform teams get a unified view across every project, model and API key, alongside the rest of the telemetry they already monitor in Elastic, without touching application code or installing SDKs in every service.

What teams need to monitor when running on the OpenAI API

Four operational needs come up over and over for teams running production workloads on the OpenAI API.

Token usage attribution

A single OpenAI organization usually serves many internal teams and products, each with its own project, its own mix of models (GPT-5.5 Pro for the hardest reasoning tasks, GPT-5.4 for everyday traffic, GPT-5.4 nano for high-volume low-cost requests, and specialized models for images, audio and embeddings) and its own user and API key footprint. When usage patterns shift, the platform team needs to know which project, model and key is driving the change so they can attribute consumption back to the right team and decide which workloads should move to a cheaper model.

Rate limit headroom

OpenAI enforces per-model rate limits on requests per minute (RPM) and tokens per minute (TPM) at the project level, not at the organization level. The first time a team learns they're close to the ceiling is usually when production traffic starts being throttled. Surfacing configured limits alongside actual consumption, per project and per model, lets platform teams see headroom in advance, plan capacity, and request limit increases before users feel the impact.

Audit visibility

Security and compliance teams need to know who created API keys, who changed project settings, who invited or removed users, and when. The integration ingests OpenAI's organization audit log so those events land in the same Elastic deployment as the usage data, ready for correlation, alerting and long-term retention. Audit log ingestion has two prerequisites: audit logging must be enabled in your OpenAI organization, and the Admin API key used by the integration must belong to an Organization Owner, because OpenAI restricts audit-log access to that role. Without both, the openai.audit dataset stays empty.

Granularity for every audience

The same data needs to serve different cadences. SREs want one-minute resolution to catch spikes and trigger throttling alerts. Platform engineers want hourly views for capacity planning. Finance and FinOps owners want daily totals that roll up cleanly for reporting. A single integration that exposes all three granularities removes the need to maintain separate pipelines for each audience.

How does Elastic poll the OpenAI Admin API?

The integration runs on Elastic Agent and uses the CEL input to poll OpenAI's Admin API on a schedule. Authentication uses a single Admin API key, stored as an encrypted Fleet secret and redacted from agent logs. From a single configuration, the integration ingests datasets from three Admin API sources:

Usage API datasets (per project, model, user and API key, with each dataset tracking the unit OpenAI exposes for that workload):

• openai.completions for chat and completion token counts (input, output, cached, audio input/output).
• openai.embeddings for embedding token counts.
• openai.moderations for moderation token counts.
• openai.images for image counts and size dimensions.
• openai.audio_speeches for text-to-speech character counts.
• openai.audio_transcriptions for speech-to-text duration in seconds.
• openai.code_interpreter_sessions for code interpreter session counts.
• openai.vector_stores for vector store byte counts.

Audit Logs API dataset:

• openai.audit for organization audit events such as API key creation, project changes and user activity.

Rate Limits API dataset:

• openai.rate_limits (new) for snapshots of configured rate limits per project and per model, including RPM, TPM, IPM, and other limit fields where OpenAI returns them, paged across all active projects on each poll.

Ingest pipelines handle parsing and field mapping so the data lands queryable, dashboard-ready, and aligned with the rest of Elastic Observability. Because the data is pulled from the Admin API at the organization level, you get this visibility without any application-side instrumentation or SDK changes.

What you need to set up OpenAI monitoring in Elastic

To get started with the Elastic OpenAI integration, you will need:

• An Elastic deployment:
• Elastic Cloud Hosted (ECH) running a recent supported version, or
• Elastic Cloud Serverless, no version requirement, works out of the box.
• An OpenAI organization with Admin API access.
• An Admin API key provisioned by an Organization Owner from the OpenAI platform settings under Settings → Admin keys. Owner-level keys are required if you want the openai.audit dataset to populate.
• Audit logging enabled in your OpenAI organization, if you want audit data.
• Elastic Agent installed on a host with outbound HTTPS access to api.openai.com, or the agentless deployment option.

How to set up the OpenAI integration

1. Generate an Admin API key in the OpenAI platform settings.
2. In Kibana, go to Management → Integrations, search for OpenAI and click Add.
3. Choose your deployment mode: agentless for a zero-install experience, or Elastic Agent on your own host.
4. Tune the defaults if you need to. Each dataset has sensible defaults:

• Usage datasets poll every 5 minutes with 1-minute buckets. Each dataset exposes a finalization_grace setting that controls when a per-minute usage bucket is considered final. The default 0s favors freshness: a bucket is ingested as soon as its minute closes. The observed behavior (which OpenAI does not document, but the integration team measured against the live API) is that bucket counts can keep rising for some time after that point, so per-minute totals at 0s can undercount during heavy bursts. Setting finalization_grace to 15m, the recommended value for accurate counts, holds a bucket back until the grace window has elapsed and brings counts much closer to the Usage API, though a small residual undercount can remain during very high-volume bursts because OpenAI's per-minute counts can be revised upward beyond any fixed grace window. The cost is delaying dashboards and the rate limit headroom alert by the grace period.
• Rate limits polls every 5 minutes. Each poll captures the full set of configured RPM, TPM and IPM limits per project and per model.
• Audit logs polls on a separate cadence and ingests all org-level audit events. Remember the prerequisites: audit logging enabled in your OpenAI organization and an Organization-Owner Admin API key.

1. Open the integration assets in Kibana. Within minutes, usage, audit, and rate-limit data starts flowing, and the prebuilt dashboards and alert rule are ready to use.

For the full configuration reference, see the OpenAI integration documentation.

What do the OpenAI rate limit dashboards show?

The integration ships with a pre-built Kibana dashboard that gives you an immediate, queryable view of your organization's OpenAI API consumption. The overview pulls headline numbers (total tokens, total invocations, top models, top projects, top users and top API keys) into one place for a quick read on the state of your OpenAI usage. The screenshot below shows the OpenAI usage overview dashboard:

From the overview, you can drill into the views that answer the operational needs introduced earlier.

Token usage by model, project and user

The token metrics panels break down token consumption (input, output, cached input, audio input/output) for the token-based datasets (openai.completions, openai.embeddings, openai.moderations) by model and over time. This is the view that tells you where your token budget is actually going, which workloads are getting the most out of prompt caching, and which projects, users or API keys are driving the bulk of your token consumption. Filter by project or model to scope the view to a single team or product. Image, audio and vector-store consumption (measured in images, characters, seconds, sessions and bytes rather than tokens) is reported in dedicated sections of the same dashboard. The token metrics panels look like this:

Rate limit headroom: per project and model (new)

The new rate limit headroom panel joins the configured limits from openai.rate_limits against actual consumption from the usage datasets, per project_id and model. For each row it reports peak one-minute used, the configured limit, and utilization percentage for requests (RPM), tokens (TPM), and images (IPM). Rows are sorted by highest TPM utilization first, with RPM and IPM utilization as tie-breakers, so the highest token-pressure rows appear at the top of the list. Utilization is computed against the peak one-minute bucket in the lookback window, never a 5- or 15-minute sum against a one-minute ceiling, so the panel reflects peak-minute pressure against the one-minute ceiling instead of averaging it away. The per-project rate limit headroom panel is shown below:

Rate limit headroom: org-wide rollup by model (new)

Because OpenAI enforces limits per project, a single per-project view doesn't answer "how much total capacity do we have for gpt-image-2 across the organization?" The new org-wide rollup panel reports the same RPM, TPM and IPM metrics summed across all active projects for each model. Both the limit and the usage figures are indicative upper bounds rather than exact org-wide numbers (the limit is a sum of per-project ceilings; the usage is a sum of each project's peak minute, which may fall in different minutes across projects), but together they give platform teams a single number to plan against when they're sizing a new workload or deciding which project should absorb a new use case. The org-wide rollup panel is shown below:

Behind the scenes, version 2.3.0 also normalizes request and token counts into shared openai.base.usage_tokens and openai.base.usage_images fields across the usage datasets, so the headroom panels render correctly even when only a subset of usage datasets is enabled.

OpenAI audit log activity in Elastic

The audit panels surface organization audit events (API key creations, project changes, user invitations and login activity) so security and compliance teams can review who did what, when, alongside the usage data. The audit dashboard is shown below:

Out-of-the-box alert for rate limit headroom (new)

The integration ships with a pre-built threshold alert rule template, [OpenAI] Rate limit headroom low, ready to install in one click from the integration's Assets tab.

The default behavior is tuned to be useful out of the box:

• Runs every 5 minutes.
• Looks back over the last 15 minutes.
• Fires after 3 consecutive matches.
• Triggers when peak one-minute TPM utilization reaches or exceeds 80% of the configured project/model limit.
• Groups alerts by project_id::model, so an incident in one project on one model doesn't get lost in an org-wide aggregate.

The 80% threshold and other parameters are editable in Kibana after you install the rule, so each team can tune the alert to its own risk tolerance.

Custom OpenAI alerts and SLOs in Elastic Observability

As with every other Elastic integration, all the OpenAI metrics and audit data is fully available to leverage in every capability in Elastic Observability, including SLOs, alerting, custom dashboards and in-depth logs exploration.

For example, to keep token consumption under control across a single project, create a custom threshold rule that sums tokens from the relevant usage dataset and fires when the daily or hourly total crosses your budget. To track model mix, define an SLO in Elastic Observability that treats OpenAI requests on your approved lower-cost model families as the "good events" (the ones that count as meeting the target) and all OpenAI requests as the "total events", grouped by openai.base.project_id and openai.base.user_id. The ratio becomes your SLI; a 7-day rolling 80% target quickly surfaces projects and users overusing more expensive models.

Choosing OpenAI usage data granularity

OpenAI usage data collected by the integration powers different cadences, with a fidelity-versus-freshness tradeoff to be aware of. One-minute usage buckets feed the rate limit headroom alert and near-real-time throttling notifications when a project approaches its ceiling: with finalization_grace set to 0s (the default), per-minute counts arrive within minutes but can undercount during heavy bursts; raising finalization_grace to 15m brings counts much closer to reconciled at the cost of delaying the dashboards and alert by the grace period; a small residual undercount can still remain for the busiest buckets. Hourly views support operational monitoring and capacity planning across projects and models. Daily aggregates roll up cleanly for FinOps reporting and reconciliation. An out-of-the-box alert ships for rate limit headroom ([OpenAI] Rate limit headroom low), and the same data can be reused for custom usage and budget thresholds without building anything from scratch.

Get started with OpenAI monitoring in Elastic

The Elastic OpenAI integration is available today in Elastic Cloud, including Elastic Cloud Hosted and Elastic Cloud Serverless. To get started, sign up for a free Elastic Cloud trial, provision an Admin API key in the OpenAI platform settings, and add the OpenAI integration from Kibana under Management → Integrations.

Within minutes you'll have token usage, audit activity, and rate limit headroom data flowing into Elasticsearch, with the prebuilt dashboards and the new throttling alert ready to use.

Despite their widespread utility of LLMs, deploying these sophisticated tools in real-world scenarios poses distinct challenges, especially in managing their complex behaviors. For users such as Site Reliability Engineers (SREs), DevOps teams, and AI/ML engineers, ensuring reliability, performance, and compliance of these models introduces an additional layer of complexity. This is where the concept of LLM Observability becomes essential. It offers crucial insights into the performance of these models, ensuring that these advanced AI systems operate both effectively and ethically.

Why LLM Observability Matters and How Elastic Makes It Easy

LLMs are not just another piece of software; they are sophisticated systems capable of human-like capabilities such as text generation, comprehension, and even coding. But with great power comes greater need for oversight. The opaque nature of these models can obscure how decisions are made and content generated. This makes it even more critical to implement robust observability to monitor and troubleshoot issues such as hallucinations, inappropriate content, cost overruns, errors and performance degradation. By monitoring these models closely, we can safeguard against unexpected outcomes and maintain user trust.

Real-World Scenarios

Let's explore real-world scenarios where companies leverage LLM-powered applications to enhance productivity and user experience, and how Elastic's LLM observability solutions monitor critical aspects of these models.

1. Generative AI for Customer Support

Companies are increasingly leveraging LLMs and generative AI to enhance customer support, using platforms like Google Vertex AI for hosting these models efficiently. With the introduction of advanced AI models such as Google's Gemini, which is integrated into Vertex AI, businesses can deploy sophisticated chatbots that manage customer inquiries, from basic questions to complex issues, in real time. These AI systems understand and respond with natural language, offering instant support for issues such as product troubleshooting or managing orders thus reducing wait times. They also learn from each interaction to improve accuracy continuously. This boosts customer satisfaction and allows human agents to focus on complex tasks, enhancing overall efficiency. Other ways that AI tools can further empower customer care agents is with real-time analytics, sentiment detection, and conversation summarization.

To support use cases like the AI-powered customer support described above, Elastic recently launched LLM observability integrations including support for LLMs hosted on GCP Vertex AI. Customers who wish to monitor foundation models such as Gemini and Imagen hosted on Google Vertex AI can benefit from Elastic’s Vertex AI integration to get a deeper understanding of model behavior and performance, and ensure that the AI-driven tools are not only effective but also reliable. Customers get out-of-the-box experience ingesting a curated set of metrics from Vertex AI as well as a pre-configured dashboard.

By continuously tracking these metrics, customers can proactively manage their AI resources, optimize operations, and ultimately enhance the overall customer experience.

Let's look at some of the metrics you get from the Google Vertex AI integration which are helpful in the context of using generative AI for customer support.

1. Prediction Latency: Measures the time taken to complete predictions, critical for real-time customer interactions.
2. Error Rate: Tracks errors in predictions, which is vital for maintaining the accuracy and reliability of AI-driven customer support.
3. Prediction Count: Counts the number of predictions made, helping assess the scale of AI usage in customer interactions.
4. Model Usage: Tracks how frequently the AI models are accessed by both virtual assistants and customer support tools.
5. Total Invocations: Measures the total number of times the AI services are used, providing insights into user engagement and dependency on these tools.
6. CPU and Memory Utilization: By observing CPU and memory usage, users can optimize resource allocation, ensuring that the AI tools are running efficiently without overloading the system.

To learn more about how Elastic's Google Vertex AI integration can augment your LLM observability, have a quick read of this blog.

2. Transforming Healthcare with Generative AI

The healthcare industry is embracing generative AI to enhance patient interactions and streamline operational workflows. By leveraging platforms like Amazon Bedrock, healthcare organizations deploy advanced large language models (LLMs) to power tools that convert doctor-patient conversations into structured medical notes, reducing administrative overhead and allowing clinicians to prioritize diagnosis and treatment. These AI-driven solutions provide real-time insights, enabling informed decision-making and improving patient outcomes. Additionally, patient-facing applications powered by LLMs offer secure access to health records, empowering individuals to manage their care proactively.

Robust observability is essential to maintain the reliability and performance of these generative AI applications in healthcare. Elastic’s Amazon Bedrock integration equips providers with tools to monitor LLM behavior, capturing critical metrics like invocation latency, error rates, token usage and guardrail invocation. Pre-configured dashboards provide visibility into prompt and completion text, enabling teams to verify the accuracy of AI-generated outputs, such as medical notes, and detect issues like hallucinations.

Additionally, customers who configure Guardrails for Amazon Bedrock to filter harmful content like hate speech, personal insults, and other inappropriate topics, can use the Bedrock Integration to observe the prompts and responses that caused the guardrail to filter them out. This helps application developers take proactive actions to maintain a safe and positive user experience.

Some of the logs and metrics that can be helpful for customers using LLMs hosted on Amazon Bedrock are the following

1. Invocation Details: This Integration records the Invocation latency, count, throttles. These metrics are critical for ensuring that generative AI models respond quickly and accurately to patient queries or appointment scheduling tasks, maintaining a seamless user experience.
2. Error Rates: Tracking error rates ensures that AI tools, such as patient query assistants or appointment systems, consistently deliver accurate and reliable results. By identifying and addressing issues early, healthcare providers can maintain trust in AI systems and prevent disruptions in critical patient interactions.
3. Token Usage: In healthcare, tracking token usage helps identify resource-intensive queries, such as detailed patient record summaries or complex symptom analyses, ensuring efficient model operation. By monitoring token usage, healthcare providers can optimize costs for AI-powered tools while maintaining scalability to handle growing patient interactions.
4. Prompt and Completion Text: Capturing prompt and completion text allows healthcare providers to analyze how AI models respond to specific patient queries or administrative tasks, ensuring meaningful and contextually accurate interactions. This insight helps refine prompts to improve the AI's understanding and ensures that generated responses, such as appointment details or treatment explanations, meet the quality standards expected in healthcare.
5. Prompt and response where guardrails intervened: Being able to track requests and responses that were deemed inappropriate by guardrails helps healthcare providers monitor what information patients are asking for. With this information users can make continuous adjustments to the LLMs to ensure appropriate responses, balancing flexibility and rich communication on the one hand, and on the other, privacy protection, hallucination prevention, and harmful content filtering.

Amazon Bedrock Gaurdrails OOTB dashboard

To learn about the Amazon Bedrock Integration, read this blog. To dive deeper into how the integration can help with observability of Guardrails for Amazon Bedrock, take a look at this blog.

3. Enhancing Telco Efficiency with GenAI

The telecommunication industry can leverage services like Azure OpenAI to transform customer interactions, optimize operations, and enhance service delivery. By integrating advanced generative AI models, telcos can offer highly personalized and responsive customer experiences across multiple channels. AI-powered virtual assistants streamline customer support by automating routine queries and providing accurate, context-aware responses, reducing the workload on human agents and enabling them to focus on complex issues while improving efficiency and satisfaction. Additionally, AI-driven insights help telcos understand customer preferences, anticipate needs, and deliver tailored offerings that boost customer loyalty. Operationally, LLMs such as Azure OpenAI enhance internal processes by enabling smarter knowledge management and faster access to critical information.

Elastic's LLM observability integrations like the Azure OpenAI integration can provide visibility into AI performance and costs, empowering telecom providers to make data-driven decisions and enhance customer engagement. It can help optimize resource allocation by analyzing call patterns, predicting service demands, and identifying trends, enabling telcos to scale their AI operations efficiently while maintaining high service quality.

Some of the key metrics and logs that Azure OpenAI that can provide insights are:

1. Error Counts: It provides critical insights into failed requests and incomplete transactions, enabling telecom providers to proactively identify and resolve issues in AI-powered applications.
2. Prompt Input and Completion Text: This captures the input queries provided to AI systems and the corresponding AI-generated outputs. These fields allow telecom providers to analyze customer queries, monitor response quality, and refine AI training datasets to improve relevance and accuracy.
3. Response Latency: It measures the time taken by AI models to generate responses, ensuring that virtual assistants and automated systems deliver quick and efficient replies to customer queries.
4. Token Usage: It tracks the number of input and output tokens processed by the AI model, offering insights into resource consumption and cost efficiency. This data helps telecom providers monitor AI usage patterns, optimize configurations, and scale resources effectively
5. Content Filter Results: In Azure OpenAI, this plays a crucial role in handling sensitive inputs provided by customers, ensuring compliance, safety, and responsible AI usage. This feature identifies and flags potentially inappropriate or harmful queries and responses in real time, enabling telecom providers to address sensitive topics with care and accuracy.

The Azure OpenAI content filtering OOTB dashboard

You can learn more about Elastic's Azure OpenAI integration from these two blogs - Part 1 and Part 2.

4. OpenAI Integration for Generative AI Applications

As AI-powered solutions become integral to modern workflows, OpenAI's sophisticated models, including language models like GPT-4o and GPT-3.5 Turbo, image generation models like DALL·E, and audio processing models like Whisper, drive innovation across applications such as virtual assistants, content creation, and speech-to-text systems. With growing complexity and scale, ensuring these models perform reliably, remain cost-efficient, and adhere to ethical guidelines is paramount. Elastic's OpenAI integration provides a robust solution, offering deep visibility into model behaviour to support seamless and responsible AI deployments.

By tapping into the OpenAI Usage API, Elastic's integration delivers actionable insights through intuitive, pre-configured dashboards, enabling Site Reliability Engineers (SREs) and DevOps teams to monitor performance and optimize resource usage across OpenAI's diverse model portfolio. This unified observability approach empowers organizations to track critical metrics, identify inefficiencies, and maintain high-quality AI-driven experiences. The following key metrics from Elastic's OpenAI integration help organizations achieve effective oversight:

1. Request Latency: Measures the time taken for OpenAI models to process requests, ensuring responsive performance for real-time applications like chatbots or transcription services.
2. Invocation Rates: Tracks the frequency of API calls across models, providing insights into usage patterns and helping identify high-demand workloads.
3. Token Usage: Monitors input and output tokens (e.g., prompt, completion, cached tokens) to optimize costs and fine-tune prompts for efficient resource consumption.
4. Error Counts: Captures failed requests or incomplete transactions, enabling proactive issue resolution to maintain application reliability.
5. Image Generation Metrics: Tracks invocation rates and output dimensions for models like DALL·E, helping assess costs and usage trends in image-based applications.
6. Audio Transcription Metrics: Monitors invocation rates and transcribed seconds for audio models like Whisper, supporting cost optimization in speech-to-text workflows.

To learn more about Elastic's OpenAI integration, read this blog.

Actionable LLM Observability

Elastic's LLM observability integrations empower users to take proactive control of their AI operations through actionable insights and real-time alerts. For instance, by setting a predefined threshold for token count, Elastic can trigger automated alerts when usage exceeds this limit, notifying Site Reliability Engineers (SREs) or DevOps teams via email, Slack, or other preferred channels. This ensures prompt awareness of potential cost overruns or resource-intensive queries, enabling teams to adjust model configurations or scale resources swiftly to maintain operational efficiency.

In the example below, the rule is set to alert the user if token_count crosses a threshold of 500.

The alert is triggered when the token count exceeds the threshold as seen below

Another example is tracking invocation spikes, such as when the number of predictions or API calls surpasses a defined Service Level Objective (SLO). For example, if a Bedrock AI-hosted model experiences a sudden surge in invocations due to increased customer interactions, Elastic can alert teams to investigate potential anomalies or scale infrastructure accordingly. These proactive measures help maintain the reliability and cost-effectiveness of LLM-powered applications.

By providing pre-configured dashboards and customizable alerts, Elastic ensures that organizations can respond to critical events in real time, keeping their AI systems aligned with cost and performance goals as well as standards for content safety and reliability.

Conclusion

LLMs are transforming industries, but their complexity requires effective oversight observability to ensure their reliability and safe use. Elastic's LLM observability integrations provide a comprehensive solution, empowering businesses to monitor performance, manage resources, and address challenges like hallucinations and content safety. As LLMs become increasingly integral to various sectors, robust observability tools like those offered by Elastic ensure that these AI-driven innovations remain dependable, cost-effective, and aligned with ethical and safety standards.

Despite the transformative potential, LLMs introduce complex challenges that necessitate a new level of observability as LLMs are notoriously opaque. Enter LLM observability: a crucial component in the lifecycle management of LLMs. This aspect becomes vital for Service Reliability Engineers (SREs) and other key stakeholders tasked with ensuring seamless, error-free operations, cost control, and minimizing the risks associated with the unpredictable nature of LLM generated responses. SREs need insights into performance metrics, error frequencies, latency issues, the cost implications of running these sophisticated models, and the prompt and response exchange with the model. Traditional monitoring tools fall short in this high-stakes environment; what’s needed is a nuanced approach to address the unique observability demands that LLMs introduce.

Elastic's LLM Observability Capabilities Address These Challenges

With Elastic’s end-to-end LLM observability you can cover a wide range of use cases. To achieve this, you can onboard two types of integrations - API-based logs and metrics and via APM instrumentation. Depending on your use case, you can also choose to use of the LLM integrations.

1. High level overview: via API-based logs and metrics. Monitoring LLM services from providers by ingesting a curated set of service metrics and logs like latency, invocation frequency, tokens, errors, and prompts and responses. Each LLM integration comes with out-of-the-box dashboards.

2. Troubleshooting applications: via APM instrumentation. Fully OTel-native tracing and auto-instrumentation for LLM-based applications through Elastic Distributions of OpenTelemetry (EDOT). Additionally, you can use third party libraries (Langtrace, OpenLit, OpenLLMetry) together with Elastic to extend the coverage to additional LLM-related technologies. 

High level overview: LLM Observability for Leading Providers

Elastic offers tailored API-based integrations for four major LLM hosting providers:

• Azure OpenAI

• OpenAI

• Amazon Bedrock

• Google Vertex AI

These integrations bring a curated set of logs and metrics collection tailored to each provider. What this means for SREs is straightforward access to pre-configured dashboards that highlight the prompts and responses, usage patterns, performance metrics, and cost details across different models and providers.

For instance, SREs keen on identifying which LLM generates the most errors or insights about the models in terms of latency, cost, or usage frequency can leverage these integrations. Imagine having the capability to instantly visualize which LLM is slowing down processes or incurring high costs, thus enabling data-driven decisions to optimize operations.

Troubleshooting applications: Tracing and Auto-Instrumentation of OpenAI, Amazon Bedrock and Google Vertex AI models

Elastic supports OTLP tracing capabilities in EDOT for applications using OpenAI models and models hosted on Amazon Bedrock and Google Vertex AI. In addition, Elastic also supports LLM tracing from third party libraries (Langtrace, OpenLIT, OpenLLMetry). 

Tracing offers a comprehensive map of an application's request flow, pinpointing granular details about each call within the system. For each transaction and span of a request, tracing shows critical information such as specific models utilized, request duration, errors encountered, tokens used per request, and the prompts and responses between the LLM.

Tracing helps SREs troubleshoot performance issues with applications developed in languages like Python, Node.js and Java." If an SRE needs to investigate latency or error issues, LLM tracing provides a zoomed-in view into the request lifecycle and allows for profound insights into whether a delay is application-specific, model-specific or systemic across deployments.

Use Cases: Bringing Elastic's Observability Features to Life

Let’s explore some practical scenarios where Elastic’s observability tools shine:

1. Understanding LLM Performance and Reliability

An SRE team looking to optimize a customer support system powered by Azure OpenAI can utilize Elastic’s Azure OpenAI integration to quickly ascertain which model variants incur higher latency or error rates. This enhances decision-making regarding model deployment or even switching providers based on performance metrics.

Similarly SREs can also use in parallel integrations for Google Vertex AI, Amazon Bedrock, and OpenAI for other applications using models hosted on these providers.

2. Troubleshooting OpenAI-Powered Applications

Consider an enterprise utilizing an OpenAI model for real-time user interactions. Encountering unexplained delays, an SRE can use OpenAI tracing to dissect the transaction pathway, identifying if one specific API call or model invocation is the bottleneck. The SRE can also check the out-of-the-box OpenAI integration dashboard to verify if the latency is only affecting this application or all model invocations across the organization.

An engineer troubleshooting the LLM-based application can also check to see what were the prompt and response exchanges with the LLM during this request so they can rule out possible impact on performance due to the input.

3. Addressing Cost and Usage Concerns

SREs are generally acutely aware of which LLM configurations are less cost-effective than required. Elastic’s integration dashboards, pre-configured to display model usage patterns, help mitigate unnecessary spending effectively. You can find out-of-the box dashboards for Azure OpenAI, OpenAI, Amazon Bedrock, and Google VertexAI models. These dashboards show key cost and usage information such as total invocations and tokens, as well as time series breakdown by model and endpoint. In addition, some integrations show more advanced usage information such as provisioned throughput units (PTU) as well as billing cost.

4. Understanding LLM Compliance 

With the Elastic Amazon Bedrock integration for Guardrails, and Azure OpenAI integration for content filtering, SREs can swiftly address security concerns, like verifying if certain user interactions prompt policy violations. Elastic's observability logs clarify whether guardrails rightly blocked potentially harmful responses, bolstering compliance assurance.

Conclusion

As LLMs continue to revolutionize the capabilities of modern applications, the role of observability becomes increasingly paramount. Elastic’s comprehensive observability framework empowers enterprises to harness the full potential of LLMs while maintaining robust operational insight and control. The integration with prominent LLM hosting providers and advanced tracing for OpenAI, Amazon Bedrock and Google Vertex AI models, equips SREs with the necessary arsenal to navigate the complex landscape of LLM-driven applications, ensuring they remain safe, reliable, efficient, and cost-effective.

In this new era of AI, balancing innovation with observability isn't just beneficial—it's essential. Whether optimizing performance, troubleshooting intricacies, or managing costs and compliance, Elastic stands at the forefront, ensuring your LLM journey is as seamless as it is groundbreaking.