Those traces show how the agent reached its recommendation and what it cost. They do not record who approved it. Below: how to read the traces, scope the three identities a run touches, and append the approval decision to a data stream the pipeline cannot rewrite.

Prerequisites

• Elastic Stack 9.5 or Serverless
• Privileges to manage Kibana advanced settings, needed to install the traces dashboard.

Where Elastic Observability records each part of an AI agent action

Four questions come up in every review of an agentic operations pipeline, and each one is answered by a different record.

| Question | Where the answer lives | Who creates it | | :---- | :---- | :---- | | How did the agent reach its recommendation? | traces-agent_builder.otel-* spans | Agent Builder, automatically | | Which tools did it call, and did they fail? | execute_tool spans in the same data stream | Agent Builder, automatically | | Whose privileges did the run execute with? | Workflow execution record and Elasticsearch security audit logs | Kibana, partly | | What did a human decide, and did the action run? | An index you write to yourself | You |

The first two are new in 9.5 and cost nothing but a toggle. The last one has no automatic source, so it is the one most pipelines are missing.

The scenario: a stale pricing cache in checkout

Three checkout-service workers serve production traffic. One of them, checkout-worker-1, was rolled to version 2026.07.26.1 and now returns HTTP 500 on every quote because its pricing cache stopped refreshing. The other two stay on 2026.07.25.3 and serve normally.

The SRE control plane pattern behind this setup connects telemetry, an Agent Builder agent that reasons over it, and Elastic Workflows that run known actions, with a human approval gate before the first step that changes production.

Telemetry arrives through the documented OpenTelemetry path, so the agent reads standard OTel fields. Elasticsearch 9.5 exposes a native OTLP endpoint, which lets an OTel SDK write directly to the cluster with no collector in between:

from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.resources import Resource

provider = LoggerProvider(
    resource=Resource.create({
        "service.name": "checkout-service",
        "deployment.environment": "production",
    })
)
provider.add_log_record_processor(
    BatchLogRecordProcessor(
        OTLPLogExporter(
            endpoint=f"{ES_URL}/_otlp/v1/logs",
            headers={"Authorization": f"ApiKey {API_KEY}"},
        )
    )
)

The endpoint speaks OTLP over protobuf and rejects application/json with HTTP 406, so send it through an SDK or collector rather than hand-built JSON. That writes 450 records to logs-generic.otel-default: 360 healthy events across the three workers and 90 PricingCacheStaleError events from the broken one. The agent is given none of that context and has to find it by querying.

Reading AI agent observability traces in Elastic

Everything Agent Builder records about a run lives in two data streams, and all of it is queryable with ES|QL.

How to turn on AI agent tracing in GenAI Settings

Open Stack Management, then GenAI Settings, and find the Agent Builder Traces section. Collect conversation traces is on by default in 9.5.

Two details on that panel matter more than the toggle. Traces are written to traces-agent_builder.otel-<space-id>, one data stream per Kibana space, with a companion logs-agent_builder.otel-<space-id> for agent-side events. These are ordinary data streams on the standard OTel index templates, not hidden system indices, so Discover, Lens, and ES|QL query them directly.

The callout states the access model plainly: anyone who can read the index can read every trace in it. Trace access is not scoped per user, so restrict the index pattern through a role before granting access to a space with sensitive conversations.

Reading the LLM trace waterfall for one agent run

A single question through the converse API asks the built-in Elastic AI Agent to investigate logs-generic.otel-default, find which pod and version are returning HTTP 500, and propose one bounded action. It answers correctly, naming checkout-worker-1 on 2026.07.26.1 with a 43% error rate against zero errors on the two workers still on 2026.07.25.3.

Select the trace icon under any agent response to open the waterfall.

One question through the converse API produced 24 spans over 43.7 seconds. The structure is an invoke_agent root, a generate_title side branch, then alternating chat and execute_tool spans as the agent queries, reads the result, and picks the next query.

Three span families carry everything you will aggregate on.

| Span name prefix | What it represents | Key attributes | | :---- | :---- | :---- | | invoke_agent | A conversation round (CHAIN) or an agent execution (AGENT) | elastic.inference.span.kind, gen_ai.agent.id | | chat | One model call | gen_ai.request.model, gen_ai.provider.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens | | execute_tool | One tool invocation | gen_ai.tool.name, gen_ai.tool.call.id, status.code |

The token breakdown for that run:

| Model | Calls | Input tokens | Output tokens | | :---- | ----: | ----: | ----: | | anthropic-claude-4.6-sonnet | 5 | 113,315 | 2,039 | | anthropic-claude-4.5-haiku | 5 | 47,211 | 651 |

Half the model calls went to the smaller model. That is fast model routing, which sends low-effort steps to a cheaper model, and the split is only visible in the trace.

The single agent investigation consumed roughly 160,000 input tokens in total. Each round replays the accumulated context, so cost scales with conversation length rather than with the length of the question.

How do you query agent tool calls with ES|QL?

Tool invocations are the part of agent behavior most worth watching, because that is where the agent touches your data. Every call is one execute_tool span, and the documented query aggregates them directly:

FROM traces-agent_builder.otel-*
| WHERE span.name LIKE "execute_tool *"
| STATS calls = COUNT(*),
        errors = COUNT(*) WHERE status.code == "Error",
        avg_ms = ROUND(AVG(duration) / 1000000.0, 1)
  BY tool = attributes.gen_ai.tool.name
| SORT calls DESC

Across that run the agent leaned on platform.core.execute_esql, with platform.core.generate_esql and load_skill behind it at two calls each. duration is in nanoseconds on the root of the document, which is why the query divides by a million for milliseconds.

Check the companion logs data stream as well. During a different run in the same session, the agent tried to call a tool that was not in its available set, and the attempt was recorded as an exception event in logs-agent_builder.otel-default, correlated to the trace by trace_id:

{
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "event_name": "exception",
  "attributes": {
    "exception.type": "toolNotFoundError",
    "exception.message": "Tool \"platform.streams.query_documents\" called but was not available"
  }
}

A blocked tool attempt is audit-relevant, and it does not appear in the trace waterfall. Querying only the traces data stream will miss it.

For aggregate views there is a managed dashboard, installed per space from the same settings panel.

Over a fifteen-minute window covering these runs, it reported 903,914 input tokens, 11,734 output tokens, and 44 LLM requests, with 35 tool spans at a 100% success rate and 0.42 seconds average duration. The dashboard is managed and read-only, so duplicate it to change a panel, and Elastic can still ship improvements to the original.

What OpenTelemetry LLM traces do not capture by default

By default, a trace records structure and cost, not content. Six toggles under Advanced privacy settings control prompts, responses, tool call details, system prompts, real tool and agent names, and real conversation and workflow IDs, and all six are off.

By default, a trace records structure and cost, not content. Select any span in the waterfall and the detail panel says so directly: "No input/output data available for this span."

Identifiers are hashed rather than dropped. The run with the blocked tool call returned conversation be30fb53-d351-4fa1-b5e1-a569816f85d9 from the API, but its spans carry gen_ai.conversation.id: b1141340d0851a46. That lets you group every span belonging to one conversation and compare conversations against each other, without exposing an identifier that ties back to a user's session.

The consequence is that you cannot join traces to conversations on the conversation ID unless you enable real IDs, and you rarely need to. The converse API hands you the correlation key directly:

{
  "conversation_id": "be30fb53-d351-4fa1-b5e1-a569816f85d9",
  "trace_id": "3f8b9722dbd371ac4b7ad75e4bed13b6",
  "model_usage": {
    "llm_calls": 22,
    "input_tokens": 518251,
    "output_tokens": 6423,
    "model": "anthropic-claude-4.6-sonnet"
  }
}

Store that trace_id in your own decision record, and the join works without weakening the privacy defaults. Enable real IDs only when exact response-to-decision attribution is required, and restrict the trace index in the same change.

Auditing AI agent actions beyond the trace

The traces stop at what the agent did, so the records that show who authorised it have to come from somewhere else.

Whose privileges does an AI agent action run with?

Three identities are involved in an agentic pipeline, and each has its own boundary.

| Identity | Runs with | Determined by | How to scope | | :---- | :---- | :---- | :---- | | Agent Builder tools | The privileges of whoever is chatting | The current user, so two people can get different data from the same question | Roles, as described in Agent Builder permissions | | Workflow steps | One stored API key shared by every elasticsearch.* and kibana.* step | The trigger: manual runs use the person who started them, scheduled runs use whoever last saved the workflow | Workflow authorization | | Trace readers | Index-level access, all or nothing | A role grant on traces-agent_builder.otel-* | A role boundary on the trace index pattern |

One consequence of the stored key belongs in any review. Deactivating a user or changing their role does not refresh it, and the workflow keeps running with the privileges it captured until someone saves it again or toggles Enabled off and back on. Revoking an engineer's access does not, by itself, stop workflows that still run as them.

Scope the investigation role to reads only:

POST /_security/role/agent-builder-observability-investigator
{
  "cluster": ["monitor_inference"],
  "indices": [
    {
      "names": ["logs-*", "metrics-*", "traces-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

Reading the agent's own traces is a separate grant, needing read and view_index_metadata on traces-agent_builder.otel-*. Keep the two roles apart, because the people who investigate incidents and the people who audit the agent are not always the same people.

Recording who approved an agent action in Elasticsearch

Run the workflow, and it stops at the approval gate, where the reviewer sees the agent's structured output rendered into the request rather than a bare confirmation prompt.

The execution record is detailed. It captures resumedAt, resumedBy, the full resumeInput payload, per-step token usage, and a deep link back to itself.

That execution history is an operational view, not an audit store. The underlying .workflows-events data stream is reserved for system operations and rejects user queries outright, so you cannot run ES|QL across a quarter of decisions, and execution history is subject to retention rather than to your compliance policy.

Write the decision to an index you control:

- name: review
  type: waitForInput
  with:
    message: |
      ## Approve the proposed checkout remediation?

      The evidence query matched {{ steps.collect_evidence.output.hits.total.value }} error events in the last hour.

      Agent classification: {{ steps.investigate.output.structured_output.incident_class }}
      Affected pod: {{ steps.investigate.output.structured_output.affected_pod }}
      Proposed action: {{ steps.investigate.output.structured_output.recommended_action }}
    schema:
      type: object
      properties:
        decision:
          type: string
          enum: ["approve", "decline"]
        reason:
          type: string
          enum: ["supported-by-evidence", "insufficient-evidence", "wrong-target", "unsafe-action"]
        notes:
          type: string
      required: ["decision", "reason"]

- name: record_decision
  type: elasticsearch.index
  with:
    index: "agent-action-audit"
    document:
      "@timestamp": "{{ now | date: '%Y-%m-%dT%H:%M:%S.%LZ' }}"
      "event.action": "agent_recommendation_reviewed"
      "incident.id": "{{ consts.incident_id }}"
      "agent.conversation_id": "{{ steps.investigate.output.conversation_id }}"
      "agent.incident_class": "{{ steps.investigate.output.structured_output.incident_class }}"
      "agent.affected_pod": "{{ steps.investigate.output.structured_output.affected_pod }}"
      "agent.recommended_action": "{{ steps.investigate.output.structured_output.recommended_action }}"
      "agent.evidence_count": "{{ steps.collect_evidence.output.hits.total.value }}"
      "review.decision": "{{ steps.review.output.response.decision }}"
      "review.reason": "{{ steps.review.output.response.reason }}"
      "review.notes": "{{ steps.review.output.response.notes }}"
      "review.responded_by": "{{ steps.review.output.respondedBy }}"
      "workflow.execution_id": "{{ execution.id }}"
      "workflow.executed_by": "{{ execution.executedBy }}"
      "workflow.execution_url": "{{ execution.url }}"

Two details in the workflow snippet above differ from the reference page.

The reviewer payload is nested one level deeper. The docs describe steps.<name>.output.<field>, but the running build returns the submitted values under response, alongside a respondedBy field:

{
  "response": { "decision": "approve", "reason": "supported-by-evidence" },
  "respondedBy": "1506416774"
}

execution.executedBy records who started the run, and respondedBy records who approved the action, which in a human-in-the-loop pipeline are usually different people.

The second detail is the timestamp. {{ now }} renders a JavaScript date string like Sun Jul 26 2026 07:37:11 GMT+0000 (Coordinated Universal Time), which Elasticsearch rejects with failed to parse date field, and execution.startedAt has the same problem. The Liquid date filter fixes it.

The workflow editor also flags steps.review.output.* as an invalid variable before the first run, because the reviewer payload shape is only known once someone responds. The warning clears after the step has real output, and the templates resolve correctly at runtime.

Making the audit data stream append-only

An audit trail the agent's own pipeline can rewrite is not an audit trail. Elasticsearch provides two independent controls, and they compose.

First, write to a data stream rather than an index, because data streams accept appends and nothing else:

PUT _index_template/agent-action-audit
{
  "index_patterns": ["agent-action-audit"],
  "data_stream": {},
  "priority": 500,
  "template": {
    "mappings": {
      "properties": {
        "@timestamp":            { "type": "date" },
        "event.action":          { "type": "keyword" },
        "incident.id":           { "type": "keyword" },
        "agent.conversation_id": { "type": "keyword" },
        "agent.evidence_count":  { "type": "long" },
        "agent.recommended_action": { "type": "keyword" },
        "review.decision":       { "type": "keyword" },
        "review.reason":         { "type": "keyword" },
        "review.responded_by":   { "type": "keyword" },
        "workflow.execution_id": { "type": "keyword" }
      }
    }
  }
}

Second, give the writer create_doc and nothing else, so it can add records but cannot reach for the by-query escape hatches:

PUT _security/role/agent-action-audit-writer
{
  "indices": [
    { "names": ["agent-action-audit"], "privileges": ["create_doc", "auto_configure"] }
  ]
}

Tested against the running cluster, that pair behaves the way an audit store should:

| Attempt as the audit writer | Result | | :---- | :---- | | Append a decision record | 201 Created | | Overwrite a record by ID | 400, only op_type: create is allowed in data streams | | _update_by_query to change a decision | 403, action unauthorized | | _delete_by_query to erase history | 403, action unauthorized | | _search to read the trail back | 403, action unauthorized |

The write-only behaviour in the last row is deliberate. The workflow that writes decisions has no reason to read them, so auditors get a separate read role and the writer stays write-only.

The two controls fail differently, which matters. The 400 comes from the data stream itself and applies to everyone, including a superuser. The 403 rows come from the role, and a superuser could still run them, which is why tamper-resistant retention means shipping records off the cluster the agent's operators administer.

For cluster-level activity, enable Elasticsearch and Kibana security audit logging and forward the logs to a monitoring deployment. On 9.5 xpack.security.audit.enabled became a dynamic cluster setting, so Elasticsearch no longer needs a restart to turn it on, though on orchestrated deployments the logs still have to be shipped somewhere readable.

Query the decision trail with ES|QL

Two runs of the workflow, one approved and one declined, produce two rows you can query alongside everything else in Elastic.

FROM agent-action-audit
| KEEP @timestamp, agent.incident_class, agent.affected_pod, agent.recommended_action,
       agent.evidence_count, review.decision, review.reason, review.responded_by,
       workflow.execution_id
| SORT @timestamp DESC

Both runs saw the same 90 error events and proposed restart-checkout-worker on checkout-worker-1. The first review approved it as supported-by-evidence, and the second declined it as wrong-target, on the argument that restarting the pod hides a pricing-feed problem rather than fixing it.

Because both decisions are structured fields, disagreement between reviews is queryable. You can count rejections per incident class and group them by reason: insufficient-evidence sends you back to the investigation path, and unsafe-action sends you to the workflow and its permission boundary.

AI agent observability limits to design around

Four behaviors are worth designing around, and each is cheaper to handle before the workflows are written.

1. Trace access is index-level, not per user. A space with sensitive conversations needs a role boundary on traces-agent_builder.otel-* rather than a UI setting.
2. The managed dashboard is not installed automatically in a new space. Add it to your space provisioning checklist.
3. The workflow execution carries its own APM traceId. It is not the same trace as the Agent Builder spans its ai.agent step produced, so correlate through the conversation ID or the trace_id returned by the agent rather than expecting one trace to span both.
4. The waitForInput output shape differs from the reference page. The submitted values arrive under response, alongside respondedBy, as covered above.

None of these blocks the pattern.

Where to start with AI agent observability

Turn trace collection on, install the dashboard in the space your agents run in, and open the waterfall for one real conversation. It shows the tool sequence, the model split, and the latency distribution that the answer text does not.

Then pick the single incident class where you already trust the runbook, and add one elasticsearch.index step after its approval gate. An append-only decision record costs one workflow step and answers the three questions a review needs: who approved this, on what evidence, and what happened next.

For the details, see Collect Agent Builder traces, the traces overview dashboard, Agent Builder permissions, workflow authorization, and the waitForInput reference.

The first time most teams learn that their OpenAI project is close to a rate limit is when production traffic starts getting throttled with HTTP 429 responses. OpenAI enforces rate limits at the project level, not at the organization level, so a single noisy workload in one project can saturate that project's RPM or TPM ceiling while the rest of the organization still has plenty of room. Without OpenAI rate limit monitoring that compares configured limits against actual consumption, headroom is invisible until it runs out.

OpenAI API monitoring in Elastic: what's new

We're pleased to announce updates to the Elastic OpenAI integration. On top of the existing token usage and audit log coverage, the integration now polls OpenAI's List project rate limits Admin API per project and rolls the results up into both per-project and org-wide views. A new openai.rate_limits dataset feeds two new dashboard panels and a prebuilt threshold alert rule, so teams can see how close each project and model is to being throttled before users experience production impact.

What the integration polls: Usage, Audit Logs and Rate Limits APIs

The Elastic OpenAI integration is built for teams running applications on the OpenAI API platform. The people accountable for it are the developers shipping those services, the platform and SRE teams keeping them running, and the finance and FinOps owners answering "how much is our software consuming, and are we within our capacity envelopes?"

The integration collects from three OpenAI Admin API surfaces:

• Usage API for usage counts across tokens, characters, seconds, sessions, bytes, and images, with project, model, user, and API key attribution where that Usage API surface provides it.
• Audit Logs API for organization audit events such as API key creation, project changes, and user activity.
• Rate Limits API for configured RPM, TPM, and IPM ceilings per project and per model, plus other limit dimensions where available; the new headroom views compare the per-minute request, token, and image limits against actual consumption.

Because everything is pulled from the Admin API at the organization level, platform teams get a unified view across every project, model and API key, alongside the rest of the telemetry they already monitor in Elastic, without touching application code or installing SDKs in every service.

What teams need to monitor when running on the OpenAI API

Four operational needs come up over and over for teams running production workloads on the OpenAI API.

Token usage attribution

A single OpenAI organization usually serves many internal teams and products, each with its own project, its own mix of models (GPT-5.5 Pro for the hardest reasoning tasks, GPT-5.4 for everyday traffic, GPT-5.4 nano for high-volume low-cost requests, and specialized models for images, audio and embeddings) and its own user and API key footprint. When usage patterns shift, the platform team needs to know which project, model and key is driving the change so they can attribute consumption back to the right team and decide which workloads should move to a cheaper model.

Rate limit headroom

OpenAI enforces per-model rate limits on requests per minute (RPM) and tokens per minute (TPM) at the project level, not at the organization level. The first time a team learns they're close to the ceiling is usually when production traffic starts being throttled. Surfacing configured limits alongside actual consumption, per project and per model, lets platform teams see headroom in advance, plan capacity, and request limit increases before users feel the impact.

Audit visibility

Security and compliance teams need to know who created API keys, who changed project settings, who invited or removed users, and when. The integration ingests OpenAI's organization audit log so those events land in the same Elastic deployment as the usage data, ready for correlation, alerting and long-term retention. Audit log ingestion has two prerequisites: audit logging must be enabled in your OpenAI organization, and the Admin API key used by the integration must belong to an Organization Owner, because OpenAI restricts audit-log access to that role. Without both, the openai.audit dataset stays empty.

Granularity for every audience

The same data needs to serve different cadences. SREs want one-minute resolution to catch spikes and trigger throttling alerts. Platform engineers want hourly views for capacity planning. Finance and FinOps owners want daily totals that roll up cleanly for reporting. A single integration that exposes all three granularities removes the need to maintain separate pipelines for each audience.

How does Elastic poll the OpenAI Admin API?

The integration runs on Elastic Agent and uses the CEL input to poll OpenAI's Admin API on a schedule. Authentication uses a single Admin API key, stored as an encrypted Fleet secret and redacted from agent logs. From a single configuration, the integration ingests datasets from three Admin API sources:

Usage API datasets (per project, model, user and API key, with each dataset tracking the unit OpenAI exposes for that workload):

• openai.completions for chat and completion token counts (input, output, cached, audio input/output).
• openai.embeddings for embedding token counts.
• openai.moderations for moderation token counts.
• openai.images for image counts and size dimensions.
• openai.audio_speeches for text-to-speech character counts.
• openai.audio_transcriptions for speech-to-text duration in seconds.
• openai.code_interpreter_sessions for code interpreter session counts.
• openai.vector_stores for vector store byte counts.

Audit Logs API dataset:

• openai.audit for organization audit events such as API key creation, project changes and user activity.

Rate Limits API dataset:

• openai.rate_limits (new) for snapshots of configured rate limits per project and per model, including RPM, TPM, IPM, and other limit fields where OpenAI returns them, paged across all active projects on each poll.

Ingest pipelines handle parsing and field mapping so the data lands queryable, dashboard-ready, and aligned with the rest of Elastic Observability. Because the data is pulled from the Admin API at the organization level, you get this visibility without any application-side instrumentation or SDK changes.

What you need to set up OpenAI monitoring in Elastic

To get started with the Elastic OpenAI integration, you will need:

• An Elastic deployment:
• Elastic Cloud Hosted (ECH) running a recent supported version, or
• Elastic Cloud Serverless, no version requirement, works out of the box.
• An OpenAI organization with Admin API access.
• An Admin API key provisioned by an Organization Owner from the OpenAI platform settings under Settings → Admin keys. Owner-level keys are required if you want the openai.audit dataset to populate.
• Audit logging enabled in your OpenAI organization, if you want audit data.
• Elastic Agent installed on a host with outbound HTTPS access to api.openai.com, or the agentless deployment option.

How to set up the OpenAI integration

1. Generate an Admin API key in the OpenAI platform settings.
2. In Kibana, go to Management → Integrations, search for OpenAI and click Add.
3. Choose your deployment mode: agentless for a zero-install experience, or Elastic Agent on your own host.
4. Tune the defaults if you need to. Each dataset has sensible defaults:

• Usage datasets poll every 5 minutes with 1-minute buckets. Each dataset exposes a finalization_grace setting that controls when a per-minute usage bucket is considered final. The default 0s favors freshness: a bucket is ingested as soon as its minute closes. The observed behavior (which OpenAI does not document, but the integration team measured against the live API) is that bucket counts can keep rising for some time after that point, so per-minute totals at 0s can undercount during heavy bursts. Setting finalization_grace to 15m, the recommended value for accurate counts, holds a bucket back until the grace window has elapsed and brings counts much closer to the Usage API, though a small residual undercount can remain during very high-volume bursts because OpenAI's per-minute counts can be revised upward beyond any fixed grace window. The cost is delaying dashboards and the rate limit headroom alert by the grace period.
• Rate limits polls every 5 minutes. Each poll captures the full set of configured RPM, TPM and IPM limits per project and per model.
• Audit logs polls on a separate cadence and ingests all org-level audit events. Remember the prerequisites: audit logging enabled in your OpenAI organization and an Organization-Owner Admin API key.

1. Open the integration assets in Kibana. Within minutes, usage, audit, and rate-limit data starts flowing, and the prebuilt dashboards and alert rule are ready to use.

For the full configuration reference, see the OpenAI integration documentation.

What do the OpenAI rate limit dashboards show?

The integration ships with a pre-built Kibana dashboard that gives you an immediate, queryable view of your organization's OpenAI API consumption. The overview pulls headline numbers (total tokens, total invocations, top models, top projects, top users and top API keys) into one place for a quick read on the state of your OpenAI usage. The screenshot below shows the OpenAI usage overview dashboard:

From the overview, you can drill into the views that answer the operational needs introduced earlier.

Token usage by model, project and user

The token metrics panels break down token consumption (input, output, cached input, audio input/output) for the token-based datasets (openai.completions, openai.embeddings, openai.moderations) by model and over time. This is the view that tells you where your token budget is actually going, which workloads are getting the most out of prompt caching, and which projects, users or API keys are driving the bulk of your token consumption. Filter by project or model to scope the view to a single team or product. Image, audio and vector-store consumption (measured in images, characters, seconds, sessions and bytes rather than tokens) is reported in dedicated sections of the same dashboard. The token metrics panels look like this:

Rate limit headroom: per project and model (new)

The new rate limit headroom panel joins the configured limits from openai.rate_limits against actual consumption from the usage datasets, per project_id and model. For each row it reports peak one-minute used, the configured limit, and utilization percentage for requests (RPM), tokens (TPM), and images (IPM). Rows are sorted by highest TPM utilization first, with RPM and IPM utilization as tie-breakers, so the highest token-pressure rows appear at the top of the list. Utilization is computed against the peak one-minute bucket in the lookback window, never a 5- or 15-minute sum against a one-minute ceiling, so the panel reflects peak-minute pressure against the one-minute ceiling instead of averaging it away. The per-project rate limit headroom panel is shown below:

Rate limit headroom: org-wide rollup by model (new)

Because OpenAI enforces limits per project, a single per-project view doesn't answer "how much total capacity do we have for gpt-image-2 across the organization?" The new org-wide rollup panel reports the same RPM, TPM and IPM metrics summed across all active projects for each model. Both the limit and the usage figures are indicative upper bounds rather than exact org-wide numbers (the limit is a sum of per-project ceilings; the usage is a sum of each project's peak minute, which may fall in different minutes across projects), but together they give platform teams a single number to plan against when they're sizing a new workload or deciding which project should absorb a new use case. The org-wide rollup panel is shown below:

Behind the scenes, version 2.3.0 also normalizes request and token counts into shared openai.base.usage_tokens and openai.base.usage_images fields across the usage datasets, so the headroom panels render correctly even when only a subset of usage datasets is enabled.

OpenAI audit log activity in Elastic

The audit panels surface organization audit events (API key creations, project changes, user invitations and login activity) so security and compliance teams can review who did what, when, alongside the usage data. The audit dashboard is shown below:

Out-of-the-box alert for rate limit headroom (new)

The integration ships with a pre-built threshold alert rule template, [OpenAI] Rate limit headroom low, ready to install in one click from the integration's Assets tab.

The default behavior is tuned to be useful out of the box:

• Runs every 5 minutes.
• Looks back over the last 15 minutes.
• Fires after 3 consecutive matches.
• Triggers when peak one-minute TPM utilization reaches or exceeds 80% of the configured project/model limit.
• Groups alerts by project_id::model, so an incident in one project on one model doesn't get lost in an org-wide aggregate.

The 80% threshold and other parameters are editable in Kibana after you install the rule, so each team can tune the alert to its own risk tolerance.

Custom OpenAI alerts and SLOs in Elastic Observability

As with every other Elastic integration, all the OpenAI metrics and audit data is fully available to leverage in every capability in Elastic Observability, including SLOs, alerting, custom dashboards and in-depth logs exploration.

For example, to keep token consumption under control across a single project, create a custom threshold rule that sums tokens from the relevant usage dataset and fires when the daily or hourly total crosses your budget. To track model mix, define an SLO in Elastic Observability that treats OpenAI requests on your approved lower-cost model families as the "good events" (the ones that count as meeting the target) and all OpenAI requests as the "total events", grouped by openai.base.project_id and openai.base.user_id. The ratio becomes your SLI; a 7-day rolling 80% target quickly surfaces projects and users overusing more expensive models.

Choosing OpenAI usage data granularity

OpenAI usage data collected by the integration powers different cadences, with a fidelity-versus-freshness tradeoff to be aware of. One-minute usage buckets feed the rate limit headroom alert and near-real-time throttling notifications when a project approaches its ceiling: with finalization_grace set to 0s (the default), per-minute counts arrive within minutes but can undercount during heavy bursts; raising finalization_grace to 15m brings counts much closer to reconciled at the cost of delaying the dashboards and alert by the grace period; a small residual undercount can still remain for the busiest buckets. Hourly views support operational monitoring and capacity planning across projects and models. Daily aggregates roll up cleanly for FinOps reporting and reconciliation. An out-of-the-box alert ships for rate limit headroom ([OpenAI] Rate limit headroom low), and the same data can be reused for custom usage and budget thresholds without building anything from scratch.

Get started with OpenAI monitoring in Elastic

The Elastic OpenAI integration is available today in Elastic Cloud, including Elastic Cloud Hosted and Elastic Cloud Serverless. To get started, sign up for a free Elastic Cloud trial, provision an Admin API key in the OpenAI platform settings, and add the OpenAI integration from Kibana under Management → Integrations.

Within minutes you'll have token usage, audit activity, and rate limit headroom data flowing into Elasticsearch, with the prebuilt dashboards and the new throttling alert ready to use.

Elastic Ramen (Root-cause Analysis & Monitoring Engine) bridges that gap. It is a local CLI agent that connects directly to Elastic Agent Builder, carrying the same conversation, skills, and Elastic context into the terminal. Ramen operates directly in the environment where fixes actually happen. No handoff. No re-auth. No translation layer. Ramen is open source and available at elastic/elastic-ramen.

Why the terminal matters

Agent Builder gives engineers a strong environment for querying observability data. Ramen takes that same capability to the two workflows that need it most.

Onboarding. Configuring collectors, managing credentials, and validating data flow all happen in the shell. A local agent can guide that work right where the credentials and tools already live.

Mitigation. The actual fix, whether restarting pods, scaling deployments, or rolling back releases, requires kubectl, gcloud, git, or internal scripts. A CLI agent runs on hardware the team already trusts, using the credentials already present on the engineer's machine.

How Ramen works

Ramen is a CLI client for Agent Builder. It is not a separate assistant with its own memory. It connects your local environment to the same conversations, skills, and tools you already use in Kibana through a simple authentication flow.

On first launch, Ramen connects to your Elastic deployment and gives you everything out of the box:

• LLM inference through the Kibana gateway, using your existing AI connector
• Native Kibana tools for managing workflows and agents
• The Agent Builder MCP server for ES|QL queries and documentation search
• An embedded elastic CLI for cluster health, data streams, and SLOs
• Built-in skills for root cause analysis and SLO management

The agent carries your investigation history across surfaces, so you never re-explain the incident when moving from the UI to the CLI. Terminal interactions sync back to Elastic automatically, building a searchable record of operational knowledge for the team.

Get started

You need an Elastic Observability Serverless project. In Kibana, open Stack Management, then Advanced Settings, or go directly to https://<your-kibana-url>/app/management/kibana/settings?query=ramen. Enable elasticRamen:enabled, then install the CLI:

npm i -g @elastic/ramen
bun add -g @elastic/ramen

You can also use the install script or download a pre-built binary from GitHub Releases:

curl -fsSL https://raw.githubusercontent.com/elastic/elastic-ramen/dev/install | bash

Once installed, connect to your deployment:

elastic-ramen --kibana-base=https://<your-kibana-url>

Ramen opens a browser auth flow, generates credentials, and stores them locally. After that, it reconnects automatically. Start a conversation in Agent Builder and resume it in the terminal with /kibana-conversations.

What is next

Ramen is the first surface of a multi-surface agent system. The same architecture extends to every surface engineers already use:

• Space-scoped collaboration for shared agent context during outages
• Slack, Teams, Jira, PagerDuty integration: start from an alert, collaborate in chat, mitigate in the terminal, one thread
• Shared memory: progressively distill conversations into durable operational context that improves future investigations

Beyond incident response, the same model applies to deployment risk analysis, production debugging, CI/CD policy checks, and cost anomaly investigation.

Summary

Ramen connects signal to action: Elastic data and Agent Builder context, plus the ability to act with local tools, in one continuous thread. Elastic as the persistent context layer, every surface you use as the interface.

Try it out on GitHub and let us know what you think.

However, the landscape of log management is shifting. By combining the industry-standard ingestion of OpenTelemetry (OTel) with the AI-driven capabilities of Elastic Streams, we can change how we monitor Windows infrastructure. This approach isn't just moving data. We are also using Large Language Models (LLMs) to understand it.

The Challenge with Traditional Windows Logging

Windows generates a massive variety of logs: System, Security, Application, Setup, and Forwarded Events. Within those categories, you have thousands of Event IDs. Historically, getting this data into an observability platform involved installing proprietary agents and configuring complex pipelines to strip out the XML headers and format the messages.

Once the data was ingested, we can try to figure out what "bad" looked like. You had to know in advance that Event ID 7031 indicated a service crash, and then write a specific alert for it. If you missed a specific Event ID or if the format changed, your monitoring went dark.

Step 1: Ingestion via OpenTelemetry

The first step in modernizing this workflow is adopting OpenTelemetry. The OTel collector has matured significantly and now offers robust support for Windows environments. By installing the collector directly on Windows servers, you can configure receivers to tap into the event log subsystems.

The beauty of this approach is standardization. You aren't locked into a vendor-specific shipping agent. The OTel collector acts as a universal router, grabbing the logs and sending them to your observability backend in this case, the Elastic logs index designed to handle high-throughput streams.

The key thing to pay attention to in this configuration is how we add this transform statement:

transform/logs-streams:
  log_statements:
    - context: resource
      statements:
        - set(attributes["elasticsearch.index"], "logs")

This works with the vanilla opentelemetry collector and when the data arrives in Elastic, it tells Elastic to use the new wired streams feature which enables all the downstream AI features we discuss in later steps.

Checkout my example configuration here

Step 2: AI-Driven Partitioning

Once the data arrives, the next challenge is organization. Dumping all Windows logs into a single logs-* index is a recipe for slow queries and confusion. In the past, we split indices based on hardcoded fields. Now, we can use AI to "fingerprint" the data.

This process involves analyzing the incoming stream to identify patterns. The system looks at the structure and content of the logs to determine their origin. For example, it can distinguish between a Windows Security Audit log and a Service Control Manager log purely based on the data shape.

The result is automatic partitioning. The system creates separate, optimized "buckets" or streams for each data type. You get a clean separation of concerns, Security logs go to one stream, File Manager logs to another, without having to write a single conditional routing rule. This partitioning is crucial for performance and for the next phase of the process: analysis.

Step 3: Significant Events and LLM Analysis

Once your data is partitioned (e.g., into a dedicated Service Control Manager stream), you can apply GenAI models to analyze the semantic meaning of that stream.

In a traditional setup, the system sees text strings. In an AI-driven setup, the system understands context. When an LLM analyzes the Service Control Manager stream, it identifies what that system is responsible for. It knows that this specific component manages the starting and stopping of system services.

Because the model understands the purpose of the log stream, it can generate suggestions for what constitutes a "Significant Event." It doesn't need you to tell it to look for crashes; it knows that for a Service Manager, a crash is a critical failure.

From Passive Storage to Proactive Suggestions

The workflow effectively automates the creation of detection rules. The LLM scans the logs and generates a list of potential problems relevant to that specific dataset, such as:

• Service Crashes: High severity anomalies where background processes terminate unexpectedly.
• Startup/Boot Failures: Critical errors preventing the OS from reaching a stable state.
• Permission Denials: Security-relevant events regarding service interactions.

It bubbles these up as suggested observations. You can review a list of potential issues, see the severity the AI has assigned to them (e.g., Critical, Warning), and with a single click, generate the query required to find those logs.

Conclusion

The combination of OpenTelemetry for standardized ingestion and AI-driven Streams for analysis turns the chaotic flood of Windows logs into a structured, actionable intelligence source. We are moving away from the era of "log everything, look at nothing" to an era where our tools understand our infrastructure as well as we do.

The barrier to effective monitoring is no longer technical complexity. Whether you are tracking security audits or debugging boot loops, leveraging LLMs to partition and analyze your streams is the new standard for observability.

Try Streams today

What's in data quality

At-a-glance summary

The summary card shows:

• Degraded documents - Documents that contain the _ignored field - see this for more info.
• Failed documents - Documents that were rejected at ingestion due to mapping conflicts or pipeline failures.

The overall quality score (Good, Degraded, Poor) is automatically calculated based on the percentage of degraded and failed documents.

Trends over time

The tab includes a time-series chart so you can track how degraded and failed documents are accumulating over time. Use the date picker to zoom into a specific range and understand when problems are spiking.

Quality issues table

A detailed table lists the types of issues affecting your stream. For each issue, you can:

• See which fields are causing problems.
• Review counts of affected documents.
• Filter by issues that have not been solved yet (Current issues only).
• Open a flyout to dive deeper into the cause of the issue and learn how to fix it.

Monitoring degraded documents

A degraded document is one that contains the _ignored field, which means one or more of its fields were ignored during indexing. One of the reasons could be that their values didn’t match the expected mappings. While the rest of the document is still indexed, a high number of degraded documents can affect query results, dashboards, and overall observability accuracy.

To help keep these issues under control, the Data quality tab provides visibility into the percentage of degraded documents in your stream.

Set up a rule to stay ahead of issues

You can use the Create rule button above the Degraded docs chart to define an alert that notifies you when the percentage of degraded documents crosses a certain threshold. This makes it easy to proactively monitor for mapping mismatches and ensure your data continues to meet quality expectations.

For more information on how to configure this rule, see Degraded docs rule conditions.

Handling failed documents with the failure store

Failure store is a special index that captures documents rejected during ingestion. Instead of losing this data, the failure store retains it in a dedicated ::failures index, allowing you to inspect the problematic documents, understand what went wrong, and fix the underlying issues.

In Data Quality tab, the failed documents are only visible if your stream has a failure store enabled, for checking failure store documents you are required to have at least read_failure_store privileges. If the failure store is not enabled, you’ll see an “Enable failure store” link that opens a modal to configure it and set the retention period. For enabling failure store you are required to have manage_failure_store privileges over the specific data stream. For further information about failure store security you can refer to Searching failures.

Once enabled, you can edit the failure store configuration or disable it at any time using the Edit button above the failed docs chart.

The failure store can also be configured in the Streams Retention tab - see this article for more information.

Technical implementation

Under the hood, the Data quality tab builds on the existing Dataset quality plugin - the same one that powers the Dataset quality page in Stack Management. However, instead of working in the context of datasets following the Data stream naming scheme, it’s now tailored specifically for streams.

To determine the quality of a stream, the UI sends three ES|QL query server requests:

1. All documents (including failures):

 FROM myStream, myStream::failures | STATS doc_count = COUNT(*)

1. Failed documents only:

 FROM myStream::failures | STATS failed_doc_count = COUNT(*)

1. Degraded documents:

FROM myStream METADATA _ignored | WHERE _ignored IS NOT NULL | STATS degraded_doc_count = COUNT(*)

The results of these queries are then used to calculate the percentages of failed and degraded documents. The overall data quality is determined using simple thresholds:

• Good: Both percentages are 0%
• Degraded: Any percentage is greater than 0% but less than 3%
• Poor: Any percentage is above 3%

For managing the failure store, Streams uses the Update data stream options API with the failure_store parameter to configure and update the failure store settings, including enabling the store and setting the retention period.

Why you’ll love this

The new Data quality tab gives you:

• Visibility into ingestion problems without digging into logs
• A clear breakdown of degraded vs. failed documents
• Insights into which fields are ignored and why
• Tools to capture and troubleshoot failed docs with the failure store

By surfacing data quality issues directly in the Streams UI, we’re making it easier to keep your data flowing reliably and to ensure your analytics are built on a strong foundation.

Try it out today

The data quality feature is available in Elastic Observability on Serverless, and coming soon for self-managed and Elastic Cloud users.

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.

For more information on Streams:

Read about Reimagining streams

Look at the Streams website

Read the Streams documentation

Streams changes that. It introduces a clear, unified way to manage how long your data lives, whether you’re using DSL or ILM. You can visualize ingestion, understand where data sits across tiers, and adjust retention with confidence, applying updates to a single stream without worrying about unintended changes elsewhere, all from a single view.

Walkthrough: Exploring the Retention Tab

Retention management lives in the Retention tab of each stream. This is your control panel for understanding how much data you’re storing, how quickly it’s growing, and how your lifecycle policies are applied. It’s also where you can monitor and configure the Failure store, which tracks and retains documents that failed to be ingested.

Metrics at a glance

At the top of the view, you’ll find an overview of key metrics:

• Storage size: the total data volume currently held by the stream.
• Ingestion averages: calculated from the selected time range, Streams extrapolates both daily and monthly averages to give you a sense of long-term trends.

This combination of near-real-time and projected values helps you quickly spot when ingestion is ramping up and whether your retention policy aligns with it.

Ingestion over time

Below the metrics, a graph shows ingestion volume over time. This information is approximated based on the number of documents over time, multiplied by the average document size in the backing index. 

Visualizing lifecycle phases

When an ILM policy is effective, the retention view becomes more visual. Streams displays a phase breakdown (hot, warm, cold, frozen) showing the data volume stored in each phase. This gives you a clear sense of how your data is distributed across the storage tiers and whether your lifecycle is doing what you expect.

Failure store

A failure store is a secondary set of indices inside a data stream, dedicated to storing documents that failed to be ingested. Within the Retention tab, you can toggle the Failure store on or off, and configure its own retention period. We’ll cover Failure store and Data quality in more detail in this article.

Updating Retention

Beyond visualizing your retention, Streams makes it easy to change how it’s managed.

Switching between DSL and ILM

You can freely switch a stream between DSL and ILM management, or update a DSL retention with just a few clicks. Streams takes care of updating the lifecycle settings at the data stream level, ensuring consistent retention across all existing backing indices, not just new ones.

Whether you prefer the simplicity of DSL or the fine-grained tiering of ILM, you can move between the two seamlessly.

Clicking “Edit data retention” opens a modal that allows you to update the stream’s configuration. From there you can update the ILM policy or set a custom retention period via DSL.

You can set a custom period, or pick an Indefinite retention for your data.

You can also update streams’ lifecycle via the Upsert stream or the Update ingest stream settings Kibana APIs.

Inherit or defer: different strategies for different stream types

Classic streams

For classic streams, you can default to the existing index template’s retention. Retention isn’t managed by Streams in this case, it follows the lifecycle configuration defined in the template just as it normally would.

This option is useful if you’re onboarding existing data streams and want to keep their lifecycle behavior intact while still benefiting from Streams’ visibility and monitoring features.

Wired streams

Wired streams live in a tree structure, and that hierarchy allows an inheritance model.

A child stream can inherit the lifecycle of its nearest ancestor that has a concrete policy (ILM or DSL). This keeps your configuration lean and consistent since you can set a single lifecycle at a higher level in the tree and let Streams automatically apply it to all relevant descendants.

If that ancestor’s lifecycle is later updated, Streams cascades the change down to all children that inherit it, so everything stays in sync.

In the figure below, we set a different retention for logs.prod and logs.staging environments. The child partitions of these environments automatically inherit the configuration.

How it works under the hood

When you apply or update a lifecycle, Streams calls Elasticsearch’s /_data_stream/_settings. This is a new API we’ve added in 8.19 / 9.1 for this purpose. 

This API is key to keeping retention consistent:

1. It applies the lifecycle directly at the data stream level, overriding any configuration from cluster settings or index templates.
2. It propagates the retention update to all existing backing indices, not just new ones, so retention remains uniform across your historical and future data.

By centralizing lifecycle management at the data stream level and applying a consistent configuration across the backing indices, we remove the ambiguity that used to exist between template-level and index-level configurations. You always know which retention policy is actually in effect, and you can see it directly in the UI.

Wrapping Up

With Streams, retention management becomes clear and consistent. You can visualize ingestion, switch between DSL and ILM, or inherit policies across streams, all without diving into templates or manual index settings.

By unifying retention into a single view, Streams turns lifecycle management into something simple, predictable, and transparent.

Sign up for an Elastic trial at cloud.elastic.co, and trial Elastic's Serverless offering which will allow you to play with all of the Streams functionality.

Additionally, check out:

Read about Reimagining streams

Look at the Streams website

Read the Streams documentation

Observability, the broken promise

To see why the current model fails, let’s look at the all-too-familiar challenge every SRE dreads: knowing a problem exists but needing to spend valuable time just trying to find where to even start the investigation.

Imagine you get a Slack message from the support team: "a few high-value customers are reporting their payments are failing." You have no shortage of alerts, but most are just flagging symptoms. You don’t know where to start. You decide to check the logs to see if there is anything obvious, starting with the systems that have the high CPU alert.

You spend a few minutes searching and grep-ing through terabytes of logs for affected customer IDs, trying to piece together the problem. Nothing. You worry that you aren’t getting all the logs to reveal the problem, so you turn on more logging in the application. Now you’re knee-deep in data, desperately trying to find patterns, errors, or other "hints" that will give you a clue as to the why.

Finally, one of the broader log queries hits on an error code associated with an impacted customer ID. This is the first real clue. You pivot your search to this new error code and after an hour of digging, you finally uncover the error message. You've finally found the why, but it was a stressful, manual hunt that took far too much time and impacted dozens more customers.

This incident perfectly illustrates the broken promise of modern observability: The complete failure of the investigation process. Investigations are a manual, reactive process that SREs are forced into every day. At Elastic, we believe metrics, traces, and logs are all essential, but their roles, and the workflow between them, must be fundamentally re-imagined for effective investigations.

Observability is about having the clearest understanding possible of the what, where, and why. Metrics are essential for understanding the what. They are the heartbeat of your system, powering the dashboards and alerts that tell you when a threshold has been breached, like high CPU utilization or error rates. But they are aggregates; they show the symptom, rarely the root cause. Traces are good at identifying the where. They map the journey of a request through a distributed system, pinpointing the specific microservice or function where latency spikes or an error originates. Yet, their effectiveness hinges on complete and consistent code instrumentation, a constant dependency on development teams that can leave you with critical visibility gaps. Logs tell you the why. They contain all the rich, contextual, and unfiltered truth of an event. If we can more proactively and efficiently extract information from logs, we can greatly improve our overall understanding of our environments.

Challenges of logs in modern environments

While logs are in the standard toolbox, they have been neglected. SREs using today’s solutions deal with several major problems:

• First, due to their unstructured nature, it’s very difficult to parse and manage logs so that they’re useful. As a result, many SRE teams spend a lot of time building and maintaining complex pipelines to help manage this process. 

• Second, logs can get expensive at high volume, which leads teams to drop them on the floor to control costs, throwing away valuable information in the process. Consequently, when an incident occurs, you waste precious time hunting for the right logs, and manually correlating across services.

• Finally, nobody has built a log solution that proactively works to find the important signals in logs and to surface those critical whys to you when you need them. As a result, log-based investigations are too painful and slow.

Why are we here? As applications became more complex, log volume became unmanageable. Instead of solving this with automation, the industry took a shortcut: it gave up on getting the most out of logs and prioritized more manageable but less informative signals.

This decision is the origin of the broken, reactive model. It forced observability into a manual loop of 'observing' alerts, rather than building automation that could help us truly understand our systems to improve how we root cause and resolve issues. This has transformed SREs from investigators into full-time data wranglers, wrestling with Grok patterns and fragile ETL scripts instead of solving outages. 

Introducing Streams to rethink how you use logs for investigations

Streams is an agentic AI solution that simplifies working with logs to help SRE teams rapidly understand the why behind an issue for faster resolution. The combination of Elasticsearch and AI is turning manual management of noisy logs into automated workflows that identify patterns, context, and meaning, marking a fundamental shift in observability.

Log everything in any format

By applying the Elasticsearch platform for context engineering to bring together retrieval and AI-driven parsing to keep up with schema changes, we are reimagining the entire log pipeline.  

Streams ingests raw logs from all your sources to a single destination. It then uses AI to partition incoming logs into their logical components and parses them to extract relevant fields for an SRE to validate, approve, or modify. Imagine a world where you simply point your logs to a single endpoint, and everything just works. Less wrestling with Grok patterns, configuring processors, and hunting for the right plugin. All of which significantly reduces the complexity. Streams is a big step towards realizing that vision.

As a result, SREs are freed from managing complex ingestion pipelines, allowing them to spend less time on data wrangling and more time preventing service disruptions.

Solve incidents faster with Significant Events 

Significant Events, a capability within Streams, uses AI to automatically surface major errors and anomalies, enabling you to be proactive in your investigations. So, instead of just combing through endless noise, you can focus on the events that truly matter, such as startup and shutdown messages, out-of-memory errors, internal server failures, and other significant signals of change. These events act as actionable markers, giving SREs early warning and clear focus to begin an investigation before service impact.

With this new foundation, logs will become your primary signal for investigation. The panicked, manual search for a needle in a digital haystack is about to be over. Significant Events acts like a smart metal detector that sifts through the chaos and only beeps when it finds issues, helping you to easily ignore all that hay and find the "needle" faster. 

Now imagine the same scenario we started with. Instead of starting a frantic, time-consuming grep through terabytes of logs. Streams has already done the heavy lifting. Its AI-driven analysis has detected a new, anomalous pattern that began before your support team even knew about it and automatically surfaced it as a significant event. Rather than you hunting for a clue, the clue finds you. 

With a single click, you have the why: a Java out-of-memory error in a specific service component. This is your starting point. You find the root cause in under two minutes and begin remediation. The customer impact is stopped, the dev team gets the specific error, and the problem is contained before it can escalate. In this case, metrics and traces were unhelpful in finding the why. The answer was waiting in the logs all along.

This ideal outcome is possible because you can both afford to keep every log and instantly find the signal within them. Elastic's cost-efficient architecture with powerful compression, searchable snapshots, and data tiering makes full retention a reality. From there, Streams automatically surfaces the significant event, ensuring that the answer is never lost in the noise.

Elastic is the only company that provides an AI-driven log-first approach to elevate your observability signals and make it dramatically faster and easier to get to why. This is built on our decades of leadership in search, relevance, and powerful analytics that provides the foundation for understanding logs at a deep, semantic level.

The vision for Streams 

The partitioning, parsing, and Significant Events you see today is just the starting point. The next step in our vision is to use the Significant Events to automatically generate critical SRE artifacts. Imagine Streams creating intelligent alerts, on-the-fly investigation dashboards, and even data-driven SLOs based only on the events that actually impact service health. From there, the goal is to use AI to drive automated Root Cause Analysis (RCA) directly from log patterns and generate remediation runbooks, turning a multi-hour hunt into an instant resolution recommendation.

Once this AI-drive log foundation is in place, our vision for Streams expands to become a unified intelligence layer that operates across all your telemetry data. It’s not just about making each signal better in isolation, but about understanding the context and relationships between them to solve complex problems. 

For metrics, Streams won’t just alert you to a single metric spike but detect a correlated anomaly across multiple, seemingly unrelated metrics e.g. p99 latency for a specific service, rise in garbage collection time, transaction success rate.

Similarly, for traces it identifies a new, unexpected service call (e.g., a new database or an external API) appears in a critical transaction path after a deployment or identifies specific span is suddenly responsible for a majority of errors across all traces, even if the overall error rate hasn't breached a threshold.

The goal is not to have separate streams for logs, metrics, and traces, but to weave them into a single narrative that automatically correlates all three signals. Ultimately, Streams is about fundamentally changing the goal from human led data gathering exercise to proactive, AI-driven resolution.

For more on Streams:

Read the Streams launch blog

Look at the Streams website

One of the main components of Elastic's Synthetic Monitoring is the ability to create user journeys, which can be done with or without code. There is a Synthetics agent,, a CLI tool that guides you through the process of creating both heartbeat monitors and user journeys and deploying your code to Elastic Observability. If you are using code to create user journeys, you are using Playwright under the hood with some additional configuration to make it easier to work with Elastic Observability.

To automatically create user journeys using TypeScript, you can create Playwright tests based on a prompt using Warp, an AI-assisted terminal, Gemini 2.5 Pro, and MCP. This application was built using Python and FastMCP, which wraps the synthetic agent to deploy browser tests to Elastic automatically. This blog post will guide you through how the application works, how to use it, and its development process. You can find the complete code on GitHub.

Solution overview

Currently, this solution is set up to run inside Warp as an MCP server; however, you can also use another client, such as Claude Desktop or Cursor. From there, you create a Python script using FastMCP, which allows you to create functions that are callable by an LLM. Within Warp, you can make a configuration file in JSON that enables you to point to your Python script and pass in all the environment variables you are working with. From there, you'll want to toggle agent mode and ask a question about creating synthetic testing or call the MCP function directly. There are many options for which LLM you can select, be sure to check out Warp's documentation to learn more about the options available.

After that, you should ask a question about creating synthetic testing or call the MCP function you are looking for. The following three functions can be used:

• diagnose_warp_mcp_config Used for debugging environment variable issues that may arise. This function likely won't be needed unless there is an issue with your configuration.

• create_and_deploy_browser_test Will automatically create Playwright tests if given the test name, the URL you want to test, and a schedule. This approach uses a template-based method, rather than a machine learning-based method, and all the tests it outputs will appear similar.

• llm_create_and_deploy_test_from_prompt Similar to create_and_deploy_browser_test, but the main difference is that it uses an LLM to create tests based on a prompt you give it. The tests should reflect the prompt you provided. To run this function you'll provide a test name, URL, prompt, and schedule.

Why create this solution as an MCP server?

The reason this was developed as an MCP server, as opposed to just a standalone script or a standard CLI, is that it can be structured and interacted with in a more conversational manner. It enables an LLM to generate dynamic Playwright testing while maintaining consistent arguments, environment variables, and responses to ensure accuracy and reliability. Thus, it becomes a reliable workflow that other agents or developers can compose with additional tools. In other words, the MCP layer turns your LLM-based test authoring into a standardized, reusable capability instead of a one-off script. To learn more about the direction of MCP, be sure to check out our article on the topic.

Implementation considerations

When creating a solution like this one, one thing to be mindful of is your use of tokens. An early version of this solution took approximately twenty minutes to create synthetic tests and ultimately led to severe rate-limiting.

Another issue faced during the building process was striking a balance between creating a template that facilitates the creation of a Playwright script and having an LLM create Playwright scripts based on prompts that didn't feel cookie-cutter. While using a more LLM approach an issue faced was that the scripts often didn't work or were based on parameters that didn't exist and a more templated approach was more reliable but felt repetitive. The final version of this solution attempted to balance this by using elements of the template while adjusting the LLM parameter of temperature, which controls the randomness or creativity of a large language model's output.

While testing this solution, a failing test also emerged that required navigating past a pop-up. In more complex cases, this may serve as a building block that requires additional domain knowledge to create a complete passing Playwright test.

How to get started

Prerequisites

• The version of Python that is used is Python 3.12.1 but you can use any version of Python higher than 3.10.
• This application uses Elastic Observability version 9.1.2, but you can use any version of Elastics Observability that is higher than 8.10. You can also use Elastic Cloud Serverless as well.
• You will also need an OpenAI API key to use the LLM capabilities of this application. You will want to configure an environment variable for your OpenAI API Key, which you can find on the API keys page in OpenAI's developer portal.

Step 1: Install the packages and clone the repository

In order for this MCP server to run locally you will need to install the the following packages:

pip install fastmcp openai
npm install -g playwright @elastic/synthetics

You will use FastMCP 2.0 to create the MCP server, and OpenAI to generate tests based on prompts that you provide. Additionally, you will want to clone the repository to obtain a local copy of the server.

Step 2: Set up a configuration file in Warp

Inside of Warp, you will want to go to the side panel, where it says MCP servers and where it says “add”.

After that, you will be prompted to add a JSON configuration file that should resemble the following. Be sure to add your own Kibana URL, update the correct path, and include your own keys and tokens.

{
 "elastic-synthetics": {
   "command": "python",
   "args": ["elastic_synthetics_server.py"],
   "env": {
     "PYTHONPATH": ".",
     "ELASTIC_KIBANA_URL": "https://your-kibana-url.elastic-cloud.com",
     "ELASTIC_API_KEY": "your-api-key-here",
     "ELASTIC_PROJECT_ID": "mcp-synthetics-demo",
     "ELASTIC_SPACE": "default",
     "ELASTIC_AUTO_PUSH": "true",
     "ELASTIC_USE_JAVASCRIPT": "false",
     "ELASTIC_INSTALL_DEPENDENCIES": "true",
     "OPENAI_API_KEY": "sk-your-openai-key",
     "LLM_MODEL": "gpt-4o"
   },
   "working_directory": "/path/to/your/file",
   "start_on_launch": true 
   }
}

Step 3: Ask a question or call the tools directly

Now that you've set up locally, you will want to toggle agent mode and select the LLM you wish to use. The reason why Gemini-Pro-2.5 was chosen for this blog post is that it provides a straightforward answer, while other LLMs selected returned a very lengthy response.

To start using the MCP tools, from your MCP server, you can ask a question that contains the test name, URL, prompt, and schedule.

You can also call the directly by typing llm_create_and_deploy_test_from_prompt() and the program will prompt you for the relevant details:

Inside Kibana, you should see your monitor listed if you click under Applications and select Monitors listed under Synthetics. You can also find a link to your monitor in the response of your MCP tool.

What's Going On Inside

This code sample consists of three primary functions, which are MCP tools that you can call from your MCP client, including diagnose_warp_mcp_config, create_and_deploy_browser_test and llm_create_and_deploy_test_from_prompt.

Debugging environment issues

There were various issues that came up while creating this application around environment variable loading, so there was a need to create an MCP that could be called depending on errors that may be present.

The tool diagnose_warp_mcp_config kicks off with a decorator @mcp.tool() which allows it to be called and listed in the list of available tools. This tool is designed to help debug issues with Elastic-specific environment variables for troubleshooting purposes. First, it loads in the environment variables and looks for the Elastic specific variables, after it does some security masking so it doesn't show any variables and hides sensitive information like API keys in the output, showing only the first eight characters followed by "…". This tool determines if the minimum required credentials (Kibana URL and API Key) are present to proceed with deployment and provides a report letting you know to address any issues that may exist.

@mcp.tool()
def diagnose_warp_mcp_config() -> Dict[str, Any]:
   """Diagnose Warp MCP environment configuration for Elastic Synthetics"""
   try:
       env_vars = load_env_from_warp_mcp()

       # Check for required variables
       kibana_url = env_vars.get('ELASTIC_KIBANA_URL') or env_vars.get('KIBANA_URL')
       api_key = env_vars.get('ELASTIC_API_KEY') or env_vars.get('API_KEY')
       project_id = env_vars.get('ELASTIC_PROJECT_ID') or env_vars.get('PROJECT_ID')
       space = env_vars.get('ELASTIC_SPACE') or env_vars.get('SPACE', 'default')

       # Mask sensitive values for display
       masked_vars = {}
       for key, value in env_vars.items():
           if 'API_KEY' in key or 'TOKEN' in key:
               masked_vars[key] = f"{value[:8]}..." if value and len(value) > 8 else "***"
           else:
               masked_vars[key] = value

       deployment_ready = bool(kibana_url and api_key)

       return safe_json_response({
           "status": "success",
           "environment_variables": masked_vars,
           "required_check": {
               "kibana_url": bool(kibana_url),
               "api_key": bool(api_key),
               "project_id": bool(project_id),
               "space": bool(space)
           },
           "deployment_ready": deployment_ready,
           "recommendations": [
               "Environment variables detected" if env_vars else "No environment variables found",
               "Kibana URL configured" if kibana_url else "Missing ELASTIC_KIBANA_URL or KIBANA_URL",
               "API Key configured" if api_key else "Missing ELASTIC_API_KEY or API_KEY",
               "Ready for deployment" if deployment_ready else "Missing required credentials"
           ]
       })

   except Exception as e:
       return safe_json_response({
           "status": "error",
           "error": str(e),
           "error_type": type(e).__name__
       })

Creating synthetic tests based on a template

While developing this solution to generate tests based on a prompt, the process wasn't always smooth. Early versions encountered issues with accuracy, hallucinations, and the creation of loops. To make progress, a version that relied on creating a test template to verify the mechanics of the solution, such as whether the test could pass and be deployed to Elastic correctly, was a logical next step.

This solution automates the entire process of creating a synthetic browser test that will regularly check if a website is working correctly, then deploys it to Elastic Observability Synthetics. Similar to diagnose_warp_mcp_config, the MCP tool create_and_deploy_browser_test starts with the decorator @mcp.tool() and checks to make sure that the proper environment variables are loaded.

From there, it creates a TypeScript test file that is based on templates and generates dynamic test steps based on the target website's characteristics, including navigating to the website, verifying the page title exists, checking page load performance, taking a screenshot, verifying page content is visible, and finally saves the test file in a synthetic_tests directory.

Finally, it wraps Elastic's CLI tool @elastic/synthetics to push the test to Kibana, allowing you to set which geographic locations to run tests from, how often to run the test, and the project and workspace settings.

You check out the full code for this MCP tool here.

Creating synthetic tests based on a prompt

While creating browser tests based on a templated approach is a good starting point, it felt generic and cookie-cutter. But it made a helpful structure to build an LLM-based function on top of.

The MCP tool llm_create_and_deploy_test_from_prompt begins by ensuring that basic parameters, including locations, schedule, and directories, are listed. Additionally, it aims to learn more about the target website to inform the AI and initialize the OpenAI client and model, which is GPT-4o.

After setting up the LLM, it converts natural language requests into actual Playwright test code, then cleans and validates the AI-generated code to prevent issues like injection attacks or malformed syntax. It draws inspiration from the templated approach, wrapping AI-generated steps within a proven, reliable test framework template. Finally, it deploys the test to Elastic in a similar manner to the previous tool.

You can find the code for this tool here.

Conclusion and next steps

Synthetic monitoring in Elastic Observability makes it easy to test complete user journeys and keep your site reliable, with simple setup and a Playwright integration. A tool like this can provide a starting point for tests that you can iterate on after.

A solution like this is just the start of an MCP implementation that automatically generates Playwright tests for you and can be expanded in the future to include heartbeat monitors, utilize the Playwright MCP server, or consider experimenting with Claude for Chrome to create synthetic testing.

Check out more articles on Observability Labs on Infrastructure Monitoring

Despite their widespread utility of LLMs, deploying these sophisticated tools in real-world scenarios poses distinct challenges, especially in managing their complex behaviors. For users such as Site Reliability Engineers (SREs), DevOps teams, and AI/ML engineers, ensuring reliability, performance, and compliance of these models introduces an additional layer of complexity. This is where the concept of LLM Observability becomes essential. It offers crucial insights into the performance of these models, ensuring that these advanced AI systems operate both effectively and ethically.

Why LLM Observability Matters and How Elastic Makes It Easy

LLMs are not just another piece of software; they are sophisticated systems capable of human-like capabilities such as text generation, comprehension, and even coding. But with great power comes greater need for oversight. The opaque nature of these models can obscure how decisions are made and content generated. This makes it even more critical to implement robust observability to monitor and troubleshoot issues such as hallucinations, inappropriate content, cost overruns, errors and performance degradation. By monitoring these models closely, we can safeguard against unexpected outcomes and maintain user trust.

Real-World Scenarios

Let's explore real-world scenarios where companies leverage LLM-powered applications to enhance productivity and user experience, and how Elastic's LLM observability solutions monitor critical aspects of these models.

1. Generative AI for Customer Support

Companies are increasingly leveraging LLMs and generative AI to enhance customer support, using platforms like Google Vertex AI for hosting these models efficiently. With the introduction of advanced AI models such as Google's Gemini, which is integrated into Vertex AI, businesses can deploy sophisticated chatbots that manage customer inquiries, from basic questions to complex issues, in real time. These AI systems understand and respond with natural language, offering instant support for issues such as product troubleshooting or managing orders thus reducing wait times. They also learn from each interaction to improve accuracy continuously. This boosts customer satisfaction and allows human agents to focus on complex tasks, enhancing overall efficiency. Other ways that AI tools can further empower customer care agents is with real-time analytics, sentiment detection, and conversation summarization.

To support use cases like the AI-powered customer support described above, Elastic recently launched LLM observability integrations including support for LLMs hosted on GCP Vertex AI. Customers who wish to monitor foundation models such as Gemini and Imagen hosted on Google Vertex AI can benefit from Elastic’s Vertex AI integration to get a deeper understanding of model behavior and performance, and ensure that the AI-driven tools are not only effective but also reliable. Customers get out-of-the-box experience ingesting a curated set of metrics from Vertex AI as well as a pre-configured dashboard.

By continuously tracking these metrics, customers can proactively manage their AI resources, optimize operations, and ultimately enhance the overall customer experience.

Let's look at some of the metrics you get from the Google Vertex AI integration which are helpful in the context of using generative AI for customer support.

1. Prediction Latency: Measures the time taken to complete predictions, critical for real-time customer interactions.
2. Error Rate: Tracks errors in predictions, which is vital for maintaining the accuracy and reliability of AI-driven customer support.
3. Prediction Count: Counts the number of predictions made, helping assess the scale of AI usage in customer interactions.
4. Model Usage: Tracks how frequently the AI models are accessed by both virtual assistants and customer support tools.
5. Total Invocations: Measures the total number of times the AI services are used, providing insights into user engagement and dependency on these tools.
6. CPU and Memory Utilization: By observing CPU and memory usage, users can optimize resource allocation, ensuring that the AI tools are running efficiently without overloading the system.

To learn more about how Elastic's Google Vertex AI integration can augment your LLM observability, have a quick read of this blog.

2. Transforming Healthcare with Generative AI

The healthcare industry is embracing generative AI to enhance patient interactions and streamline operational workflows. By leveraging platforms like Amazon Bedrock, healthcare organizations deploy advanced large language models (LLMs) to power tools that convert doctor-patient conversations into structured medical notes, reducing administrative overhead and allowing clinicians to prioritize diagnosis and treatment. These AI-driven solutions provide real-time insights, enabling informed decision-making and improving patient outcomes. Additionally, patient-facing applications powered by LLMs offer secure access to health records, empowering individuals to manage their care proactively.

Robust observability is essential to maintain the reliability and performance of these generative AI applications in healthcare. Elastic’s Amazon Bedrock integration equips providers with tools to monitor LLM behavior, capturing critical metrics like invocation latency, error rates, token usage and guardrail invocation. Pre-configured dashboards provide visibility into prompt and completion text, enabling teams to verify the accuracy of AI-generated outputs, such as medical notes, and detect issues like hallucinations.

Additionally, customers who configure Guardrails for Amazon Bedrock to filter harmful content like hate speech, personal insults, and other inappropriate topics, can use the Bedrock Integration to observe the prompts and responses that caused the guardrail to filter them out. This helps application developers take proactive actions to maintain a safe and positive user experience.

Some of the logs and metrics that can be helpful for customers using LLMs hosted on Amazon Bedrock are the following

1. Invocation Details: This Integration records the Invocation latency, count, throttles. These metrics are critical for ensuring that generative AI models respond quickly and accurately to patient queries or appointment scheduling tasks, maintaining a seamless user experience.
2. Error Rates: Tracking error rates ensures that AI tools, such as patient query assistants or appointment systems, consistently deliver accurate and reliable results. By identifying and addressing issues early, healthcare providers can maintain trust in AI systems and prevent disruptions in critical patient interactions.
3. Token Usage: In healthcare, tracking token usage helps identify resource-intensive queries, such as detailed patient record summaries or complex symptom analyses, ensuring efficient model operation. By monitoring token usage, healthcare providers can optimize costs for AI-powered tools while maintaining scalability to handle growing patient interactions.
4. Prompt and Completion Text: Capturing prompt and completion text allows healthcare providers to analyze how AI models respond to specific patient queries or administrative tasks, ensuring meaningful and contextually accurate interactions. This insight helps refine prompts to improve the AI's understanding and ensures that generated responses, such as appointment details or treatment explanations, meet the quality standards expected in healthcare.
5. Prompt and response where guardrails intervened: Being able to track requests and responses that were deemed inappropriate by guardrails helps healthcare providers monitor what information patients are asking for. With this information users can make continuous adjustments to the LLMs to ensure appropriate responses, balancing flexibility and rich communication on the one hand, and on the other, privacy protection, hallucination prevention, and harmful content filtering.

Amazon Bedrock Gaurdrails OOTB dashboard

To learn about the Amazon Bedrock Integration, read this blog. To dive deeper into how the integration can help with observability of Guardrails for Amazon Bedrock, take a look at this blog.

3. Enhancing Telco Efficiency with GenAI

The telecommunication industry can leverage services like Azure OpenAI to transform customer interactions, optimize operations, and enhance service delivery. By integrating advanced generative AI models, telcos can offer highly personalized and responsive customer experiences across multiple channels. AI-powered virtual assistants streamline customer support by automating routine queries and providing accurate, context-aware responses, reducing the workload on human agents and enabling them to focus on complex issues while improving efficiency and satisfaction. Additionally, AI-driven insights help telcos understand customer preferences, anticipate needs, and deliver tailored offerings that boost customer loyalty. Operationally, LLMs such as Azure OpenAI enhance internal processes by enabling smarter knowledge management and faster access to critical information.

Elastic's LLM observability integrations like the Azure OpenAI integration can provide visibility into AI performance and costs, empowering telecom providers to make data-driven decisions and enhance customer engagement. It can help optimize resource allocation by analyzing call patterns, predicting service demands, and identifying trends, enabling telcos to scale their AI operations efficiently while maintaining high service quality.

Some of the key metrics and logs that Azure OpenAI that can provide insights are:

1. Error Counts: It provides critical insights into failed requests and incomplete transactions, enabling telecom providers to proactively identify and resolve issues in AI-powered applications.
2. Prompt Input and Completion Text: This captures the input queries provided to AI systems and the corresponding AI-generated outputs. These fields allow telecom providers to analyze customer queries, monitor response quality, and refine AI training datasets to improve relevance and accuracy.
3. Response Latency: It measures the time taken by AI models to generate responses, ensuring that virtual assistants and automated systems deliver quick and efficient replies to customer queries.
4. Token Usage: It tracks the number of input and output tokens processed by the AI model, offering insights into resource consumption and cost efficiency. This data helps telecom providers monitor AI usage patterns, optimize configurations, and scale resources effectively
5. Content Filter Results: In Azure OpenAI, this plays a crucial role in handling sensitive inputs provided by customers, ensuring compliance, safety, and responsible AI usage. This feature identifies and flags potentially inappropriate or harmful queries and responses in real time, enabling telecom providers to address sensitive topics with care and accuracy.

The Azure OpenAI content filtering OOTB dashboard

You can learn more about Elastic's Azure OpenAI integration from these two blogs - Part 1 and Part 2.

4. OpenAI Integration for Generative AI Applications

As AI-powered solutions become integral to modern workflows, OpenAI's sophisticated models, including language models like GPT-4o and GPT-3.5 Turbo, image generation models like DALL·E, and audio processing models like Whisper, drive innovation across applications such as virtual assistants, content creation, and speech-to-text systems. With growing complexity and scale, ensuring these models perform reliably, remain cost-efficient, and adhere to ethical guidelines is paramount. Elastic's OpenAI integration provides a robust solution, offering deep visibility into model behaviour to support seamless and responsible AI deployments.

By tapping into the OpenAI Usage API, Elastic's integration delivers actionable insights through intuitive, pre-configured dashboards, enabling Site Reliability Engineers (SREs) and DevOps teams to monitor performance and optimize resource usage across OpenAI's diverse model portfolio. This unified observability approach empowers organizations to track critical metrics, identify inefficiencies, and maintain high-quality AI-driven experiences. The following key metrics from Elastic's OpenAI integration help organizations achieve effective oversight:

1. Request Latency: Measures the time taken for OpenAI models to process requests, ensuring responsive performance for real-time applications like chatbots or transcription services.
2. Invocation Rates: Tracks the frequency of API calls across models, providing insights into usage patterns and helping identify high-demand workloads.
3. Token Usage: Monitors input and output tokens (e.g., prompt, completion, cached tokens) to optimize costs and fine-tune prompts for efficient resource consumption.
4. Error Counts: Captures failed requests or incomplete transactions, enabling proactive issue resolution to maintain application reliability.
5. Image Generation Metrics: Tracks invocation rates and output dimensions for models like DALL·E, helping assess costs and usage trends in image-based applications.
6. Audio Transcription Metrics: Monitors invocation rates and transcribed seconds for audio models like Whisper, supporting cost optimization in speech-to-text workflows.

To learn more about Elastic's OpenAI integration, read this blog.

Actionable LLM Observability

Elastic's LLM observability integrations empower users to take proactive control of their AI operations through actionable insights and real-time alerts. For instance, by setting a predefined threshold for token count, Elastic can trigger automated alerts when usage exceeds this limit, notifying Site Reliability Engineers (SREs) or DevOps teams via email, Slack, or other preferred channels. This ensures prompt awareness of potential cost overruns or resource-intensive queries, enabling teams to adjust model configurations or scale resources swiftly to maintain operational efficiency.

In the example below, the rule is set to alert the user if token_count crosses a threshold of 500.

The alert is triggered when the token count exceeds the threshold as seen below

Another example is tracking invocation spikes, such as when the number of predictions or API calls surpasses a defined Service Level Objective (SLO). For example, if a Bedrock AI-hosted model experiences a sudden surge in invocations due to increased customer interactions, Elastic can alert teams to investigate potential anomalies or scale infrastructure accordingly. These proactive measures help maintain the reliability and cost-effectiveness of LLM-powered applications.

By providing pre-configured dashboards and customizable alerts, Elastic ensures that organizations can respond to critical events in real time, keeping their AI systems aligned with cost and performance goals as well as standards for content safety and reliability.

Conclusion

LLMs are transforming industries, but their complexity requires effective oversight observability to ensure their reliability and safe use. Elastic's LLM observability integrations provide a comprehensive solution, empowering businesses to monitor performance, manage resources, and address challenges like hallucinations and content safety. As LLMs become increasingly integral to various sectors, robust observability tools like those offered by Elastic ensure that these AI-driven innovations remain dependable, cost-effective, and aligned with ethical and safety standards.

Despite the transformative potential, LLMs introduce complex challenges that necessitate a new level of observability as LLMs are notoriously opaque. Enter LLM observability: a crucial component in the lifecycle management of LLMs. This aspect becomes vital for Service Reliability Engineers (SREs) and other key stakeholders tasked with ensuring seamless, error-free operations, cost control, and minimizing the risks associated with the unpredictable nature of LLM generated responses. SREs need insights into performance metrics, error frequencies, latency issues, the cost implications of running these sophisticated models, and the prompt and response exchange with the model. Traditional monitoring tools fall short in this high-stakes environment; what’s needed is a nuanced approach to address the unique observability demands that LLMs introduce.

Elastic's LLM Observability Capabilities Address These Challenges

With Elastic’s end-to-end LLM observability you can cover a wide range of use cases. To achieve this, you can onboard two types of integrations - API-based logs and metrics and via APM instrumentation. Depending on your use case, you can also choose to use of the LLM integrations.

1. High level overview: via API-based logs and metrics. Monitoring LLM services from providers by ingesting a curated set of service metrics and logs like latency, invocation frequency, tokens, errors, and prompts and responses. Each LLM integration comes with out-of-the-box dashboards.

2. Troubleshooting applications: via APM instrumentation. Fully OTel-native tracing and auto-instrumentation for LLM-based applications through Elastic Distributions of OpenTelemetry (EDOT). Additionally, you can use third party libraries (Langtrace, OpenLit, OpenLLMetry) together with Elastic to extend the coverage to additional LLM-related technologies. 

High level overview: LLM Observability for Leading Providers

Elastic offers tailored API-based integrations for four major LLM hosting providers:

• Azure OpenAI

• OpenAI

• Amazon Bedrock

• Google Vertex AI

These integrations bring a curated set of logs and metrics collection tailored to each provider. What this means for SREs is straightforward access to pre-configured dashboards that highlight the prompts and responses, usage patterns, performance metrics, and cost details across different models and providers.

For instance, SREs keen on identifying which LLM generates the most errors or insights about the models in terms of latency, cost, or usage frequency can leverage these integrations. Imagine having the capability to instantly visualize which LLM is slowing down processes or incurring high costs, thus enabling data-driven decisions to optimize operations.

Troubleshooting applications: Tracing and Auto-Instrumentation of OpenAI, Amazon Bedrock and Google Vertex AI models

Elastic supports OTLP tracing capabilities in EDOT for applications using OpenAI models and models hosted on Amazon Bedrock and Google Vertex AI. In addition, Elastic also supports LLM tracing from third party libraries (Langtrace, OpenLIT, OpenLLMetry). 

Tracing offers a comprehensive map of an application's request flow, pinpointing granular details about each call within the system. For each transaction and span of a request, tracing shows critical information such as specific models utilized, request duration, errors encountered, tokens used per request, and the prompts and responses between the LLM.

Tracing helps SREs troubleshoot performance issues with applications developed in languages like Python, Node.js and Java." If an SRE needs to investigate latency or error issues, LLM tracing provides a zoomed-in view into the request lifecycle and allows for profound insights into whether a delay is application-specific, model-specific or systemic across deployments.

Use Cases: Bringing Elastic's Observability Features to Life

Let’s explore some practical scenarios where Elastic’s observability tools shine:

1. Understanding LLM Performance and Reliability

An SRE team looking to optimize a customer support system powered by Azure OpenAI can utilize Elastic’s Azure OpenAI integration to quickly ascertain which model variants incur higher latency or error rates. This enhances decision-making regarding model deployment or even switching providers based on performance metrics.

Similarly SREs can also use in parallel integrations for Google Vertex AI, Amazon Bedrock, and OpenAI for other applications using models hosted on these providers.

2. Troubleshooting OpenAI-Powered Applications

Consider an enterprise utilizing an OpenAI model for real-time user interactions. Encountering unexplained delays, an SRE can use OpenAI tracing to dissect the transaction pathway, identifying if one specific API call or model invocation is the bottleneck. The SRE can also check the out-of-the-box OpenAI integration dashboard to verify if the latency is only affecting this application or all model invocations across the organization.

An engineer troubleshooting the LLM-based application can also check to see what were the prompt and response exchanges with the LLM during this request so they can rule out possible impact on performance due to the input.

3. Addressing Cost and Usage Concerns

SREs are generally acutely aware of which LLM configurations are less cost-effective than required. Elastic’s integration dashboards, pre-configured to display model usage patterns, help mitigate unnecessary spending effectively. You can find out-of-the box dashboards for Azure OpenAI, OpenAI, Amazon Bedrock, and Google VertexAI models. These dashboards show key cost and usage information such as total invocations and tokens, as well as time series breakdown by model and endpoint. In addition, some integrations show more advanced usage information such as provisioned throughput units (PTU) as well as billing cost.

4. Understanding LLM Compliance 

With the Elastic Amazon Bedrock integration for Guardrails, and Azure OpenAI integration for content filtering, SREs can swiftly address security concerns, like verifying if certain user interactions prompt policy violations. Elastic's observability logs clarify whether guardrails rightly blocked potentially harmful responses, bolstering compliance assurance.

Conclusion

As LLMs continue to revolutionize the capabilities of modern applications, the role of observability becomes increasingly paramount. Elastic’s comprehensive observability framework empowers enterprises to harness the full potential of LLMs while maintaining robust operational insight and control. The integration with prominent LLM hosting providers and advanced tracing for OpenAI, Amazon Bedrock and Google Vertex AI models, equips SREs with the necessary arsenal to navigate the complex landscape of LLM-driven applications, ensuring they remain safe, reliable, efficient, and cost-effective.

In this new era of AI, balancing innovation with observability isn't just beneficial—it's essential. Whether optimizing performance, troubleshooting intricacies, or managing costs and compliance, Elastic stands at the forefront, ensuring your LLM journey is as seamless as it is groundbreaking.

The investment paradox of observability in 2025

Here's something fascinating: 96% of executives in our survey expect observability to remain a key investment area. Yet almost all of them (97%) are hitting roadblocks in realizing full value. And surprisingly, the primary hurdles for observability are not technical or complicated in nature, can you guess what they might be?

For 2025, IT leaders are challenged with financial hurdles for their observability. I'm seeing this tension play out constantly in conversations with leaders - they know they need to invest, but they're grappling with budget constraints, licensing costs, and proving ROI for their organizations. This creates an interesting dynamic where organizations must carefully balance increasing investment with rigorous cost optimization and business metrics.

What's particularly interesting is how this paradox is forcing organizations to become more strategic about their investments. Leaders are no longer just throwing money at the problem - they're thinking carefully about how to maximize value from every dollar spent.

Why observability maturity is making all the difference

The data really jumps out at me here. The gap between observability experts and newcomers tells a compelling story that I wasn't expecting to see. Expert organizations are significantly outperforming their peers across every key metric:

• 91% of expert organizations are deploying applications and infrastructure faster (compared to just 34% of those in early stages)

• 82% are successfully reducing operational costs (versus 56% of early-stage organizations)

• 71% achieve better MTTR for incidents (while only 40% of early-stage organizations do)

What I find particularly fascinating is how some benefits go beyond just maturity levels. About 80% of organizations report better customer issue response times regardless of their maturity stage. It tells me that even basic observability delivers immediate customer-facing value. This is crucial information for organizations just starting their observability journey - they can expect to see tangible benefits right from the start. But the overarching story may be that observability maturity leads teams from reactive to proactive and allows them to focus on higher level, value-add activities.

Cost management: the new imperative

The numbers around cost management paint a clear picture of where the industry is heading - 97% of IT decision-makers are actively managing observability costs, and 86% feel personally responsible for business outcomes.

I'm seeing a clear trend where leaders are taking concrete steps in their day to day work:

• Consolidating their observability toolset while maintaining capabilities, they don’t want to lose anything
• Implementing usage-based pricing models
• Establishing clear ROI metrics
• Creating cross-functional teams to optimize spending

This isn't just about cutting costs - it's about being smarter with resources. Organizations are learning that more tools don't necessarily mean better observability.

Two technologies reshaping the observability landscape

AI's growing impact

The enthusiasm for AI is remarkable - 94% of respondents see its tremendous potential. What fascinates me is how concerns about Generative AI reliability have actually decreased from 64% to 55% over the past year.

Leaders are particularly excited about:

• Automated correlation of logs, metrics, and traces (72% of respondents)
• Predictive analytics for preventing outages
• Natural language interfaces for querying observability data
• Automated root cause analysis

The key shift I'm seeing for the upcoming year is the move from AI as a buzzword to AI as a practical tool delivering real value in observability workflows.

Generative AI capabilities paired with retrieval augmented generation (RAG) capabilities allow organizations to leverage the power of LLMs and private data (e.g., runbooks, alerts, business data) to deliver relevant and meaningful results and identify and solve problems faster while reducing noise.

OpenTelemetry's continued momentum

Looking at expert organizations, 80% are either experimenting with or have deployed OpenTelemetry. This isn't just about technology adoption - it's about building for the future with open standards. The correlation between OpenTelemetry adoption and overall observability maturity is correlated and unmistakable.

What's particularly interesting is how OpenTelemetry is changing the vendor landscape. Organizations are increasingly demanding OpenTelemetry support from their vendors, seeing it as a way to future-proof their observability investments and avoid vendor lock-in. Thinking back to how Linux shifted the server landscape, can we expect to see the same in the observability domain?

Business integration and insights deepens

Here's what I find most compelling: 64% of expert organizations are frequently correlating operational data with business outcomes, while only 9% of early-stage organizations do the same. This represents a fundamental shift from technical monitoring to business observability.

This isn't just about uptime anymore - organizations are increasingly using observability data to:

• Make informed business decisions
• Improve customer experience
• Optimize resource allocation
• Drive innovation

Looking ahead

As we continue through 2025, I'm seeing observability mature beyond its initial promise. Organizations are focusing less on basic implementation and more on delivering real business value through:

• Deeper business integration, like mapping system performance directly to revenue metrics
• Optimized cost management through new data lake technology, efficient storage and intelligent retention
• AI-enhanced capabilities powered by LLMs and Agentic AI
• Standardized instrumentation through OpenTelemetry, reducing vendor lock-in

The path to success in 2025 isn't just about having the right tools - it's about building mature practices that deliver measurable business value while managing costs effectively. The organizations that can balance these competing demands while maintaining focus on business outcomes are the ones pulling ahead.

What are you seeing in your organization's observability journey? Are these trends aligning with your experience?

If you would like to dig in deeper on emerging observability trends, download our full report or watch the on-demand webinar, 2025 Observability trends: Maturing beyond the hype and delivering results!

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic provides the ability to ingest OpenTelemetry data with native OTel SDKs, the off the shelf OTel collector, or even Elastic’s Distributions of OpenTelemetry (EDOT). EDOT enables you to bring in logs, metrics and traces for your GenAI application and for K8s. However you will also generally need libraries to help trace specific components in your application. In tracing GenAI applications you can pick from a large set of libraries.

• OpenTelemetry OpenAI Instrumentation-v2 - allows tracing LLM requests and logging of messages made by the OpenAI Python API library. (note v2 is built by OpenTelemetry, the non v2 version is from a specific vendor and not OpenTelemetry)

• OpenTelemetry VertexAI Instrumentation - allows tracing LLM requests and logging of messages made by the VertexAI Python API library

• Langtrace - commercially available library which supports all LLMs in one library, and all traces are also OTel native.

• Elastic’s EDOT - which recently added tracing. See blog.

As you can see OpenTelemetry is the defacto mechanism that is converging to collect and ingest. OpenTelemetry is growing its support for this but it is also early days.

In this blog, we will walk through how to, with minimal code, observe a RAG based chatbot application with tracing using Langtrace. We previously covered Langtrace in a blog to highlight tracing Langchain.

In this blog we used langtrace OpenAI, Amazon Bedrock, Cohere, and others in one library.

Pre-requisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar.

• An account on your favorite LLM (OpenAI, AzureOpen AI, etc), with API keys

• Be familiar with EDOT to understand how we bring in logs, metrics, and traces from the application through the OTel Collector

• Kubernetes cluster - I’ll be using Amazon EKS

• Look at Langtrace documentation also.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have OTel Collector with EDOT configuration on your K8s cluster, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns (saves you time in analysis), and view logs from K8s.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up

In order to properly set up the Chatbot-app on K8s with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app, and modify one of the python files.

2. Next create a docker container that can be used in Kubernetes. The Docker build here in the Chatbot-app is good to use.

3. Collect all needed env variables. In this example we are using OpenAI, but the files can be modified for any of the LLMs. Hence you will have to get a few environmental variables loaded into the cluster. In the github repo there is a env.example for docker. You can pick and chose what is needed or not needed and adjust appropriately in the K8s file below.

4. Set up your K8s Cluster, and then install the OpenTelemetry collector with the appropriate yaml file and credentials. This will help collect K8s cluster logs and metrics also.

5. Utilize the two yaml files listed below to ensure you can run it on Kubernetes.

• Init-index-job.yaml - Initiates the index in elasticsearch with the local corporate information

• k8s-deployment-chatbot-rag-app.yaml - initializes the application frontend and backend.

1. Open the app on the load balancer URL against the chatbot-app service in K8s

2. Go to Elasticsearch and look at Discover for logs, go to APM and look for your chatbot-app and review the traces, and finally.

Modify the code for tracing with Langtrace

Once you curl the app and untar, go to the chatbot-rag-app directory:

curl https://codeload.github.com/elastic/elasticsearch-labs/tar.gz/main | 
tar -xz --strip=2 elasticsearch-labs-main/example-apps/chatbot-rag-app
cd elasticsearch-labs-main/example-apps/chatbot-rag-app

Next open the app.py file in the api directory and add the following

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

FlaskInstrumentor().instrument_app(app)

into the code:

import os
import sys
from uuid import uuid4

from chat import ask_question
from flask import Flask, Response, jsonify, request
from flask_cors import CORS

from opentelemetry.instrumentation.flask import FlaskInstrumentor

from langtrace_python_sdk import langtrace

langtrace.init(batch=False)

app = Flask(__name__, static_folder="../frontend/build", static_url_path="/")
CORS(app)

FlaskInstrumentor().instrument_app(app)

@app.route("/")

See the items in BOLD which will add in the langtrace library, and the opentelemetry flask instrumentation. This combination will provide and end to end trace for the https call all the way down to the calls to Elasticsearch, and to OpenAI (or other LLMs).

Create the docker container

Use the Dockerfile that is in the chatbot-rag-app directory as is and add the following line:

RUN pip3 install --no-cache-dir langtrace-python-sdk

into the Dockerfile:

COPY requirements.txt ./requirements.txt
RUN pip3 install -r ./requirements.txt
RUN pip3 install --no-cache-dir langtrace-python-sdk
COPY api ./api
COPY data ./data

EXPOSE 4000

This enables the langtrace-python-sdk to be installed into the docker container so the langtrace libraries can be used properly.

Collecting the proper env variables:

First collect the env variables from Elastic:

Envs for index initialization in Elastic:

ELASTICSEARCH_URL=https://aws.us-west-2.aws.found.io
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASSWORD=elastic

# The name of the Elasticsearch indexes
ES_INDEX=workplace-app-docs
ES_INDEX_CHAT_HISTORY=workplace-app-docs-chat-history

The ELASTICSEARCH_URL can be found in cloud.elastic.co when you bring up your instance. The user and password, you will need to setup in Elastic.

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer xxxxx"

These credentials are found in Elastic under APM integration and under OpenTelemetry

Envs for LLMs

In this example we’re using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

All these variables will be needed in the Kubernetes yamls in the next step

Setup K8s cluster and load up OTel Collector with EDOT

This step is outlined in the following Blog. It’s a simple three step process.

This step will bring in all the K8s cluster logs and metrics and setup the OTel collector.

Setup secrets, initialize indices, and start the app

Now that the cluster is up, and you have your environmental variables, you will need to

1. Install and run the k8s-deployments.yaml with the variables

2. Initialize the index

Essentially run the following:

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

Here are the two yamls you should use. Also found here

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: genai-chatbot-langtrace-secrets
type: Opaque
stringData:
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://1234567.apm.us-west-2.aws.cloud.es.io:443"
 ELASTICSEARCH_URL: "YOUR_ELASTIC_SEARCH_URL"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OPENAI_API_KEY: "XXXXXXX"  

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-chatbot-langtrace
spec:
  replicas: 2
  selector:
    matchLabels:
      app: genai-chatbot-langtrace
  template:
    metadata:
      labels:
        app: genai-chatbot-langtrace
    spec:
      containers:
      - name: genai-chatbot-langtrace
        image:65765.amazonaws.com/genai-chatbot-langtrace2:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=genai-chatbot-langtrace,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        envFrom:
        - secretRef:
            name: genai-chatbot-langtrace-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: genai-chatbot-langtrace-service
spec:
  selector:
    app: genai-chatbot-langtrace
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
#update your image location for chatbot rag app
        image: your-image-location:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-langtrace-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

Play with app and review telemetry in Elastic

Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel, Elastic’s EDOT and Langtrace gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this.

Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

This app is also now insturmented with EDOT, and you can visualize the Chatbot's traces to OpenAI, as well as relevant logs, and metrics from the application. By running the app as instructed in the github repo with Docker you can see these traces on a local stack. But how about running it against serverless, Elastic cloud or even with Kubernetes?

In this blog we will walk through how to set up Elastic's RAG Based Chatbot application with Elastic cloud and Kubernetes.

Prerequisites:

In order to follow along, these few pre-requisites are needed

• An Elastic Cloud account — sign up now, and become familiar with Elastic's OpenTelemetry configuration. With Serverless no version required. With regular cloud minimally 8.17

• Git clone the RAG based Chatbot application and go through the tutorial on how to bring it up and become more familiar and how to bring up the application using Docker.

• An account on OpenAI with API keys

• Kubernetes cluster to run the RAG based Chatbot app

• The instructions in this blog are also found in observability-examples in github.

Application OpenTelemetry output in Elastic

Chatbot-rag-app

The first item that you will need to get up and running is the ChatBotApp, and once up you should see the following:

As you select some of the questions you will set a response based on the index that was created in Elasticsearch when the app initializes. Additionally there will be queries that are made to LLMs.

Traces, logs, and metrics from EDOT in Elastic

Once you have the application running on your K8s cluster or with Docker, and Elastic Cloud up and running you should see the following:

Logs:

In Discover you will see logs from the Chatbotapp, and be able to analyze the application logs, any specific log patterns, which saves you time in analysis.

Traces:

In Elastic Observability APM, you can also see tha chatbot details, which include transactions, dependencies, logs, errors, etc.

When you look at traces, you will be able to see the chatbot interactions in the trace.

1. You will see the end to end http call

2. Individual calls to elasticsearch

3. Specific calls such as invoke actions, and calls to the LLM

You can also get individual details of the traces, and look at related logs, and metrics related to that trace,

Metrics:

In addition to logs, and traces, any instrumented metrics will also get ingested into Elastic.

Setting it all up with Docker

In order to properly set up the Chatbot-app on Docker with telemetry sent over to Elastic, a few things must be set up:

1. Git clone the chatbot-rag-app

2. Modify the env file as noted in the github README with the following exception:

Use your Elastic cloud's OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADER instead.

You can find these in the Elastic Cloud under integrations->APM

Envs for sending the OTel instrumentation you will need the following:

OTEL_EXPORTER_OTLP_ENDPOINT="https://123456789.apm.us-west-2.aws.cloud.es.io:443"
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20xxxxx"

Notice the %20 in the headers. This will be needed to account for the space in credentials.

1. Set the following to false - OTEL_SDK_DISABLED=false

2. Set the envs for LLMs

In this example we're using OpenAI, hence only three variables are needed.

LLM_TYPE=openai
OPENAI_API_KEY=XXXX
CHAT_MODEL=gpt-4o-mini

1. Run the docker container as noted

docker compose up --build --force-recreate

1. Play with the app at localhost:4000

2. Then log into Elastic cloud and see the output as shown previously.

Run chatbot-rag-app on Kubernetes

In order to set this up, you can follow the following repo on Observability-examples which has the Kubernetes yaml files being used. These will also point to Elastic Cloud.

1. Set up the Kubernetes Cluster (we're using EKS)

2. Get the appropriate ENV variables:

• Find the OTEL_EXPORTER_OTLP_ENDPOINT/HEADER variables as noted in the pervious for Docker.

• Get your OpenAI Key

• Elasticsearch URL, and username and password.

1. Follow the instructions in the following github repo in observability examples to run two Kubernetes yaml files.

Essentially you need only replace the secret variables in k8s-deployment.yaml, and run

kubectl create -f k8s-deployment.yaml
kubectl create -f init-index-job.yaml

The app needs to be running first, then we use the app to initialize Elasticsearch with indices for the app.

Init-index-job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: init-elasticsearch-index-test
spec:
  template:
    spec:
      containers:
      - name: init-index
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        workingDir: /app/api
        command: ["python3", "-m", "flask", "--app", "app", "create-index"]
        env:
        - name: FLASK_APP
          value: "app"
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: ES_INDEX
          value: "workplace-app-docs"
        - name: ES_INDEX_CHAT_HISTORY
          value: "workplace-app-docs-chat-history"
        - name: ELASTICSEARCH_URL
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_URL
        - name: ELASTICSEARCH_USER
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_USER
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: chatbot-regular-secrets
              key: ELASTICSEARCH_PASSWORD
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
      restartPolicy: Never
  backoffLimit: 4

k8s-deployment.yaml

apiVersion: v1
kind: Secret
metadata:
  name: chatbot-regular-secrets
type: Opaque
stringData:
  ELASTICSEARCH_URL: "https://yourelasticcloud.es.us-west-2.aws.found.io"
  ELASTICSEARCH_USER: "elastic"
  ELASTICSEARCH_PASSWORD: "elastic"
  OTEL_EXPORTER_OTLP_HEADERS: "Authorization=Bearer%20xxxx"
  OTEL_EXPORTER_OTLP_ENDPOINT: "https://12345.apm.us-west-2.aws.cloud.es.io:443"
  OPENAI_API_KEY: "YYYYYYYY"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatbot-regular
spec:
  replicas: 2
  selector:
    matchLabels:
      app: chatbot-regular
  template:
    metadata:
      labels:
        app: chatbot-regular
    spec:
      containers:
      - name: chatbot-regular
        image: ghcr.io/elastic/elasticsearch-labs/chatbot-rag-app:latest
        ports:
        - containerPort: 4000
        env:
        - name: LLM_TYPE
          value: "openai"
        - name: CHAT_MODEL
          value: "gpt-4o-mini"
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.name=chatbot-regular,service.version=0.0.1,deployment.environment=dev"
        - name: OTEL_SDK_DISABLED
          value: "false"
        - name: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
          value: "true"
        - name: OTEL_EXPERIMENTAL_RESOURCE_DETECTORS
          value: "process_runtime,os,otel,telemetry_distro"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "http/protobuf"
        - name: OTEL_METRIC_EXPORT_INTERVAL
          value: "3000"
        - name: OTEL_BSP_SCHEDULE_DELAY
          value: "3000"
        envFrom:
        - secretRef:
            name: chatbot-regular-secrets
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"

---
apiVersion: v1
kind: Service
metadata:
  name: chatbot-regular-service
spec:
  selector:
    app: chatbot-regular
  ports:
  - port: 80
    targetPort: 4000
  type: LoadBalancer

Open App with LoadBalancer URL

Run the kubectl get services command and get the URL for the chatbot app

% kubectl get services
NAME                                 TYPE           CLUSTER-IP    EXTERNAL-IP                                                               PORT(S)                                                                     AGE
chatbot-regular-service            LoadBalancer   10.100.130.44    xxxxxxxxx-1515488226.us-west-2.elb.amazonaws.com   80:30748/TCP                                                                6d23h

1. Play with app and review telemetry in Elastic

2. Once you go to the URL, you should see all the screens we described earlier in the beginning of this blog.

Conclusion

With Elastic's Chatbot-rag-app you have an example of how to build out a OpenAI driven RAG based chat application. However, you still need to understand how well it performs, whether its working properly, etc. Using OTel and Elastic’s EDOT gives you the ability to achieve this. Additionally, you will generally run this application on Kubernetes. Hopefully this blog provides the outline of how to achieve this. Here are the other Tracing blogs:

App Observability with LLM (Tracing)-

• Observing LangChain with Langtrace and OpenTelemetry

• Observing LangChain with OpenLit Tracing

• Instrumenting LangChain with OpenTelemetry

LLM Observability -

• Elevate LLM Observability with GCP Vertex AI Integration

• LLM Observability on AWS Bedrock

• LLM Observability for Azure OpenAI

• LLM Observability for Azure OpenAI v2

Last year, we announced Elastic Distribution of OpenTelemetry (a.k.a. EDOT) language SDKs, which collect logs, traces and metrics from applications. When this was announced, we didn’t yet support Large Language Model (LLM) providers such as OpenAI. This limited insight developers had into Generative AI (GenAI) applications.

In a prior post, we reviewed LLM observability focus, such as token usage, chat latency and knowing which tools (like DuckDuckGo) your application uses. With the right logs, traces and metrics, developers can answer questions like "Which version of a model generated this response?" or "What was the exact chat prompt created by my RAG application?"

In the last six months, Elastic invested a lot of energy alongside others in the OpenTelemetry community towards shared specifications on these areas, including code to collect LLM related logs, metrics and traces. Our goal was to extend the zero code (agent) approach EDOT brings to GenAI use cases.

Today, we announce our first GenAI instrumentation capability in the EDOT language SDKs: OpenAI. Below, you’ll see how to observe GenAI applications using our Python, Node.js and Java EDOT SDKs.

Example application

Many of us may be familiar with ChatGPT, which is frontend for OpenAI’s GPT model family. Using this, you can ask a question and the assistant might reply correctly depending on what you ask and text the LLM was trained on.

Here’s an example of an esoteric question answered by ChatGPT:

Our example application will simply ask this predefined question and print the result. We’ll write it in three languages: Python, JavaScript and Java.

We’ll execute each with a "zero code" (agent) approach, so that logs, metrics and traces are captured and visible in an Elastic Stack configured with Kibana and APM server. If you don’t have a stack running, use instructions from Elasticsearch Labs to set one up.

Regardless of programming language, three variables are needed: the OpenAI API key, the location of your Elastic APM server, and the service name of the application. You’ll write these to a file named .env.

OPENAI_API_KEY=sk-YOUR_API_KEY
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200
OTEL_SERVICE_NAME=openai-example

By default instrumentations does not capture the content sent to the OpenAI API in the GenAI events sent to logs, if you want to capture it add the following:

OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true

Each time the application is run, it sends logs, traces and metrics to the APM server, which you can find by querying Kibana like this for the application "openai-example"

http://localhost:5601/app/apm/services/openai-example/transactions

When you choose a trace, you’ll see the LLM request made by the OpenAI SDK, and HTTP traffic caused by it:

Select the logs tab to see the exact request and response to OpenAI. This data is critical for Q/A and evaluation use cases.

You can also go to the Metrics Explorer and make a graph of "gen_ai.client.token.usage" or "gen_ai.client.operation.duration" over all the times you ran the application:

http://localhost:5601/app/metrics/explorer

Continue to see exactly how this application looks and is run, in Python, Java and Node.js. Those already using our EDOT language SDKs will be familiar with how this works.

Python

Assuming you have python installed, the first thing would be to setup a virtual environment and install the required packages: the OpenAI client, a helper tool to read the .env file and our EDOT Python package:

python3 -m venv .venv
source .venv/bin/activate
pip install openai "python-dotenv[cli]" elastic-opentelemetry

Next, run edot-bootstrap which analyzes the code to install any relevant instrumentation available:

edot-bootstrap —-action=install

Now, create your .envfile, as described earlier in this article, and the below source code in chat.py

import os

import openai

CHAT_MODEL = os.environ.get("CHAT_MODEL", "gpt-4o-mini")


def main():
  client = openai.Client()

  messages = [
    {
      "role": "user",
        "content": "Answer in up to 3 words: Which ocean contains Bouvet Island?",
    }
  ]

  chat_completion = client.chat.completions.create(model=CHAT_MODEL, messages=messages)
  print(chat_completion.choices[0].message.content)

if __name__ == "__main__":
  main()

Now you can run everything with:

dotenv run -- opentelemetry-instrument python chat.py

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example (along with the instructions) in the Python EDOT repository here.

Finally, if you would like to try a more comprehensive example, take a look at chatbot-rag-app which uses OpenAI with Elasticsearch’s Elser retrieval model.

Java

There are multiple popular ways to initialize a Java project. Since we are using OpenAI, the first step is to configure the dependency com.openai:openai-java and write the below source as Chat.java.

package openai.example;

import com.openai.client.OpenAIClient;
import com.openai.client.okhttp.OpenAIOkHttpClient;
import com.openai.models.*;


final class Chat {

  public static void main(String[] args) {
    String chatModel = System.getenv().getOrDefault("CHAT_MODEL", "gpt-4o-mini");

    OpenAIClient client = OpenAIOkHttpClient.fromEnv();

    String message = "Answer in up to 3 words: Which ocean contains Bouvet Island?";
    ChatCompletionCreateParams params = ChatCompletionCreateParams.builder()
        .addMessage(ChatCompletionUserMessageParam.builder()
          .content(message)
          .build())
        .model(chatModel)
        .build();

    ChatCompletion chatCompletion = client.chat().completions().create(params);
    System.out.println(chatCompletion.choices().get(0).message().content().get());
  }
}

Build the project such that all dependencies are in a single jar. For example, if using Gradle, you would use the com.gradleup.shadowplugin.

Next, create your .envfile, as described earlier, and download shdotenv which we’ll use to load it.

curl -O -L https://github.com/ko1nksm/shdotenv/releases/download/v0.14.0/shdotenv
chmod +x ./shdotenv

At this point, you have a jar and configuration you can use to run the OpenAI example. The next step is to download the EDOT Java javaagent binary. This is the part that records and exports logs, metrics and traces.

curl -o elastic-otel-javaagent.jar -L 'https://oss.sonatype.org/service/local/artifact/maven/redirect?r=snapshots&g=co.elastic.otel&a=elastic-otel-javaagent&v=LATEST'

Assuming you assembled a file named openai-example-all.jar, run it with EDOT like this:

./shdotenv java -javaagent:elastic-otel-javaagent.jar -jar openai-example-all.jar

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example in the EDOT Java source repository here.

Node.js

Assuming you already have npm installed and configured, run the following commands to initialize a project for the example. This includes the openai package and @elastic/opentelemetry-node (EDOT Node.js)

npm init -y
npm install openai @elastic/opentelemetry-node

Next, create your .env file, as described earlier in this article and the below source code in index.js

const {OpenAI} = require('openai');

let chatModel = process.env.CHAT_MODEL ?? 'gpt-4o-mini';

async function main() {
 const client = new OpenAI();
 const completion = await client.chat.completions.create({
  model: chatModel,
  messages: [
   {
    role: 'user',
    content: 'Answer in up to 3 words: Which ocean contains Bouvet Island?',
   },
  ],
 });
 console.log(completion.choices[0].message.content);
}

main();

With this in place, run the above source with EDOT like this:

node --env-file .env --require @elastic/opentelemetry-node index.js

Finally, look for a trace for the service named "openai-example" in Kibana. You should see a transaction named "chat gpt-4o-mini".

Rather than copy/pasting above, you can find a working copy of this example in the EDOT Node.js source repository here.

Finally, if you would like to try a more comprehensive example, take a look at openai-embeddings which uses OpenAI with Elasticsearch as a vector database!

Closing Notes

Above you’ve seen how to observe the official OpenAI SDK in three different languages, using Elastic Distribution of OpenTelemetry (EDOT).

It is important to note that some of the OpenAI SDKs and also OpenTelemetry specifications around generative AI are experimental. If you find this helps you, or find glitches, please join our slack and let us know about it.

Several LLM platforms accept requests from the OpenAI client SDK, by setting OPENAI_BASE_URL and choosing relevant models. During development, we tested against OpenAI Platform and Azure OpenAI Service. We also ran integration tests against Ollama, contributing improvements its OpenAI support released in v0.5.12. Whatever your choice of OpenAI compatible platform, we hope this new tooling helps you understand your LLM usage.

Finally, while the first Generative AI SDK instrumented with EDOT is OpenAI, you’ll see more soon. We are already working on Bedrock, and collaborating with others in the OpenTelemetry community for other platforms. Keep watching this blog for exciting updates.

There are several options to trace for Langchain. Langtrace is one such option. Langtrace is an open-source observability software that lets you capture, debug and analyze traces and metrics from all your applications. Langtrace automatically captures traces from LLM APIs/inferences, Vector Databases, and LLM-based Frameworks. Langtrace stands out due to its seamless integration with popular LLM frameworks and its ability to provide deep insights into complex AI workflows without requiring extensive manual instrumentation.

Langtrace has an SDK, a lightweight library that can be installed and imported into your project to collect traces. The traces are OpenTelemetry-based and can be exported to Elastic without using a Langtrace API key.

OpenTelemetry (OTel) is now broadly accepted as the industry standard for tracing. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. 

Hence, many LangChain-based applications will have multiple components beyond just LLM interactions. Using OpenTelemetry with LangChain is essential. 

This blog will cover how you can use Langtrace SDK to trace a simple LangChain Chat app connecting to Azure OpenAI, perform a search in DuckDuckGoSearch and export the output to Elastic.

Pre-requisites:

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration

• Have a LangChain app to instrument

• Be familiar with using OpenTelemetry’s Python SDK 

• An account on your favorite LLM (AzureOpen AI), with API keys

• The application we used in this blog, called langchainChat can be found in Github langhcainChat. It is built using Azure OpenAI and DuckDuckGo, but you can easily modify it for your LLM and search of choice.

App Overview and output in Elastic:

To showcase the combined power of Langtrace and Elastic, we created a simple LangChain app that performs the following steps:

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a LangChain.

3. Utilizes chain tools to perform a search using DuckDuckGo.

4. The LLM processes the search results and returns the relevant information to the user.

Here is a sample interaction:

Here is what the service view looks like after we ran a few queries.

As you can see, Elastic Observability’s APM recognizes the LangChain app and also shows the average latency, throughput, and transactions. Our average latency is 30s since it takes that log for humans to type the query (twice).

You can also select other tabs to see, dependencies, errors, metrics, and more. One interesting part of Elastic APM is the ability to use universal profiling (eBPF) output also analyzed for this service. Here is what our service’s dependency is (Azure OpenAI) with its average latency, throughput, and failed transactions:

We see Azure OpenAI is on average 4s to give us the results.

If we drill into transactions and look at the trace for our queries on Taylor Swift and Pittsburgh Steelers, we can see both queries and their corresponding spans.

In this trace:

1. The user makes a query

2. Azure OpenAI is called, but it uses a tool (DuckDuckGo) to obtain some results

3. Azure OpenAI reviews and returns a summary to the end user

4. Repeats for another query

We noticed that the other long span (other than Azure OpenAI) is Duckduckgo (~1000ms). We can individually look at the span and review the data:

Configuration:

How do we make all this show up in Elastic? Let's go over the steps:

OpenTelemetry Configuration

To leverage the full capabilities of OpenTelemetry with Langtrace and Elastic, we need to configure the SDK to generate traces and properly set up Elastic’s endpoint and authorization. Detailed instructions can be found in the OpenTelemetry Auto-Instrumentation setup documentation.

OpenTelemetry Environment variables:

For Elastic, you can set the following OpenTelemetry environment variables either in your Linux/Mac environment or directly in the code:

OTEL_EXPORTER_OTLP_ENDPOINT=12345.apm.us-west-2.aws.cloud.es.io:443
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20ZZZZZZZ"
OTEL_RESOURCE_ATTRIBUTES="service.name=langchainChat,service.version=1.0,deployment.environment=production"

In this setup:

• OTEL_EXPORTER_OTLP_ENDPOINT is configured to send traces to Elastic.

• OTEL_EXPORTER_OTLP_HEADERS provides the necessary authorization for the Elastic APM server.

• OTEL_RESOURCE_ATTRIBUTES define key attributes like the service name, version, and deployment environment.

These values can be easily obtained from Elastic’s APM configuration screen under the OpenTelemetry section.

Note: No agent is required; the OTLP trace messages are sent directly to Elastic’s APM server, simplifying the setup process.

Langtrace Library:

OpenTelemetry's auto-instrumentation can be extended to trace additional frameworks using instrumentation packages. For this blog post, you will need to install the Langtrace Python SDK:

pip install langtrace-python-sdk 

After installation, you can add the following code to your project:

from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

from langtrace_python_sdk import langtrace, with_langtrace_root_span

Instrumentation:

Once the necessary libraries are installed and the environment variables are configured, you can use auto-instrumentation to trace your application. For example, run the following command to instrument your LangChain application with Elastic:

opentelemetry-instrument python langtrace-elastic-demo.py

The Langtrace OpenTelemetry library correctly captures the flow with minimal manual instrumentation, apart from integrating the OpenTelemetry library. Additionally, the LLM spans captured by Langtrace also include useful metadata such as token counts, model hyper-parameter settings etc. Note that the generated spans follow the OTEL GenAI semantics described here.

In summary, the instrumentation process involves:

1. Capturing customer input from the command line (Queries).

2. Sending these queries to the Azure OpenAI LLM via a LangChain.

3. Utilizing chain tools, such as DuckDuckGo, to perform searches.

4. The LLM processes the results and returns the relevant information to the user.

Conclusion

By combining the power of Langtrace with Elastic, developers can achieve unparalleled visibility into their LangChain applications, ensuring optimized performance and quicker debugging. This powerful combination simplifies the complex task of monitoring AI-driven systems, enabling you to focus on what truly matters—delivering value to your users. Throughout this blog,we've covered the following essential steps and concepts:

• How to manually instrument Langchain with OpenTelemetry

• How to properly initialize OpenTelemetry and add a custom span

• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

• How to view and analyze traces in Elastic Observability APM

These steps provide a clear and actionable guide for developers looking to integrate robust tracing capabilities into their LangChain applications.

We hope this guide makes understanding and implementing OpenTelemetry tracing for LangChain simple, ensuring seamless integration with Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

• Python: Auto-instrumentation, Manual instrumentation

• Java: Auto-instrumentation, Manual instrumentation 

• Node.js: Auto-instrumentation, Manual instrumentation

• .NET: Auto-instrumentation, Manual instrumentation

• Elastic APM - Langtrace AI Docs

There are several options to trace for LangChain. One is Langsmith, ideal for detailed tracing and a complete breakdown of requests to large language models (LLMs). However, it is specific to Langchain. OpenTelemetry (OTel) is now broadly accepted as the industry standard for tracing. As one of the major Cloud Native Computing Foundation (CNCF) projects, with as many commits as Kubernetes, it is gaining support from major ISVs and cloud providers delivering support for the framework. 

Hence, many LangChain-based applications will have multiple components beyond just LLM interactions. Using OpenTelemetry with LangChain is essential. OpenLLMetry is an available option for tracing Langchain apps in addition to Langsmith.

This blog will show how you can get LangChain tracing into Elastic using the OpenLLMetry library opentelemetry-instrumentation-langchain.

Pre-requisites:

• An Elastic Cloud account — sign up now, and become familiar with Elastic’s OpenTelemetry configuration

• Have a LangChain app to instrument

• Be familiar with using OpenTelemetry’s Python SDK 

• An account on your favorite LLM, with API keys

Overview

In highlighting tracing I created a simple LangChain app that does the following:

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a LangChain.

3. Chain tools are set to use the search with Tavily 

4. The LLM uses the output which returns the relevant information to the user.

As you can see Elastic Observability’s APM recognizes the LangChain App, and also shows the full trace (done with manual instrumentation):

As the above image shows:

1. The user makes a query
2. Azure OpenAI is called, but it uses a tool (Tavily) to obtain some results
3. Azure OpenAI reviews and returns a summary to the end user

The code was manually instrumented, but auto-instrument can also be used.

OpenTelemetry Configuration

In using OpenTelemetry, we need to configure the SDK to generate traces and configure Elastic’s endpoint and authorization. Instructions can be found in OpenTelemetry Auto-Instrumentation setup documentation.

OpenTelemetry Environment variables:

OpenTelemetry Environment variables for Elastic can be set as follows in linux (or in the code).

OTEL_EXPORTER_OTLP_ENDPOINT=12345.apm.us-west-2.aws.cloud.es.io:443
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer%20ZZZZZZZ"
OTEL_RESOURCE_ATTRIBUTES="service.name=langchainChat,service.version=1.0,deployment.environment=production"

As you can see OTEL_EXPORTER_OTLP_ENDPOINT is set to Elastic, and the corresponding authorization header is also provided. These can be easily obtained from Elastic’s APM configuration screen under OpenTelemetry

Note: No agent is needed, we simply send the OTLP trace messages directly to Elastic’s APM server. 

OpenLLMetry Library:

OpenTelemetry's auto-instrumentation can be extended to trace other frameworks via instrumentation packages.

First, you must install the following package: 

pip install opentelemetry-instrumentation-langchain

This library was developed by OpenLLMetry. 

Then you will need to add the following to the code.

from opentelemetry.instrumentation.langchain import LangchainInstrumentor
LangchainInstrumentor().instrument()

Instrumentation

Once the libraries are added, and the environment variables are set, you can use auto-instrumentation With auto-instrumentation, the following:

opentelemetry-instrument python tavilyAzureApp.py

The OpenLLMetry library does pull out the flow correctly with minimal manual instrumentation except for adding the OpenLLMetry library.

1. Takes customer input on the command line. (Queries)

2. Sends these to the Azure OpenAI LLM via a Lang chain.

3. Chain tools are set to use the search with Tavily 

4. The LLM uses the output which returns the relevant information to the user.

Manual-instrumentation

If you want to get more details out of the application, you will need to manually instrument. To get more traces follow my Python instrumentation guide. This guide will walk you through setting up the necessary OpenTelemetry bits, Additionally, you can also look at the documentation in OTel for instrumenting in Python.

Note that the env variables OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_ENDPOINT are set as noted in the section above. You can also set up the OTEL_RESOURCE_ATTRIBUTES.

Once you follow the steps in either guide and initiate the tracer, you will have to essentially just add the span where you want to get more details. In the example below, only one line of code is added for span initialization.

Look at the placement of with tracer.start_as_current_span("getting user query") as span: below

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer("newsQuery")

async def chat_interface():
    print("Welcome to the AI Chat Interface!")
    print("Type 'quit' to exit the chat.")

    with tracer.start_as_current_span("getting user query") as span:
        while True:
            user_input = input("\nYou: ").strip()

            if user_input.lower() == 'quit':
                print("Thank you for chatting. Goodbye!")
                break

            print("AI: Thinking...")
            try:
                result = await chain.ainvoke({"query": user_input})
                print(f"AI: {result.content}")
            except Exception as e:
                print(f"An error occurred: {str(e)}")


if __name__ == "__main__":
    asyncio.run(chat_interface())

As you can see, with manual instrumentation, we get the following trace:

Which calls out when we enter our query function. async def chat_interface()

Conclusion

In this blog, we discussed the following:

• How to manually instrument LangChain with OpenTelemetry

• How to properly initialize OpenTelemetry and add a custom span

• How to easily set the OTLP ENDPOINT and OTLP HEADERS with Elastic without the need for a collector

• See traces in Elastic Observability APM

Hopefully, this provides an easy-to-understand walk-through of instrumenting LangChain with OpenTelemetry and how easy it is to send traces into Elastic.

Additional resources for OpenTelemetry with Elastic:

• Independence with OpenTelemetry on Elastic

• Modern observability and security on Kubernetes with Elastic and OpenTelemetry

• 3 models for logging with OpenTelemetry and Elastic

• Adding free and open Elastic APM as part of your Elastic Observability deployment

• Monitor OpenAI API and GPT models with OpenTelemetry and Elastic

• Futureproof your observability platform with OpenTelemetry and Elastic

• Instrumentation resources:

• Python: Auto-instrumentation, Manual instrumentation

• Java: Auto-instrumentation, Manual instrumentation 

• Node.js: Auto-instrumentation, Manual instrumentation

• .NET: Auto-instrumentation, Manual instrumentation

Also log into cloud.elastic.co to try out Elastic with a free trial.

Our data transformation pipelines run daily and process the data that feed our internal dashboards, reports, analyses, and Machine Learning (ML) models.

There have been incidents in the past when the pipelines have failed, the source tables contained wrong data or we have introduced a change into our SQL code that has caused data quality issues, and we only realized once we saw it in a weekly report that was showing an anomalous number of records. That’s why we have built a monitoring system that proactively alerts us about these types of incidents as soon as they happen and helps us with visualizations and analyses to understand their root cause, saving us several hours or days of manual investigations.

We have leveraged our own Observability Solution to help solve this challenge, monitoring the entire lifecycle of our dbt implementation. This setup enables us to track the behavior of our models and conduct data quality testing on the final tables. We export dbt process logs from run jobs and tests into Elasticsearch and utilize Kibana to create dashboards, set up alerts, and configure Machine Learning jobs to monitor and assess issues.

The following diagram shows our complete architecture. In a follow-up article, we’ll also cover how we observe our python data processing and ML model processes using OTEL and Elastic - stay tuned.

Why monitor dbt pipelines with Elastic?

With every invocation, dbt generates and saves one or more JSON files called artifacts containing log data on the invocation results. dbt run and dbt test invocation logs are stored in the file run_results.json, as per the dbt documentation:

This file contains information about a completed invocation of dbt, including timing and status info for each node (model, test, etc) that was executed. In aggregate, many run_results.json can be combined to calculate average model runtime, test failure rates, the number of record changes captured by snapshots, etc.

Monitoring dbt run invocation logs can help solve several issues, including tracking and alerting about table volumes, detecting excessive slot time from resource-intensive models, identifying cost spikes due to slot time or volume, and pinpointing slow execution times that may indicate scheduling issues. This system was crucial when we merged a PR with a change in our code that had an issue, producing a sudden drop in the number of daily rows in upstream Table A. By ingesting the dbt run logs into Elastic, our anomaly detection job quickly identified anomalies in the daily row counts for Table A and its downstream tables, B, C, and D. The Data Analytics team received an alert notification about the issue, allowing us to promptly troubleshoot, fix and backfill the tables before it affected the weekly dashboards and downstream ML models.

Monitoring dbt test invocation logs can also address several issues, such as identifying duplicates in tables, detecting unnoticed alterations in allowed values for specific fields through validation of all enum fields, and resolving various other data processing and quality concerns. With dashboards and alerts on data quality tests, we proactively identify issues like duplicate keys, unexpected category values, and increased nulls, ensuring data integrity. In our team, we had an issue where a change in one of our raw lookup tables produced duplicated rows in our user table, doubling the number of users reported. By ingesting the dbt test logs into Elastic, our rules detected that some duplicate tests had failed. The team received an alert notification about the issue, allowing us to troubleshoot it right away by finding the upstream table that was the root cause. These duplicates meant that downstream tables had to process 2x the amount of data, creating a spike in the bytes processed and slot time. The anomaly detection and alerts on the dbt run logs also helped us spot these spikes for individual tables and allowed us to quantify the impact on our billing.

Processing our dbt logs with Elastic and Kibana allows us to obtain real-time insights, helps us quickly troubleshoot potential issues, and keeps our data transformation processes running smoothly. We set up anomaly detection jobs and alerts in Kibana to monitor the number of rows processed by dbt, the slot time, and the results of the tests. This lets us catch real-time incidents, and by promptly identifying and fixing these issues, Elastic makes our data pipeline more resilient and our models more cost-effective, helping us stay on top of cost spikes or data quality issues.

We can also correlate this information with other events ingested into Elastic, for example using the Elastic Github connector, we can correlate data quality test failures or other anomalies with code changes to find the root cause of the commit or PR that caused the issues. By ingesting application logs into Elastic, we can also analyze if these issues in our pipelines have affected downstream applications, increasing latency, throughput or error rates using APM. Ingesting billing, revenue data or web traffic, we could also see the impact in business metrics.

How to export dbt invocation logs to Elasticsearch

We use the Python Elasticsearch client to send the dbt invocation logs to Elastic after we run our dbt run and dbt test processes daily in production. The setup just requires you to install the Elasticsearch Python client and obtain your Elastic Cloud ID (go to https://cloud.elastic.co/deployments/, select your deployment and find the Cloud ID) and Elastic Cloud API Key (following this guide)

This python helper function will index the results from your run_results.json file to the specified index. You just need to export the variables to the environment:

• RESULTS_FILE: path to your run_results.json file
• DBT_RUN_LOGS_INDEX: the name you want to give to dbt run logs index in Elastic, e.g. dbt_run_logs
• DBT_TEST_LOGS_INDEX: the name you want to give to the dbt test logs index in Elastic, e.g. dbt_test_logs
• ES_CLUSTER_CLOUD_ID
• ES_CLUSTER_API_KEY

Then call the function log_dbt_es from your python code or save this code as a python script and run it after executing your dbt run or dbt test commands:

from elasticsearch import Elasticsearch, helpers
import os
import sys
import json

def log_dbt_es():
   RESULTS_FILE = os.environ["RESULTS_FILE"]
   DBT_RUN_LOGS_INDEX = os.environ["DBT_RUN_LOGS_INDEX"]
   DBT_TEST_LOGS_INDEX = os.environ["DBT_TEST_LOGS_INDEX"]
   es_cluster_cloud_id = os.environ["ES_CLUSTER_CLOUD_ID"]
   es_cluster_api_key = os.environ["ES_CLUSTER_API_KEY"]


   es_client = Elasticsearch(
       cloud_id=es_cluster_cloud_id,
       api_key=es_cluster_api_key,
       request_timeout=120,
   )


   if not os.path.exists(RESULTS_FILE):
       print(f"ERROR: {RESULTS_FILE} No dbt run results found.")
       sys.exit(1)


   with open(RESULTS_FILE, "r") as json_file:
       results = json.load(json_file)
       timestamp = results["metadata"]["generated_at"]
       metadata = results["metadata"]
       elapsed_time = results["elapsed_time"]
       args = results["args"]
       docs = []
       for result in results["results"]:
           if result["unique_id"].split(".")[0] == "test":
               result["_index"] = DBT_TEST_LOGS_INDEX
           else:
               result["_index"] = DBT_RUN_LOGS_INDEX
           result["@timestamp"] = timestamp
           result["metadata"] = metadata
           result["elapsed_time"] = elapsed_time
           result["args"] = args
           docs.append(result)
        = helpers.bulk(es_client, docs)
   return "Done"

# Call the function
log_dbt_es()

If you want to add/remove any other fields from run_results.json, you can modify the above function to do it.

Once the results are indexed, you can use Kibana to create Data Views for both indexes and start exploring them in Discover.

Go to Discover, click on the data view selector on the top left and “Create a data view”.

Now you can create a data view with your preferred name. Do this for both dbt run (DBT_RUN_LOGS_INDEX in your code) and dbt test (DBT_TEST_LOGS_INDEX in your code) indices:

Going back to Discover, you’ll be able to select the Data Views and explore the data.

dbt run alerts, dashboards and ML jobs

The invocation of dbt run executes compiled SQL model files against the current database. dbt run invocation logs contain the following fields:

• unique_id: Unique model identifier
• execution_time: Total time spent executing this model run

The logs also contain the following metrics about the job execution from the adapter:

• adapter_response.bytes_processed
• adapter_response.bytes_billed
• adapter_response.slot_ms
• adapter_response.rows_affected

We have used Kibana to set up Anomaly Detection jobs on the above-mentioned metrics. You can configure a multi-metric job split by unique_id to be alerted when the sum of rows affected, slot time consumed, or bytes billed is anomalous per table. You can track one job per metric. If you have built a dashboard of the metrics per table, you can use this shortcut to create the Anomaly Detection job directly from the visualization. After the jobs are created and are running on incoming data, you can view the jobs and add them to a dashboard using the three dots button in the anomaly timeline:

We have used the ML job to set up alerts that send us emails/slack messages when anomalies are detected. Alerts can be created directly from the Jobs (Machine Learning > Anomaly Detection Jobs) page, by clicking on the three dots at the end of the ML job row:

We also use Kibana dashboards to visualize the anomaly detection job results and related metrics per table, to identify which tables consume most of our resources, to have visibility on their temporal evolution, and to measure aggregated metrics that can help us understand month over month changes.

dbt test alerts and dashboards

You may already be familiar with tests in dbt, but if you’re not, dbt data tests are assertions you make about your models. Using the command dbt test, dbt will tell you if each test in your project passes or fails. Here is an example of how to set them up. In our team, we use out-of-the-box dbt tests (unique, not_null, accepted_values, and relationships) and the packages dbt_utils and dbt_expectations for some extra tests. When the command dbt test is run, it generates logs that are stored in run_results.json.

dbt test logs contain the following fields:

• unique_id: Unique test identifier, tests contain the “test” prefix in their unique identifier
• status: result of the test, pass or fail
• execution_time: Total time spent executing this test
• failures: will be 0 if the test passes and 1 if the test fails
• message: If the test fails, reason why it failed

The logs also contain the metrics about the job execution from the adapter.

We have set up alerts on document count (see guide) that will send us an email / slack message when there are any failed tests. The rule for the alerts is set up on the dbt test Data View that we have created before, the query filtering on status:fail to obtain the logs for the tests that have failed, and the rule condition is document count bigger than 0. Whenever there is a failure in any test in production, we get an alert with links to the alert details and dashboards to be able to troubleshoot them:

We have also built a dashboard to visualize the tests run, tests failed, and their execution time and slot time to have a historical view of the test run:

Finding Root Causes with the AI Assistant

The most effective way for us to analyze these multiple sources of information is using the AI Assistant to help us troubleshoot the incidents. In our case, we got an alert about a test failure, and we used the AI Assistant to give us context on what happened. Then we asked if there were any downstream consequences, and the AI Assistant interpreted the results of the Anomaly Detection job, which indicated a spike in slot time for one of our downstream tables and the increase of the slot time vs. the baseline. Then, we asked for the root cause, and the AI Assistant was able to find and provide us a link to a PR from our Github changelog that matched the start of the incident and was the most probable cause.

Conclusion

As a Data Analytics team, we are responsible for guaranteeing that the tables, charts, models, reports, and dashboards we provide to stakeholders are accurate and contain the right sources of information. As teams grow, the number of models we own becomes larger and more interconnected, and it isn’t easy to guarantee that everything is running smoothly and providing accurate results. Having a monitoring system that proactively alerts us on cost spikes, anomalies in row counts, or data quality test failures is like having a trusted companion that will alert you in advance if something goes wrong and help you get to the root cause of the issue.

dbt invocation logs are a crucial source of information about the status of our data pipelines, and Elastic is the perfect tool to extract the maximum potential out of them. Use this blog post as a starting point for utilizing your dbt logs to help your team achieve greater reliability and peace of mind, allowing them to focus on more strategic tasks rather than worrying about potential data issues.

In managing NGINX Elastic provides several capabilities:

1. Easy ingest, parsing, and out-of-the-box dashboards. Check out the simple how-to in our docs. Based on logs, these dashboards show several items over time, response codes, errors, top pages, data volume, browsers used, active connections, drop rates, and much more.

2. Out-of-the-box ML-based anomaly detection jobs for your NGINX logs. These jobs help pinpoint anomalies against request rates, IP address request rates, URL access, status codes, and visitor rate anomalies.

3. ES|QL which helps work through logs and build out charts during analysis.

4. Elastic’s GenAI Assistant provides a simple natural language interface that helps analyze all the logs and can pull out issues from ML jobs and even create dashboards. The Elastic AI Assistant also automatically uses ES|QL.

5. NGINX SLOs - Finally Elastic provides the ability to define and monitor SLOs for your NGINX logs. While most SLOs are metrics-based, Elastic allows you to create logs-based SLOs. We detailed this in a previous blog.

NGINX logs are another example of why logs are great.  Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting and NGINX is usually the starting point for most analyses. 

In today’s blog, we’ll cover how the out-of-the-box ML-based anomaly detection jobs can help RCA, and how Elastic’s GenAI Assistant helps easily work through logs to pinpoint issues in minutes. 

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Bring up an NGINX server on a host. OR run an application with NGINX as a front end and drive traffic.

• Install the NGINX integration and assets and review the dashboards as noted in the docs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

In our scenario, we use data from 3 months from our Elastic environment to help highlight the features. Hence you might need to run your application with traffic for a specific time frame to follow along.

Analyzing the issues with AI Assistant

As detailed in a previous blog, you can get alerted on issues via SLO monitoring against NGINX logs. Let’s assume you have an SLO based on status codes as we outlined in the previous blog. You can immediately analyze the issue via the AI Assistant. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo)

AI Assistant analysis:

• Using lens graph all http response status codes < 400 and > =400 from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer - We wanted to simply understand the amount of requests resulting in status code >= 400 and graph the results. We see that 15% of the requests were not successful, hence an SLO alert being triggered.

• Which ip address (field source.adress) has the highest number of http.response.status.code >= 400 from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer  - We were curious is there was a specific IP address not having successful requests. 72.57.0.53, with a count of 25,227 occurrences is daily high but not the ensure 2 failed requests.

• What country (source.geo.country_iso_code) is source.address=72.57.0.53 coming from. Use filebeat-nginx-elasticco-anon-2017. - Again we were curious if this came from a specific country. And the IP address 72.57.0.53 is coming from the country with the ISO code IN, which corresponds to India. Nothing out of the ordinary.

• Did source.address=72.57.0.53 have any (http.response.status.code < 400) from filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer -  Oddly the IP address in question only had 4000+ successful responses. Meaning its not malicious, and points to something else.

• What are the different status codes (http.response.status.code>=400), from source.address=72.57.0.53. Use filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer. Provide counts for each status code - We are curious whether or not we see any 502, which there were none, but most of the failures were 404. 

• What are the different status codes (http.response.status.code>=400). Use filebeat-nginx-elasticco-anon-2017. http.response.status.code is not an integer. Provide counts for each status code - Regardless of a specific address, what is the largest number of status code occurrences > 400. This also points to 404. 

• What does a high 404 count from a specific IP address mean from NGINX logs? - Asking this question, we need to understand the potential causes of this from our application. From the answers, we can rule out security probing and web scraping, as we validated that a specific address 72.57.0.53 has a low non-success request status code. It also rules out User error. Hence this points potentially to Broken Links or Missing Resources.

Watch the flow:

Potential issue:

It seems that we potentially have an issue with the backend serving specific answers or having issues with resources (database, or broken links). This is cursing the higher-than-normal non-successful status codes>=400.

Key highlights from AI Assistant:

As you watched this video you will notice a few things:

1. We analyzed millions of logs in a matter of minutes using a set of simple natural language queries. 

2. We didn’t need to know any special query language. The AI Assistant used Elastic’s ES|QL but can similarly use KQL also. 

3. The AI Assistant easily builds out graphs

4. The AI Assistant is accessing and using internal information stored in Elastic’s indices. Vs a simple “google foo” based AI Assistant. This is enabled through RAG, and the AI Assistant can also bring up known issues in github, runbooks, and other useful internal information.

Check out the following blog on how the AI Assistant uses RAG to retrieve internal information. Specifically using github and runbooks.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of NGINX log management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.When using NGINX, there are several out-of-the-box anomaly detection jobs. These work specifically on NGINX access logs.

• Low_request_rate_nginx - Detect low request rates

• Source_ip_request_rate_nginx - Detect unusual source IPs - high request rates

• Source_ip_url_count_nginx - Detect unusual source IPs - high distinct count of URLs

• Status_code_rate_nginx - Detect unusual status code rates

• Visitor_rate_nginx - Detect unusual visitor rates

Being right out of the box, lets look at the job - Status_code_rate_nginx, which is related to our previous analysis.

With a few simple clicks we immediately get an analysis showing a specific IP address - 72.57.0.53, having higher than normal non-successful requests. Oddly we also found this is using the AI Assistant.

We can take this further with conversations with the AI Assistant, look at the logs, and/or even look at the other ML anomaly jobs.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze NGINX logs without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). 

Check out other resources on NGINX logs:

Out-of-the-box anomaly detection jobs for NGINX

Using the NGINX integration to ingest and analyze NGINX Logs

NGINX Logs based SLOs in Elastic

Using GitHub issues, runbooks, and other internal information for RCAs with Elastic’s RAG based AI Assistant

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

1. Where traffic is coming in from and going out to from the deployment, and within the deployment. This helps identify unusual or unauthorized communications

2. Traffic volumes detecting spikes or drops which could indicate service issues in production or an increase in customer traffic

3. Latency and Performance bottlenecks - with VPC Flow logs, you can look at latency for a flow (in and outflows), and understand patterns

4. Accepted and rejected traffic helps determine where potential security threats and misconfigurations lie. 

AWS VPC Logs is a great example of how logs are great. Logging is an important part of Observability, for which we generally think of metrics and tracing. However, the amount of logs an application and the underlying infrastructure output can be significantly daunting with VPC Logs. However, it also provides a significant amount of insight.

Before we proceed, it is important to understand what Elastic provides in managing AWS and VPC Flow logs:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

In today’s blog, we’ll cover how Elastics’ other features can support analyzing and RCA for potential VPC flow logs even more easily. Specifically, we will focus on managing the number of rejects, as this helps ensure there weren’t any unauthorized or unusual activities:

1. Set up an easy-to-use SLO (newly released) to detect when things are potentially degrading

2. Create an ML job to analyze different fields of the VPC Flow log

3. Using our newly released RAG-based AI Assistant to help analyze the logs without needing to know Elastic’s query language nor how to even graph on Elastic

4. ES|QL will help understand and analyze add latency for patterns.

In subsequent blogs, we will use AI Assistant and ESQL to show how to get other insights beyond just REJECT/ACCEPT from VPC Flow log.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up this demonstration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).

• Follow the steps in the following blog to get AWS’s three-tier app installed instructed in git, and bring in the AWS VPC Flow logs.

• Ensure you have an ML node configured in your Elastic stack

• To use the AI Assistant you will need a trial or upgrade to Platinum.

SLO with VPC Flow Logs

Elastic’s SLO capability is based directly on the Google SRE Handbook. All the definitions and semantics are utilized as described in Google’s SRE handbook. Hence users can perform the following on SLOs in Elastic:

• Define an SLO on Logs not just metrics - Users can use KQL (log-based query), service availability, service latency, custom metric, histogram metric, or a timeslice metric.
• Define SLO, SLI, Error budget and burn rates. Users can also use occurrence versus time slice-based budgeting. 
• Manage, with dashboards, all the SLOs in a singular location.
• Trigger alerts from the defined SLO, whether the SLI is off, the burn rate is used up, or the error rate is X.

Setting up an SLO for VPC is easy. You simply create a query you want to trigger off. In our case, we look for all the good events where aws.vpcflow.action=ACCEPT and we define the target at 85%. 

As the following example shows, over the last 7 days, we have exceeded our budget by 43%. Additionally, we have not complied for the last 7 days.

Analyzing the SLO with AI Assistant

Now that we see that there is an issue with the VPC Flows, we immediately work with the AI Assistant to start analyzing the SLO. Because it's a chat interface we simply open the AI Assistant and work through some simple analysis: (See Animated GIF for a demo below)

AI Assistant analysis:

• what were the top 3 source.address that had aws.vpcflow.action=REJECT over the last 7 days, which is causing this SLO issue? - We wanted to simply see what could be causing the loss in error budget. Were there any particular source.addresses causing a heavy reject rate.

• The answer: A table with the highest count = 42670 and source.address = 79.110.62.185

• There is one singular source.address that is causing the loss in SLO. 

• What is the largest number of  aws.vpcflow.action=REJECT in a 30 min time frame for the last 3 days where the source.address=79.110.62.185? - After understanding that a specific source.address is causing the loss in SLO, we want to understand the averages. 

• **The answer: ** "The largest number of aws.vpcflow.action=REJECT in a 30-minute time frame for the last 3 days where the source.address is 79.110.62.185 is 229. This occurred on 2024-06-01T04:00:00.000Z.”

• It means there must be a low REJECT rate but fairly consistent vs spiky over the last 7 days.

• for the logs with source.address="79.110.62.185" was there any country code of source.geo.country_iso_code field present. If yes what is the value - Given the last question showed a low REJECT rate, it only means that this was fairly consistent vs spiky over the last 7 days.

• The answer: Yes, there is a country code present in the source.geo.country_iso_code field for logs with source.address="79.110.62.185". The value is BG (Bulgaria).

• Is there a specific destination.address where source.address=79.110.62.185 is getting a aws.vpcflow.action=REJECT. Give me both the destination.address and the number of REJECTs for that destination.address?

• The answer: destination.address of 10.0.0.27 is giving a reject number of 53433 in this time frame.

• Graph the number of REJECT vs ACCEPT for source.address="79.110.62.185" over the last 7 days. The graph is on a daily basis in a singular graph - We asked this question to see what the comparison is between ACCEPT and REJECT. 

• The answer: See the animated GIF to see that the generated graph is fairly stable

• Were there any source.address that had a spike, high reject rate in. a 30min period over the 30 days? - We wanted to see if there was any other spike 

• The answer - Yes, there was a source.address that had a spike in high reject rates in a 30-minute period over the last 30 days. source.address: 185.244.212.67, Reject Count: 8975, Time Period: 2024-05-22T03:00:00.000Z

Watch the flow

Potential issue:

he server handling requests from source 79.110.62.185 is potentially having an issue.

Again using logs, we essentially asked the AI Assistant to give the eni ids where the internal ip address was 10.0.0.27

From our AWS console, we know that this is the webserver. Further analysis in Elastic, and with the developers we realized there is a new version that was installed recently causing a problem with connections.

Locating anomalies with ML

While using the AI Assistant is great for analyzing information, another important aspect of VPC flow management is to ensure you can manage log spikes and anomalies. Elastic has a machine learning platform that allows you to develop jobs to analyze specific metrics or multiple metrics to look for anomalies.

VPC Flow logs come with a large amount of information. The full set of fields is listed in AWS docs. We will use a specific subset to help detect anomalies.

We were setting up anomalies for aws.vpcflow.action=REJECT, which requires us to use multimetric anomaly detection in Elastic.

The config we used utilizes:

Detectors:

• destination.address

• destination.port

Influencers:

• source.address

• aws.vpcflow.action

• destination.geo.region_iso_code

The way we set this up will help us understand if there is a large spike in REJECT/ACCEPT against destination.address values from a specific source.address and/or destination.geo.region_iso_code location.

The job once run reveals something interesting:

Notice that source.address 185.244.212.67 has had a high REJECT rate in the last 30 days. 

Notice where we found this before? In the AI Assistant!!!!!

While we can run the AI Assistant and find this sort of anomaly, the ML job can be setup to run continuously and alert us on such spikes. This will help us understand if there are any issues with the webserver like we found above or even potential security attacks.

Conclusion:

You’ve now seen how easily Elastic’s RAG-based AI Assistant can help analyze VPC Flows without even the need to know query syntax, understand where the data is, and understand even the fields. Additionally, you’ve also seen how we can alert you when a potential issue or degradation in service (SLO). Check out our other blogs on AWS VPC Flow analysis in Elastic:

1. A full set of integrations to manage VPC Flows and the entire end-to-end deployment on AWS. 

2. Elastic has a simple-to-use AWS Firehose integration. 

3. Elastic’s tools such as Discover, spike analysis,  and anomaly detection help provide you with better insights and analysis.

4. And a set of simple Out-of-the-box dashboards

Try it out

Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. Not taking advantage of Elastic on the cloud? Start a free trial.

All of this is also possible in your environment. Learn how to get started today.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Enhanced troubleshooting for SREs

Site reliability engineers (SRE) in large organizations often face challenges in locating necessary information for troubleshooting alerts, monitoring systems, or deriving insights due to scattered and potentially outdated resources. This issue is particularly significant for less experienced SREs who may require assistance even with the presence of a runbook. Recurring incidents pose another problem, as the on-call individual may lack knowledge about previous resolutions and subsequent steps. Mature SRE teams often invest considerable time in system improvements to minimize "fire-fighting," utilizing extensive automation and documentation to support on-call personnel.

Elastic® addresses these challenges by combining generative AI models with relevant search results from your internal data using RAG. The Observability AI Assistant's internal Knowledge Base, powered by our semantic search retrieval model ELSER, can recall information at any point during a conversation, providing RAG responses based on internal knowledge.

This Knowledge Base can be enriched with your organization's information, such as runbooks, GitHub issues, internal documentation, and Slack messages, allowing the AI Assistant to provide specific assistance. The Assistant can also document and store specific information from an ongoing conversation with an SRE while troubleshooting issues, effectively creating runbooks for future reference. Furthermore, the Assistant can generate summaries of incidents, system status, runbooks, post-mortems, or public announcements.

This ability to retrieve, summarize, and present contextually relevant information is a game-changer for SRE teams, transforming the work from chasing documents and data to an intuitive, contextually sensitive user experience.The Knowledge Base (see requirements) serves as a central repository of Observability knowledge, breaking documentation silos and integrating tribal knowledge, making this information accessible to SREs enhanced with the power of LLMs.

Your LLM provider may collect query telemetry when using the AI Assistant. If your data is confidential or has sensitive details, we recommend you verify the data treatment policy of the LLM connector you provided to the AI Assistant.

In this blog post, we will cover different ways to enrich your Knowledge Base (KB) with internal information. We will focus on a specific alert, indicating that there was an increase in logs with “502 Bad Gateway” errors that has surpassed the alert’s threshold.

How to troubleshoot an alert with the Knowledge Base

Before the KB has been enriched with internal information, when the SRE asks the AI Assistant about how to troubleshoot an alert, the response from the LLM will be based on the data it learned during training; however, the LLM is not able to answer questions related to private, recent, or emerging knowledge. In this case, when asking for the steps to troubleshoot the alert, the response will be based on generic information.

However, once the KB has been enriched with your runbooks, when your team receives a new alert on “502 Bad Gateway” Errors, they can use AI Assistant to access the internal knowledge to troubleshoot it, using semantic search to find the appropriate runbook in the Knowledge Base.

In this blog, we will cover different ways to add internal information on how to troubleshoot an alert to the Knowledge Base:

1. Ask the assistant to remember the content of an existing runbook.

2. Ask the Assistant to summarize and store in the Knowledge Base the steps taken during a conversation and store it as a runbook.

3. Import your runbooks from GitHub or another external source to the Knowledge Base using our Connector and APIs.

After the runbooks have been added to the KB, the AI Assistant is now able to recall the internal and specific information in the runbooks. By leveraging the retrieved information, the LLM could provide more accurate and relevant recommendations for troubleshooting the alert. This could include suggesting potential causes for the alert, steps to resolve the issue, preventative measures for future incidents, or asking the assistant to help execute the steps mentioned in the runbook using functions. With more accurate and relevant information at hand, the SRE could potentially resolve the alert more quickly, reducing downtime and improving service reliability.

Your Knowledge Base documents will be stored in the indices .kibana-observability-ai-assistant-kb-*. Have in mind that LLMs have restrictions on the amount of information the model can read and write at once, called token limit. Imagine you're reading a book, but you can only remember a certain number of words at a time. Once you've reached that limit, you start to forget the earlier words you've read. That's similar to how a token limit works in an LLM.

To keep runbooks within the token limit for Retrieval Augmented Generation (RAG) models, ensure the information is concise and relevant. Use bullet points for clarity, avoid repetition, and use links for additional information. Regularly review and update the runbooks to remove outdated or irrelevant information. The goal is to provide clear, concise, and effective troubleshooting information without compromising the quality due to token limit constraints. LLMs are great for summarization, so you could ask the AI Assistant to help you make the runbooks more concise.

Ask the assistant to remember the content of an existing runbook

The easiest way to store a runbook into the Knowledge Base is to just ask the AI Assistant to do it! Open a new conversation and ask “Can you store this runbook in the KB for future reference?” followed by pasting the content of the runbook in plain text.

The AI Assistant will then store it in the Knowledge Base for you automatically, as simple as that.

Ask the Assistant to summarize and store the steps taken during a conversation in the Knowledge Base

You can also ask the AI Assistant to remember something while having a conversation — for example, after you have troubleshooted an alert using the AI Assistant, you could ask to "remember how to troubleshoot this alert for next time." The AI Assistant will create a summary of the steps taken to troubleshoot the alert and add it to the Knowledge Base, effectively creating runbooks for future reference. Next time you are faced with a similar situation, the AI Assistant will recall this information and use it to assist you.

In the following demo, the user asks the Assistant to remember the steps that have been followed to troubleshoot the root cause of an alert, and also to ping the Slack channel when this happens again. In a later conversation with the Assistant, the user asks what can be done about a similar problem, and the AI Assistant is able to remember the steps and also reminds the user to ping the Slack channel.

After receiving the alert, you can open the AI Assistant chat and test troubleshooting the alert. After investigating an alert, ask the AI Assistant to summarize the analysis and the steps taken to root cause. To remember them for the next time, we have a similar alert and add extra instruction like to warn the Slack channel.

The Assistant will use the built-in functions to summarize the steps and store them into your Knowledge Base, so they can be recalled in future conversations.

Open a new conversation, and ask what are the steps to take when troubleshooting a similar alert to the one we just investigated. The Assistant will be able to recall the information stored in the KB that is related to the specific alert, using semantic search based on ELSER, and provide a summary of the steps taken to troubleshoot it, including the last indication of informing the Slack channel.

Import your runbooks stored in GitHub to the Knowledge Base using APIs or our GitHub Connector

You can also add proprietary data into the Knowledge Base programmatically by ingesting it (e.g., GitHub Issues, Markdown files, Jira tickets, text files) into Elastic.

If your organization has created runbooks that are stored in Markdown documents in GitHub, follow the steps in the next section of this blog post to index the runbook documents into your Knowledge Base.

The steps to ingest documents into the Knowledge Base are the following:

Ingest your organization’s knowledge into Elasticsearch

Option 1: Use the Elastic web crawler . Use the web crawler to programmatically discover, extract, and index searchable content from websites and knowledge bases. When you ingest data with the web crawler, a search-optimized Elasticsearch® index is created to hold and sync webpage content.

Option 2: Use Elasticsearch's Index API . Watch tutorials that demonstrate how you can use the Elasticsearch language clients to ingest data from an application.

Option 3: Build your own connector. Follow the steps described in this blog: How to create customized connectors for Elasticsearch.

Option 4: Use Elasticsearch Workplace Search connectors . For example, the GitHub connector can automatically capture, sync, and index issues, Markdown files, pull requests, and repos.

• Follow the steps to configure the GitHub Connector in GitHub to create an OAuth App from the GitHub platform.

• Now you can connect a GitHub instance to your organization. Head to your organization’s Search > Workplace Search administrative dashboard, and locate the Sources tab.

• Select GitHub (or GitHub Enterprise) in the Configured Sources list, and follow the GitHub authentication flow as presented. Upon the successful authentication flow, you will be redirected to Workplace Search and will be prompted to select the Organization you would like to synchronize.

• After configuring the connector and selecting the organization, the content should be synchronized and you will be able to see it in Sources. If you don’t need to index all the available content, you can specify the indexing rules via the API. This will help shorten indexing times and limit the size of the index. See Customizing indexing.

• The source has created an index in Elastic with the content (Issues, Markdown Files…) from your organization. You can find the index name by navigating to Stack Management > Index Management , activating the Include hidden Indices button on the right, and searching for “GitHub.”

• You can explore the documents you have indexed by creating a Data View and exploring it in Discover. Go to Stack Management > Kibana > Data Views > Create data view and introduce the data view Name, Index pattern (make sure you activate “Allow hidden and system indices” in advanced options), and Timestamp field:

• You can now explore the documents in Discover using the data view:

Reindex your internal runbooks into the AI Assistant’s Knowledge Base Index, using it's semantic search pipeline

Your Knowledge Base documents are stored in the indices .kibana-observability-ai-assistant-kb-*. To add your internal runbooks imported from GitHub to the KB, you just need to reindex the documents from the index you created in the previous step to the KB’s index. To add the semantic search capabilities to the documents in the KB, the reindex should also use the ELSER pipeline preconfigured for the KB, .kibana-observability-ai-assistant-kb-ingest-pipeline.

By creating a Data View with the KB index, you can explore the content in Discover.

You execute the query below in Management > Dev Tools , making sure to replace the following, both on “_source” and “inline”:

• InternalDocsIndex : name of the index where your internal docs are stored
• text_field : name of the field with the text of your internal docs
• timestamp : name of the field of the timestamp in your internal docs
• public : (true or false) if true, makes a document available to all users in the defined Kibana Space (if is defined) or in all spaces (if is not defined); if false, document will be restricted to the user indicated in
• (optional) space : if defined, restricts the internal document to be available in a specific Kibana Space
• (optional) user.name : if defined, restricts the internal document to be available for a specific user
• (optional) "query" filter to index only certain docs (see below)

POST _reindex
{
    "source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ]
    },
    "dest": {
        "index": ".kibana-observability-ai-assistant-kb-000001",
        "pipeline": ".kibana-observability-ai-assistant-kb-ingest-pipeline"
    },
    "script": {
        "inline": "ctx._source.text=ctx._source.remove(\"<text_field>\");ctx._source.namespace=\"<space>\";ctx._source.is_correction=false;ctx._source.public=<public>;ctx._source.confidence=\"high\";ctx._source['@timestamp']=ctx._source.remove(\"<timestamp>\");ctx._source['user.name'] = \"<user.name>\""
    }
}

You may want to specify the type of documents that you reindex in the KB — for example, you may only want to reindex Markdown documents (like Runbooks). You can add a “query” filter to the documents in the source. In the case of GitHub, runbooks are identified with the “type” field containing the string “file,” and you could add that to the reindex query like indicated below. To add also GitHub Issues, you can also include in the query “type” field containing the string “issues”:

"source": {
        "index": "<InternalDocsIndex>",
        "_source": [
            "<text_field>",
            "<timestamp>",
            "namespace",
            "is_correction",
            "public",
            "confidence"
        ],
    "query": {
      "terms": {
        "type": ["file"]
      }
    }

Great! Now that the data is stored in your Knowledge Base, you can ask the Observability AI Assistant any questions about it:

Conclusion

In conclusion, leveraging internal Observability knowledge and adding it to the Elastic Knowledge Base can greatly enhance the capabilities of the AI Assistant. By manually inputting information or programmatically ingesting documents, SREs can create a central repository of knowledge accessible through the power of Elastic and LLMs. The AI Assistant can recall this information, assist with incidents, and provide tailored observability to specific contexts using Retrieval Augmented Generation. By following the steps outlined in this article, organizations can unlock the full potential of their Elastic AI Assistant.

Start enriching your Knowledge Base with the Elastic AI Assistant today and empower your SRE team with the tools they need to excel. Follow the steps outlined in this article and take your incident management and alert remediation processes to the next level. Your journey toward a more efficient and effective SRE operation begins now.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

In this blog post, we may have used or referred to third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch, ESRE, Elasticsearch Relevance Engine and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

As a single platform, Elastic® helps SREs unify and correlate limitless telemetry data, including metrics, logs, traces, and profiling, into a single datastore — Elasticsearch®. By then applying the power of Elastic’s advanced machine learning (ML), AIOps, AI Assistant, and analytics, you can break down silos and turn data into insights. As a full-stack observability solution, everything from infrastructure monitoring to log monitoring and application performance monitoring (APM) can be found in a single, unified experience.

In Elastic 8.11, a technical preview is now available of Elastic’s new piped query language, ES|QL (Elasticsearch Query Language), which transforms, enriches, and simplifies data investigations. Powered by a new query engine, ES|QL delivers advanced search capabilities with concurrent processing, improving speed and efficiency, irrespective of data source and structure. Accelerate resolution by creating aggregations and visualizations from one screen, delivering an iterative, uninterrupted workflow.

Advantages of ES|QL for SREs

SREs using Elastic Observability can leverage ES|QL to analyze logs, metrics, traces, and profiling data, enabling them to pinpoint performance bottlenecks and system issues with a single query. SREs gain the following advantages when managing high dimensionality and high cardinality data with ES|QL in Elastic Observability:

• Improved operational efficiency: By using ES|QL, SREs can create more actionable notifications with aggregated values as thresholds from a single query, which can also be managed through the Elastic API and integrated into DevOps processes.
• Enhanced analysis with insights: ES|QL can process diverse observability data, including application, infrastructure, business data, and more, regardless of the source and structure. ES|QL can easily enrich the data with additional fields and context, allowing the creation of visualizations for dashboards or issue analysis with a single query.
• Reduced mean time to resolution: ES|QL, when combined with Elastic Observability's AIOps and AI Assistant, enhances detection accuracy by identifying trends, isolating incidents, and reducing false positives. This improvement in context facilitates troubleshooting and the quick pinpointing and resolution of issues.

ES|QL in Elastic Observability not only enhances an SRE's ability to manage the customer experience, an organization's revenue, and SLOs more effectively but also facilitates collaboration with developers and DevOps by providing contextualized aggregated data.

In this blog, we will cover some of the key use cases SREs can leverage with ES|QL:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

I will work through these use cases by showcasing how an SRE can solve a problem in an application instrumented with OpenTelemetry and running on Kubernetes. The OpenTelemetry (OTel) demo is on an Amazon EKS cluster, with Elastic Cloud 8.11 configured.

You can also check out our Elastic Observability ES|QL Demo, which walks through ES|QL functionality for Observability.

ES|QL with AI Assistant

As an SRE, you are monitoring your OTel instrumented application with Elastic Observability, and while in Elastic APM, you notice some issues highlighted in the service map.

Using Elastic AI Assistant, you can easily ask for analysis, and in particular, we check on what the overall latency is across the application services.

My APM data is in traces-apm*. What's the average latency per service over the last hour? Use ESQL, the data is mapped to ECS

The Elastic AI Assistant generates an ES|QL query, which we run in the AI Assistant to get a list of the average latencies across all the application services. We can easily see the top four are:

• load generator
• front-end proxy
• frontendservice
• checkoutservice

With a simple natural language query in the AI Assistant, it generated a single ES|QL query that helped list out the latencies across the services.

Noticing that there is an issue with several services, we decide to start with the frontend proxy. As we work through the details, we see significant failures, and through Elastic APM failure correlation , it becomes apparent that the frontend proxy is not properly completing its calls to downstream services.

ES|QL insightful and contextual analysis in Discover

Knowing that the application is running on Kubernetes, we investigate if there are issues in Kubernetes. In particular, we want to see if there are any services having issues.

We use the following query in ES|QL in Elastic Discover:

from metrics-* | where kubernetes.container.status.last_terminated_reason != "" and kubernetes.namespace == "default" | stats reason_count=count(kubernetes.container.status.last_terminated_reason) by kubernetes.container.name, kubernetes.container.status.last_terminated_reason | where reason_count > 0

ES|QL helps analyze 1,000s/10,000s of metric events from Kubernetes and highlights two services that are restarting due to OOMKilled.

The Elastic AI Assistant, when asked about OOMKilled, indicates that a container in a pod was killed due to an out-of-memory condition.

We run another ES|QL query to understand the memory usage for emailservice and productcatalogservice.

ES|QL easily found the average memory usage fairly high.

We can now further investigate both of these services’ logs, metrics, and Kubernetes-related data. However, before we continue, we create an alert to track heavy memory usage.

Actionable alerts with ES|QL

Suspecting a specific issue, that might recur, we simply create an alert that brings in the ES|QL query we just ran that will track for any service that exceeds 50% in memory utilization.

We modify the last query to find any service with high memory usage:

FROM metrics*
| WHERE @timestamp >= NOW() - 1 hours
| STATS avg_memory_usage = AVG(kubernetes.pod.memory.usage.limit.pct) BY kubernetes.deployment.name | where avg_memory_usage > .5

With that query, we create a simple alert. Notice how the ES|QL query is brought into the alert. We simply connect this to pager duty. But we can choose from multiple connectors like ServiceNow, Opsgenie, email, etc.

With this alert, we can now easily monitor for any services that exceed 50% memory utilization in their pods.

Make the most of your data with ES|QL

In this post, we demonstrated the power ES|QL brings to analysis, operations, and reducing MTTR. In summary, the three use cases with ES|QL in Elastic Observability are as follows:

• ES|QL integrated with the Elastic AI Assistant, which uses public LLM and private data, enhances the analysis experience anywhere in Elastic Observability.
• SREs can, in a single ES|QL query, break down, analyze, and visualize observability data from multiple sources and across any time frame.
• Actionable alerts can be easily created from a single ES|QL query, enhancing operations.

Elastic invites SREs and developers to experience this transformative language firsthand and unlock new horizons in their data tasks. Try it today at https://ela.st/free-trial now in technical preview.

• Elastic Observability Tour
• The power of effective log management
• Transforming Observability with the AI Assistant
• ES|QL announcement blog

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Elastic helps reduce the operational work to support more efficient operations, but users still need a way to investigate and understand everything from the cause of an issue to the meaning of specific error messages. As an operations user, if you haven’t run into a particular error before or it's part of some runbook, you will likely go to Google and start searching for information.

OpenAI’s ChatGPT is becoming an interesting generative AI tool that helps provide more information using the models behind it. What if you could use OpenAI to obtain deeper insights (even simple semantics) for an error in your production or development environment? You can easily tie Elastic to OpenAI’s API to achieve this.

Kubernetes, a mainstay in most deployments (on-prem or in a cloud service provider) requires a significant amount of expertise — even if that expertise is to manage a service like GKE, EKS, or AKS.

In this blog, I will cover how you can use Elastic’s watcher capability to connect Elastic to OpenAI and ask it for more information about the error logs Elastic is ingesting from a Kubernetes cluster(s). More specifically, we will use Azure’s OpenAI Service. Azure OpenAI is a partnership between Microsoft and OpenAI, so the same models from OpenAI are available in the Microsoft version.

While this blog goes over a specific example, it can be modified for other types of errors Elastic receives in logs. Whether it's from AWS, the application, databases, etc., the configuration and script described in this blog can be modified easily.

Prerequisites and config

If you plan on following this blog, here are some of the components and details we used to set up the configuration:

• Ensure you have an account on Elastic Cloud and a deployed stack (see instructions here).
• We used a GCP GKE Kubernetes cluster, but you can use any Kubernetes cluster service (on-prem or cloud based) of your choice.
• We’re also running with a version of the OpenTelemetry Demo. Directions for using Elastic with OpenTelemetry Demo are here.
• We also have an Azure account and Azure OpenAI service configured. You will need to get the appropriate tokens from Azure and the proper URL endpoint from Azure’s OpenAI service.
• We will use Elastic’s dev tools, the console to be specific, to load up and run the script, which is an Elastic watcher.
• We will also add a new index to store the results from the OpenAI query.

Here is the configuration we will set up in this blog:

As we walk through the setup, we’ll also provide the alternative setup with OpenAI versus Azure OpenAI Service.

Setting it all up

Over the next few steps, I’ll walk through:

• Getting an account on Elastic Cloud and setting up your K8S cluster and application
• Gaining Azure OpenAI authorization (alternative option with OpenAI)
• Identifying Kubernetes error logs
• Configuring the watcher with the right script
• Comparing the output from Azure OpenAI/OpenAI versus ChatGPT UI

Step 0: Create an account on Elastic Cloud

Follow the instructions to get started on Elastic Cloud.

Once you have the Elastic Cloud login, set up your Kubernetes cluster and application. A complete step-by-step instructions blog is available here. This also provides an overview of how to see Kubernetes cluster metrics in Elastic and how to monitor them with dashboards.

Step 1: Azure OpenAI Service and authorization

When you log in to your Azure subscription and set up an instance of Azure OpenAI Service, you will be able to get your keys under Manage Keys.

There are two keys for your OpenAI instance, but you only need KEY 1 .

Additionally, you will need to get the service URL. See the image above with our service URL blanked out to understand where to get the KEY 1 and URL.

If you are not using Azure OpenAI Service and the standard OpenAI service, then you can get your keys at:

**https** ://platform.openai.com/account/api-keys

You will need to create a key and save it. Once you have the key, you can go to Step 2.

Step 2: Identifying Kubernetes errors in Elastic logs

As your Kubernetes cluster is running, Elastic’s Kubernetes integration running on the Elastic agent daemon set on your cluster is sending logs and metrics to Elastic. The telemetry is ingested, processed, and indexed. Kubernetes logs are stored in an index called .ds-logs-kubernetes.container_logs-default-* (* is for the date), and an automatic data stream logs-kubernetes.container_logs is also pre-loaded. So while you can use some of the out-of-the-box dashboards to investigate the metrics, you can also look at all the logs in Elastic Discover.

While any error from Kubernetes can be daunting, the more nuanced issues occur with errors from the pods running in the kube-system namespace. Take the pod konnectivity agent, which is essentially a network proxy agent running on the node to help establish tunnels and is a vital component in Kubernetes. Any error will cause the cluster to have connectivity issues and lead to a cascade of issues, so it’s important to understand and troubleshoot these errors.

When we filter out for error logs from the konnectivity agent, we see a good number of errors.

But unfortunately, we still can’t understand what these errors mean.

Enter OpenAI to help us understand the issue better. Generally, you would take the error message from Discover and paste it with a question in ChatGPT (or run a Google search on the message).

One error in particular that we’ve run into but do not understand is:

E0510 02:51:47.138292       1 client.go:388] could not read stream err=rpc error: code = Unavailable desc = error reading from server: read tcp 10.120.0.8:46156->35.230.74.219:8132: read: connection timed out serverID=632d489f-9306-4851-b96b-9204b48f5587 agentID=e305f823-5b03-47d3-a898-70031d9f4768

The OpenAI output is as follows:

ChatGPT has given us a fairly nice set of ideas on why this rpc error is occurring against our konnectivity-agent.

So how can we get this output automatically for any error when those errors occur?

Step 3: Configuring the watcher with the right script

What is an Elastic watcher? Watcher is an Elasticsearch feature that you can use to create actions based on conditions, which are periodically evaluated using queries on your data. Watchers are helpful for analyzing mission-critical and business-critical streaming data. For example, you might watch application logs for errors causing larger operational issues.

Once a watcher is configured, it can be:

1. Manually triggered
2. Run periodically
3. Created using a UI or a script

In this scenario, we will use a script, as we can modify it easily and run it as needed.

We’re using the DevTools Console to enter the script and test it out:

The script is listed at the end of the blog in the appendix. It can also be downloaded here .

The script does the following:

1. It runs continuously every five minutes.
2. It will search the logs for errors from the container konnectivity-agent.
3. It will take the first error’s message, transform it (re-format and clean up), and place it into a variable first_hit.

"script": "return ['first_hit': ctx.payload.first.hits.hits.0._source.message.replace('\"', \"\")]"

1. The error message is sent into OpenAI with a query:

What are the potential reasons for the following kubernetes error:
  { { ctx.payload.second.first_hit } }

1. If the search yielded an error, it will proceed to then create an index and place the error message, pod.name (which is konnectivity-agent-6676d5695b-ccsmx in our setup), and OpenAI output into a new index called chatgpt_k8_analyzed.

To see the results, we created a new data view called chatgpt_k8_analyzed against the newly created index:

In Discover, the output on the data view provides us with the analysis of the errors.

For every error the script sees in the five minute interval, it will get an analysis of the error. We could alternatively also use a range as needed to analyze during a specific time frame. The script would just need to be modified accordingly.

Step 4. Output from Azure OpenAI/OpenAI vs. ChatGPT UI

As you noticed above, we got relatively the same result from the Azure OpenAI API call as we did by testing out our query in the ChatGPT UI. This is because we configured the API call to run the same/similar model as what was selected in the UI.

For the API call, we used the following parameters:

"request": {
             "method" : "POST",
             "Url": "https://XXX.openai.azure.com/openai/deployments/pme-gpt-35-turbo/chat/completions?api-version=2023-03-15-preview",
             "headers": {"api-key" : "XXXXXXX",
                         "content-type" : "application/json"
                        },
             "body" : "{ \"messages\": [ { \"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, { \"role\": \"user\", \"content\": \"What are the potential reasons for the following kubernetes error: {{ctx.payload.second.first_hit}}\"}], \"temperature\": 0.5, \"max_tokens\": 2048}" ,
              "connection_timeout": "60s",
               "read_timeout": "60s"
                            }

By setting the role: system with You are a helpful assistant and using the gpt-35-turbo url portion, we are essentially setting the API to use the davinci model, which is the same as the ChatGPT UI model set by default.

Additionally, for Azure OpenAI Service, you will need to set the URL to something similar the following:

https://YOURSERVICENAME.openai.azure.com/openai/deployments/pme-gpt-35-turbo/chat/completions?api-version=2023-03-15-preview

If you use OpenAI (versus Azure OpenAI Service), the request call (against https://api.openai.com/v1/completions) would be as such:

"request": {
            "scheme": "https",
            "host": "api.openai.com",
            "port": 443,
            "method": "post",
            "path": "\/v1\/completions",
            "params": {},
            "headers": {
               "content-type": "application\/json",
               "authorization": "Bearer YOUR_ACCESS_TOKEN"
                        },
            "body": "{ \"model\": \"text-davinci-003\",  \"prompt\": \"What are the potential reasons for the following kubernetes error: {{ctx.payload.second.first_hit}}\",  \"temperature\": 1,  \"max_tokens\": 512,     \"top_p\": 1.0,      \"frequency_penalty\": 0.0,   \"presence_penalty\": 0.0 }",
            "connection_timeout_in_millis": 60000,
            "read_timeout_millis": 60000
          }

If you are interested in creating a more OpenAI-based version, you can download an alternative script and look at another blog from an Elastic community member.

Gaining other insights beyond Kubernetes logs

Now that the script is up and running, you can modify it using different:

• Inputs
• Conditions
• Actions
• Transforms

Learn more on how to modify it here. Some examples of modifications could include:

1. Look for error logs from application components (e.g., cartService, frontEnd, from the OTel demo), cloud service providers (e.g., AWS/Azure/GCP logs), and even logs from components such as Kafka, databases, etc.
2. Vary the time frame from running continuously to running over a specific range.
3. Look for specific errors in the logs.
4. Query for analysis on a set of errors at once versus just one, which we demonstrated.

The modifications are endless, and of course you can run this with OpenAI rather than Azure OpenAI Service.

Conclusion

I hope you’ve gotten an appreciation for how Elastic Observability can help you connect to OpenAI services (Azure OpenAI, as we showed, or even OpenAI) to better analyze an error log message instead of having to run several Google searches and hunt for possible insights.

Here’s a quick recap of what we covered:

• Developing an Elastic watcher script that can be used to find and send Kubernetes errors into OpenAI and insert them into a new index
• Configuring Azure OpenAI Service or OpenAI with the right authorization and request parameters

Ready to get started? Sign up for Elastic Cloud and try out the features and capabilities I’ve outlined above to get the most value and visibility out of your OpenTelemetry data.

Appendix

Watcher script

PUT _watcher/watch/chatgpt_analysis
{
    "trigger": {
      "schedule": {
        "interval": "5m"
      }
    },
    "input": {
      "chain": {
          "inputs": [
              {
                  "first": {
                      "search": {
                          "request": {
                              "search_type": "query_then_fetch",
                              "indices": [
                                "logs-kubernetes*"
                              ],
                              "rest_total_hits_as_int": true,
                              "body": {
                                "query": {
                                  "bool": {
                                    "must": [
                                      {
                                        "match": {
                                          "kubernetes.container.name": "konnectivity-agent"
                                        }
                                      },
                                      {
                                        "match" : {
                                          "message":"error"
                                        }
                                      }
                                    ]
                                  }
                                },
                                "size": "1"
                              }
                            }
                        }
                    }
                },
                {
                    "second": {
                        "transform": {
                            "script": "return ['first_hit': ctx.payload.first.hits.hits.0._source.message.replace('\"', \"\")]"
                        }
                    }
                },
                {
                    "third": {
                        "http": {
                            "request": {
                                "method" : "POST",
                                "url": "https://XXX.openai.azure.com/openai/deployments/pme-gpt-35-turbo/chat/completions?api-version=2023-03-15-preview",
                                "headers": {
                                    "api-key" : "XXX",
                                    "content-type" : "application/json"
                                },
                                "body" : "{ \"messages\": [ { \"role\": \"system\", \"content\": \"You are a helpful assistant.\"}, { \"role\": \"user\", \"content\": \"What are the potential reasons for the following kubernetes error: {{ctx.payload.second.first_hit}}\"}], \"temperature\": 0.5, \"max_tokens\": 2048}" ,
                                "connection_timeout": "60s",
                                "read_timeout": "60s"
                            }
                        }
                    }
                }
            ]
        }
    },
    "condition": {
      "compare": {
        "ctx.payload.first.hits.total": {
          "gt": 0
        }
      }
    },
    "actions": {
        "index_payload" : {
            "transform": {
                "script": {
                    "source": """
                        def payload = [:];
                        payload.timestamp = new Date();
                        payload.pod_name = ctx.payload.first.hits.hits[0]._source.kubernetes.pod.name;
                        payload.error_message = ctx.payload.second.first_hit;
                        payload.chatgpt_analysis = ctx.payload.third.choices[0].message.content;
                        return payload;
                    """
                }
            },
            "index" : {
                "index" : "chatgpt_k8s_analyzed"
            }
        }
    }
}

Additional logging resources:

• Getting started with logging on Elastic (quickstart)
• Ingesting common known logs via integrations (compute node example)
• List of integrations
• Ingesting custom application logs into Elastic
• Enriching logs in Elastic
• Analyzing Logs with Anomaly Detection (ML) and AIOps

Common use case examples with logs:

• Nginx log management
• AWS VPC Flow log management
• PostgreSQL issue analysis with AIOps

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.

Screenshots of Microsoft products used with permission from Microsoft.

Since this is a pretty new technology, we wouldn’t want to burden our shiny new code with proprietary technology, would we? No, we would not, and that is why we are going to use OpenTelemetry to monitor our ChatGPT code in this blog. This is particularly relevant for me as I recently created a service to generate meeting notes from Zoom calls. If I am to release this into the wild, how much is it going to cost me and how do I make sure it is available?

OpenAI APIs to the rescue

The OpenAI API is pretty awesome, there is no doubt. It also gives us the information shown below in each response to each API call, which can help us with understanding what we are being charged. By using the token counts, the model, and the pricing that OpenAI has put up on its website, we can calculate the cost. The question is, how do we get this information into our monitoring tools?

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "logprobs": null,
      "text": "\n\nElastic is an amazing observability tool because it provides a comprehensive set of features for monitoring"
    }
  ],
  "created": 1680281710,
  "id": "cmpl-70CJq07gibupTcSM8xOWekOTV5FRF",
  "model": "text-davinci-003",
  "object": "text_completion",
  "usage": {
    "completion_tokens": 20,
    "prompt_tokens": 9,
    "total_tokens": 29
  }
}

OpenTelemetry to the rescue

OpenTelemetry is truly a fantastic piece of work. It has had so much adoption and work committed to it over the years, and it seems to really be getting to the point where we can call it the Linux of Observability. We can use it to record logs, metrics, and traces and get those in a vendor neutral way into our favorite observability tool — in this case, Elastic Observability.

With the latest and greatest otel libraries in Python, we can auto-instrument external calls, and this will help us understand how OpenAI calls are performing. Let's take a sneak peek at our sample Python application, which implements Flask and the ChatGPT API and also has OpenTelemetry. If you want to try this yourself, take a look at the GitHub link at the end of this blog and follow these steps.

Set up Elastic Cloud account (if you already don’t have one)

1. Sign up for a two-week free trial at https://www.elastic.co/cloud/elasticsearch-service/signup.
2. Create a deployment.

Once you are logged in, click Add integrations.

Click on APM Integration.

Then scroll down to get the details you need for this blog:

Be sure to set the following Environment variables, replacing the variables with data you get from Elastic as above and OpenAI from here, and then run these export commands on the command line.

export OPEN_AI_KEY=sk-abcdefgh5ijk2l173mnop3qrstuvwxyzab2cde47fP2g9jij
export OTEL_EXPORTER_OTLP_AUTH_HEADER=abc9ldeofghij3klmn
export OTEL_EXPORTER_OTLP_ENDPOINT=https://123456abcdef.apm.us-west2.gcp.elastic-cloud.com:443

And install the following Python libraries:

pip3 install opentelemetry-api
pip3 install opentelemetry-sdk
pip3 install opentelemetry-exporter-otlp
pip3 install opentelemetry-instrumentation
pip3 install opentelemetry-instrumentation-requests
pip3 install openai
pip3 install flask

Here is a look at the code we are using for the example application. In the real world, this would be your own code. All this does is call OpenAI APIs with the following message: “Why is Elastic an amazing observability tool?”

import openai
from flask import Flask
import monitor  # Import the module
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
import urllib
import os
from opentelemetry import trace
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.requests import RequestsInstrumentor

# OpenTelemetry setup up code here, feel free to replace the “your-service-name” attribute here.
resource = Resource(attributes={
    SERVICE_NAME: "your-service-name"
})
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=os.getenv('OTEL_EXPORTER_OTLP_ENDPOINT'),
        headers="Authorization=Bearer%20"+os.getenv('OTEL_EXPORTER_OTLP_AUTH_HEADER')))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
RequestsInstrumentor().instrument()



# Initialize Flask app and instrument it

app = Flask(__name__)
# Set OpenAI API key
openai.api_key = os.getenv('OPEN_AI_KEY')


@app.route("/completion")
@tracer.start_as_current_span("do_work")
def completion():
    response = openai.Completion.create(
        model="text-davinci-003",
        prompt="Why is Elastic an amazing observability tool?",
        max_tokens=20,
        temperature=0
    )
    return response.choices[0].text.strip()

if __name__ == "__main__":
    app.run()

This code should be fairly familiar to anyone who has implemented OpenTelemetry with Python here — there is no specific magic. The magic happens inside the “monitor” code that you can use freely to instrument your own OpenAI applications.

Monkeying around

Inside the monitor.py code, you will see we do something called “Monkey Patching.” Monkey patching is a technique in Python where you dynamically modify the behavior of a class or module at runtime by modifying its attributes or methods. Monkey patching allows you to change the functionality of a class or module without having to modify its source code. It can be useful in situations where you need to modify the behavior of an existing class or module that you don't have control over or cannot modify directly.

What we want to do here is modify the behavior of the “Completion” call so we can steal the response metrics and add them to our OpenTelemetry spans. You can see how we do that below:

def count_completion_requests_and_tokens(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        counters['completion_count'] += 1
        response = func(*args, **kwargs)
        token_count = response.usage.total_tokens
        prompt_tokens = response.usage.prompt_tokens
        completion_tokens = response.usage.completion_tokens
        cost = calculate_cost(response)
        strResponse = json.dumps(response)
        # Set OpenTelemetry attributes
        span = trace.get_current_span()
        if span:
            span.set_attribute("completion_count", counters['completion_count'])
            span.set_attribute("token_count", token_count)
            span.set_attribute("prompt_tokens", prompt_tokens)
            span.set_attribute("completion_tokens", completion_tokens)
            span.set_attribute("model", response.model)
            span.set_attribute("cost", cost)
            span.set_attribute("response", strResponse)
        return response
    return wrapper
# Monkey-patch the openai.Completion.create function
openai.Completion.create = count_completion_requests_and_tokens(openai.Completion.create)

By adding all this data to our Span, we can actually send it to our OpenTelemetry OTLP endpoint (in this case it will be Elastic). The benefit of doing this is that you can easily use the data for search or to build dashboards and visualizations. In the final step, we also want to calculate the cost. We do this by implementing the following function, which will calculate the cost of a single request to the OpenAI APIs.

def calculate_cost(response):
    if response.model in ['gpt-4', 'gpt-4-0314']:
        cost = (response.usage.prompt_tokens * 0.03 + response.usage.completion_tokens * 0.06) / 1000
    elif response.model in ['gpt-4-32k', 'gpt-4-32k-0314']:
        cost = (response.usage.prompt_tokens * 0.06 + response.usage.completion_tokens * 0.12) / 1000
    elif 'gpt-3.5-turbo' in response.model:
        cost = response.usage.total_tokens * 0.002 / 1000
    elif 'davinci' in response.model:
        cost = response.usage.total_tokens * 0.02 / 1000
    elif 'curie' in response.model:
        cost = response.usage.total_tokens * 0.002 / 1000
    elif 'babbage' in response.model:
        cost = response.usage.total_tokens * 0.0005 / 1000
    elif 'ada' in response.model:
        cost = response.usage.total_tokens * 0.0004 / 1000
    else:
        cost = 0
    return cost

Elastic to the rescue

Once we are capturing all this data, it’s time to have some fun with it in Elastic. In Discover, we can see all the data points we sent over using the OpenTelemetry library:

With these labels in place, it is very easy to build a dashboard. Take a look at this one I built earlier (which is also checked into my GitHub Repository):

We can also see Transactions, Latency of the OpenAI service, and all the spans related to our ChatGPT service calls.

In the transaction view, we can also see how long specific OpenAI calls have taken:

Some requests to OpenAI here have taken over 3 seconds. ChatGPT can be very slow, so it’s important for us to understand how slow this is and if users are becoming frustrated.

Summary

We looked at monitoring ChatGPT with OpenTelemetry with Elastic. ChatGPT is a worldwide phenomenon and it’s going to no doubt grow and grow, and pretty soon everyone will be using it. Because it can be slow to get responses out, it is critical that people are able to understand the performance of any code that is using this service.

There is also the issue of cost, since it’s incredibly important to understand if this service is eating into your margins and if what you are asking for is profitable for your business. With the current economic environment, we have to keep an eye on profitability.

Take a look at the code for this solution here. And please feel free to use the “monitor” library to instrument your own OpenAI code.

Interested in learning more about Elastic Observability? Check out the following resources:

• An Introduction to Elastic Observability
• Observability Fundamentals Training
• Watch an Elastic Observability demo
• Observability Predictions and Trends for 2023

And sign up for our Elastic Observability Trends Webinar featuring AWS and Forrester, not to be missed!

In this blog post, we may have used third party generative AI tools, which are owned and operated by their respective owners. Elastic does not have any control over the third party tools and we have no responsibility or liability for their content, operation or use, nor for any loss or damage that may arise from your use of such tools. Please exercise caution when using AI tools with personal, sensitive or confidential information. Any data you submit may be used for AI training or other purposes. There is no guarantee that information you provide will be kept secure or confidential. You should familiarize yourself with the privacy practices and terms of use of any generative AI tools prior to use.

Elastic, Elasticsearch and associated marks are trademarks, logos or registered trademarks of Elasticsearch N.V. in the United States and other countries. All other company and product names are trademarks, logos or registered trademarks of their respective owners.