From a 582ms latency spike to the team that owns it, using Kibana Discover

A checkout service is running at 582ms p95 against a 350ms SLO target. One KQL query in Kibana Discover finds it. Working out which team owns that service takes an ES|QL query that joins the metric documents to a small service catalog index using LOOKUP JOIN. Below, that investigation runs in order. A data view narrows the scope and filter pills keep it visible, which matters more than it sounds when someone else has to reconstruct what you searched. KQL does most of the work. Lucene query syntax handles the one case that needs a regex, and ES|QL takes over once filtering stops answering the question.

Prerequisites

To follow along, you need:

• An Elasticsearch cluster with Kibana. Everything up to the ES|QL section works on any recent version; LOOKUP JOIN is generally available in Elasticsearch 9.1 and was a technical preview in 9.0, so use 9.1 or later for the last section.
• No special license tier. Everything in this article, including LOOKUP JOIN, works on the free basic license.
• The two small sample indices created in the next section.

Why checkout latency increased in production

The example starts with a common operations question:

Why did checkout latency increase in production, and which team owns the service?

The metrics index contains 15-minute service measurements for four services across three regions. One service, checkout-api, has higher p95 latency in us-central1 during the investigation window. The goal is to get from all metrics to the small set of documents that explain the issue.

The walkthrough follows these steps:

1. Select the right data view and time range.
2. Use UI filters to include, exclude, disable, and pin criteria.
3. Use KQL for the main field and range search.
4. Switch to Lucene when regular expression syntax is useful.
5. Use ES|QL mode with LOOKUP JOIN to enrich metrics with service catalog data.

Set up the sample metrics index

The walkthrough searches a metrics index named o11y-labs-discover-service-metrics. Create it with keyword fields for the service dimensions and numeric fields for the measurements:

PUT o11y-labs-discover-service-metrics
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "service": {
        "properties": {
          "name": { "type": "keyword" },
          "environment": { "type": "keyword" },
          "version": { "type": "keyword" }
        }
      },
      "cloud": { "properties": { "region": { "type": "keyword" } } },
      "host": { "properties": { "name": { "type": "keyword" } } },
      "metrics": {
        "properties": {
          "latency": { "properties": { "p95_ms": { "type": "float" } } },
          "cpu": { "properties": { "pct": { "type": "float" } } },
          "error": { "properties": { "rate": { "type": "float" } } }
        }
      }
    }
  }
}

Each document is one 15-minute measurement for one service in one region:

POST o11y-labs-discover-service-metrics/_bulk
{ "index": {} }
{ "@timestamp": "2026-06-30T16:15:00.000Z", "service": { "name": "checkout-api", "environment": "production", "version": "2026.06.30-1" }, "cloud": { "region": "us-central1" }, "host": { "name": "checkout-api-us-central1-01" }, "metrics": { "latency": { "p95_ms": 582.6 }, "cpu": { "pct": 0.81 }, "error": { "rate": 0.041 } } }
{ "index": {} }
{ "@timestamp": "2026-06-30T16:15:00.000Z", "service": { "name": "payments-api", "environment": "production", "version": "2026.06.29-7" }, "cloud": { "region": "us-central1" }, "host": { "name": "payments-api-us-central1-01" }, "metrics": { "latency": { "p95_ms": 231.4 }, "cpu": { "pct": 0.31 }, "error": { "rate": 0.008 } } }

To reproduce the screenshots, index one document per service, region, and 15-minute interval:

• Services: checkout-api, checkout-worker, payments-api, inventory-api
• Regions: us-central1, us-east4, europe-west1
• Window: 14:00 to 19:45 UTC on June 30, 2026, giving 24 intervals of 15 minutes
• Documents per interval: 12 production, plus two staging (checkout-api and payments-api, both in us-central1)
• Total: 24 intervals × 14 documents = 336 documents

The exact values do not matter, as long as checkout-api in us-central1 reports metrics.latency.p95_ms above 500 between 15:45 and 18:45 UTC and stays well under 500 ms everywhere else.

Instead of indexing everything by hand, you can run the supporting notebook, which generates the full 336-document dataset, creates both indices, and verifies the final ES|QL query.

The ES|QL section also uses a second, four-document lookup index for service catalog data. We will create it when we get there.

Choose a data view in Kibana Discover

The data view is the first filter in Discover. It decides which Elasticsearch indices are searched, which time field drives the histogram, and which fields are available in the left field list.

For this walkthrough, the Discover data view points to:

o11y-labs-discover-service-metrics

The time field is @timestamp. That matters because the time picker limits the documents before you add a query, a filter pill, or a selected field.

Use a narrow data view when you can. For example, a data view that targets only service metrics makes Discover easier to scan than a broad logs-*,metrics-* data view when you already know the question is about metrics.

Once the data view is selected, add the fields that support the investigation:

• service.name
• service.environment
• cloud.region
• metrics.latency.p95_ms
• metrics.cpu.pct
• metrics.error.rate

Filter pills in Discover: include, exclude, disable, and pin

UI filters are useful when you want a visible, editable list of constraints. They are also helpful when you are exploring fields from the document table and want Discover to write the field syntax for you.

In the document table, use the field actions (the +/- icons that appear when you hover over a value) to include or exclude it. For example:

service.environment: production
NOT cloud.region: us-east4
service.version: 2026.06.29-7  (disabled)

These three filters show the main filter controls:

• Include a value when you want only matching documents.
• Exclude a value when a dimension is not part of the problem.
• Temporarily disable a filter when you want to keep it nearby but remove it from the current query.
• Pin a filter when it should stay active as you move between Kibana apps.

Pinned filters are useful for investigations that cross app boundaries. For example, you can pin service.environment: production before moving from Discover to dashboards, Lens, or another view. Disabled filters are useful for testing a theory without deleting the context that got you there.

The key habit is to keep the filters readable. If a query has a long search expression and many hidden assumptions, another engineer has to reconstruct your thinking. Filter pills make the major scope decisions visible.

KQL query syntax for field, range, and boolean searches

KQL, the Kibana Query Language, is a good default for Discover searches. It supports field names, exact values, ranges, wildcards, and boolean logic in a readable form.

For the checkout latency example, this KQL query narrows the view to one service, one region, and high p95 latency:

service.name : "checkout-api" and cloud.region : "us-central1" and metrics.latency.p95_ms >= 500

Read it from left to right:

• service.name : "checkout-api" keeps one service.
• cloud.region : "us-central1" keeps one cloud region.
• metrics.latency.p95_ms >= 500 keeps latency samples at or above 500 ms.

You can add the environment in KQL:

service.environment : "production" and service.name : "checkout-api" and cloud.region : "us-central1" and metrics.latency.p95_ms >= 500

Or you can keep service.environment: production as a UI filter. Both approaches are valid. For shared investigations, we prefer stable scope, such as environment and service, as filter pills, and the active hypothesis, such as a latency threshold, in the search bar.

KQL also works well for combining fields:

service.environment : "production" and
(service.name : "checkout-api" or service.name : "payments-api") and
metrics.error.rate > 0.02

This is useful when a user-facing flow crosses multiple services. You can compare a small group of services without switching data views or creating a dashboard first.

Lucene query syntax in Kibana: searching with regular expressions

Lucene query syntax is the option in Kibana that supports regular expressions. KQL does not, so when you need a regex in the search bar, open the query menu at the right of the search bar and switch the language to Lucene.

For example, this Lucene query searches production services whose names start with checkout- and whose p95 latency is above 500 ms:

service.name:/checkout-.*/ AND service.environment:production AND metrics.latency.p95_ms:>500

Lucene syntax is more compact, but it is also easier to misread. Use it when it gives you something you cannot express as clearly in KQL, such as a regex pattern over a field. For everyday field, value, and range filtering, KQL is usually easier for a teammate to review.

How to join two indices in Discover with ES|QL LOOKUP JOIN

Classic Discover mode is good when you want to search, filter, inspect fields, and look at raw documents. ES|QL in Discover is better when the question needs transformation before the result is useful. Use the Query in ES|QL button in the Discover toolbar to switch modes.

In this example, raw metrics tell us that checkout-api latency is high. They do not tell us who owns that service or what latency target the service is expected to meet. That data lives in a small service catalog lookup index.

Create a lookup index for service catalog data

PUT o11y-labs-service-catalog-lookup
{
  "settings": {
    "index.mode": "lookup"
  },
  "mappings": {
    "properties": {
      "service": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "owner": {
        "properties": {
          "team": {
            "type": "keyword"
          }
        }
      },
      "slo": {
        "properties": {
          "latency_target_ms": {
            "type": "long"
          }
        }
      },
      "runbook": {
        "properties": {
          "url": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

One catalog document can attach ownership and an SLO target to the service:

POST o11y-labs-service-catalog-lookup/_doc/checkout-api
{
  "service": {
    "name": "checkout-api"
  },
  "owner": {
    "team": "checkout-platform"
  },
  "slo": {
    "latency_target_ms": 350
  },
  "runbook": {
    "url": "https://runbooks.example.com/checkout-api/latency"
  }
}

Run the LOOKUP JOIN query

Now Discover can run an ES|QL query that joins the metric documents with that catalog metadata using LOOKUP JOIN. Remember that this command needs Elasticsearch 9.1 or later, that the lookup index must be created with index.mode: lookup, and that the join field, service.name here, must be mapped as keyword in the lookup index.

FROM o11y-labs-discover-service-metrics
| WHERE @timestamp >= "2026-06-30T15:00:00.000Z" AND @timestamp <= "2026-06-30T18:45:00.000Z"
| WHERE service.environment == "production"
| LOOKUP JOIN o11y-labs-service-catalog-lookup ON service.name
| WHERE owner.team == "checkout-platform" AND metrics.latency.p95_ms > slo.latency_target_ms
| KEEP @timestamp, service.name, cloud.region, metrics.latency.p95_ms, slo.latency_target_ms, owner.team
| SORT @timestamp DESC

This is the part classic mode does not cover. Classic Discover can filter the metric documents, but ES|QL can enrich those rows with data from another index before displaying the result.

The result table answers a more operational question than the original search. It shows the affected service, the region, the latency value, the target, and the owning team in one view.

This pattern is useful for more than ownership. You can keep small lookup indices for service tier, deployment ring, escalation channel, business capability, or runbook URL. Then you can join that context into metric searches at investigation time.

How to choose the right Discover search method

The most useful workflow is not one search language for everything. It is a progression from broad scope to specific evidence.

For a real investigation, start with the smallest data view that still contains the data you need. Add filter pills for stable scope. Use KQL for the active search. Switch to Lucene only when regex syntax is worth the extra complexity. Move to ES|QL when the question needs enrichment, aggregation, or reshaping.

Field naming conventions that make metrics easier to search

Metric search works best when the field names carry enough context. The examples above use Elastic Common Schema-style fields where possible:

• service.name for the monitored service.
• service.environment for production, staging, or development.
• cloud.region for the deployment region.
• host.name for host-level drill-down.
• Numeric metric fields under metrics.*.

You do not need this exact schema to use Discover, but predictable field names make the search bar and filter pills much easier to use. They also make saved searches and screenshots easier to understand during a handoff.

For service catalog data, keep the lookup index small and stable. Fields like service owner, tier, SLO target, and runbook URL change less often than raw metrics. That makes them good candidates for LOOKUP JOIN during analysis.

Run the walkthrough on your own cluster

Use Discover as a drill-down path, not only as a document table. In this walkthrough, we:

• Scoped the search with a narrow data view and the time picker before writing any query.
• Made the investigation scope visible and shareable with include, exclude, disabled, and pinned filter pills.
• Used KQL for readable field, range, and boolean searches.
• Switched to Lucene only for the regex case KQL cannot express.
• Enriched metric documents with ownership and SLO data from a lookup index using ES|QL LOOKUP JOIN.

To try the full flow on your own cluster, run the supporting notebook, which creates both indices and the incident data used in every example.

Related documentation:

• Discover
• Data views
• Filtering
• KQL
• Lucene query syntax
• ES|QL in Discover
• LOOKUP JOIN

Related Observability Labs articles:

• Exploring metrics from a new time series data stream in Discover

• Explore and analyze metrics with ease in Elastic Observability

• Traces in Discover for deeper application insights in Elastic Observability

• Connecting the dots: ES|QL joins for richer observability insights

• Common ES|QL queries for Kubernetes monitoring