How Elastic Significant Events turns 100 log samples into detection rules

Build and observe AI agents where your data already lives. Begin with Elastic Agent Builder. You can also start a free cloud trial or try Elastic on your local machine today.

Significant Events writes detection rules for a log source by reading about 100 of its documents, and which 100 it reads decides what those rules can find. Two jobs, in order: build an evidence-backed description of the source, then generate queries from it. The units of that description are Knowledge Indicators, and log sampling produced 27 of them from the OpenTelemetry demo source this series uses, covering 13 services, 8 dependency edges between them, 4 technologies, the orchestrator and the schema. This post is about how those 100 documents get picked. Drawn uniformly, they describe whatever the busiest service logs on every request.

Part 2 of this series will generate queries from the facts found here. Part 3 will turn them into Significant Events.

Why LLM generated detection rules fail on an unfamiliar log source

You have a new log source. It holds millions of documents, you did not write the services that produce them, and you need detection rules for it. Writing them by hand means reading the logs first, which does not scale.

Pointing a language model at the raw logs is the obvious shortcut, and when we tried it, the output was a generic rule like log.level: error. On a chatty source, that rule fires constantly. On the source this series uses as its example, it returns nothing for the whole incident. The source is OpenTelemetry-shaped, so severity is stored in severity_text, and log.level only resolves because Elasticsearch keeps it as an alias of that field. Either name finds the same thing: only two of its services set a severity at all, and always at info. The handful of documents that fit in a prompt cannot tell the model which case it is in.

Significant Events separates two jobs that a single prompt blurs together. First we build a bounded, evidence-backed description of what is in the source. Then we generate detection queries from that description. We call the units of that description Knowledge Indicators (KIs). The split buys control: extraction and generation can be evaluated separately, so a bad query traces to a missing fact or a misused one, and the KI table is a review point where you correct the system's understanding without writing a query. The cost is that what extraction misses, generation cannot recover.

This series follows the pipeline: source, feature KIs, query KIs, rules, events, Significant Events. This part 1 covers feature KIs and the principle that shaped how we extract them: the model knows only what it is shown. Code decides which documents it reads and what prior knowledge it sees while reading them. Both problems in this post, coverage and identity, turned out to be problems of what the model sees.

The series pipeline. This post covers the highlighted stage; parts 2 and 3 move the highlight along.

A source is a set of log documents defined by an Elasticsearch Query Language (ES|QL) query, a FROM over an index pattern optionally narrowed with WHERE, registered on the Sources page in Elastic nightshift. The running example is the OpenTelemetry demo application on Kubernetes, with its container logs landing in logs.otel. Partway through the capture, we pointed the checkout service at the wrong port for the payment service and tripled the load generator's users. From then on, every order fails at the payment step while traffic keeps climbing.

What a Knowledge Indicator records about a log source

Start with one document, sampled while a service could not reach one of its dependencies. Abridged; the stack trace continues.

{
  "@timestamp": "2026-10-16T13:37:42.885Z",
  "resource": {
    "attributes": {
      "app": "frontend",
      "k8s.namespace.name": "otel-demo",
      "k8s.deployment.name": "frontend",
      "container.image.tag": "1.12.0-frontend"
    },
    "schema_url": "https://opentelemetry.io/schemas/1.40.0"
  },
  "body": {
    "text": "Error: 13 INTERNAL: failed to charge card: could not charge the card: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.104.231.249:9999: i/o timeout\"\n    at callErrorFromStatus (/app/node_modules/@grpc/grpc-js/build/src/call.js:31:19)\n    at Object.onReceiveStatus (/app/node_modules/@grpc/grpc-js/build/src/client.js:193:76)\n    ..."
  }
}

Here is the entity KI the extraction model produced from it and documents like it, with the run metadata and document references left out.

{
  "id": "frontend",
  "type": "entity",
  "subtype": "service",
  "title": "Frontend Service",
  "description": "Frontend service in the otel-demo namespace serves the user-facing application and makes backend gRPC calls during checkout flows. The sample directly identifies the frontend deployment and shows a failing backend RPC from frontend code.",
  "properties": { "name": "frontend", "namespace": "otel-demo" },
  "confidence": 92,
  "evidence": [
    "app=frontend",
    "k8s.deployment.name=frontend",
    "Error: 13 INTERNAL: failed to charge card: could not charge the card: rpc error: code = Unavailable desc = connection error",
    "at /app/.next/server/pages/api/checkout.js:1:1041"
  ],
  "filter": { "field": "k8s.deployment.name", "eq": "frontend" }
}

evidence quotes the log content the claim rests on, so you can check the claim instead of trusting it. filter selects this service's documents; it scopes generated queries and, as we'll see, steers sampling. confidence runs 0 to 100 and tracks how direct the evidence is: a service named by a resource attribute scores high, a component inferred from a request pattern scores lower, and a dependency is never more confident than the two components it connects.

The model extracts five types.

From the document above and others like it, the source's KIs also include these, evidence abridged to one line each:

[
  { "id": "checkout-payment-grpc", "type": "dependency", "subtype": "service_dependency",
    "properties": { "source": "checkout", "target": "payment", "protocol": "grpc" },
    "confidence": 56, "evidence": ["paymentSvcAddr:payment:9999"] },
  { "id": "grpc-js", "type": "technology", "subtype": "framework",
    "properties": { "library": "@grpc/grpc-js" },
    "confidence": 92, "evidence": ["at ServiceClientImpl.makeUnaryRequest (/app/node_modules/@grpc/grpc-js/build/src/client.js:161:32)"] },
  { "id": "otel-schema", "type": "schema", "subtype": "otel",
    "properties": { "schema_family": "otel" },
    "confidence": 85, "evidence": ["resource.attributes.k8s.deployment.name present"] }
]

Part 2 will lean on exactly these KIs: the entity's filter scopes its queries, the dependency says what can fail, the technology says what the failure looks like, and the schema says which field to look in.

Computed Knowledge Indicators: Field inventory, log patterns, log samples

Not every KI comes from the model. Computed KIs are produced by ES|QL against the source: a field inventory with value distributions (dataset_analysis), the categorized message patterns with their counts and a sample of each (log_patterns), and a set of raw log samples (log_samples). They are how real log text reaches the query generator in part 2, which receives a description of the source and never an arbitrary batch of raw documents.

The 27 feature KIs extracted from the example source: 13 entities, 8 dependency edges between them, 4 technologies, the Kubernetes infrastructure, and the schema. Every one is expiring, so it stays only as long as extraction keeps observing it.

How log sampling picks which documents the model reads

The extraction loop: Five batches of 20 documents

Onboarding a source runs extraction first and query generation second, and extraction is a loop. Each iteration samples a batch of documents and hands it, with what is already known about the source, to an agent that returns new and updated features. Code reconciles them against stored KIs, persists the changes, and decides whether to iterate again. By default a run is at most five batches of 20 documents, 100 in all, and it stops early once two consecutive batches add nothing new. A single batch is no bigger than the one-shot prompt that fails; it has to fit in the model's context alongside everything already known. The loop is what makes the difference: each batch is steered toward what previous batches did not explain, and results accumulate in the KI store instead of in one context window.

The extraction step is an Agent Builder agent; a Kibana Workflow owns the loop and makes every run inspectable in the Workflows UI. The agent reasons about what a batch says; code decides which documents it sees and whether what it said matches something already known. Sampling and identity are where correctness lives, so we kept them in code we can measure.

One onboarding run. Only the highlighted step is the model; the loop, the sampling, and the matching are code. The feedback edge is what makes the loop converge.

Why uniform random sampling misses what detection needs

A few batches against millions of documents. Drawn uniformly, they describe the source's frequency distribution, which is dominated by whatever the busiest service logs on every request. The quiet services, rare dependencies, and components that only appear during deploys or failures are exactly what detection needs, and uniform sampling under-represents them. So each batch is drawn from three arms and merged.

How one batch is assembled. The entity-filtered arm excludes every filter found so far in the run, so each batch is drawn from the unexplained remainder; the diverse arm rotates its frequency bands between iterations.

Benchmark: Three-arm log sampling against uniform random

The benchmark that settled this design compared the three-arm sampler against uniform random.

Methodology. Four OpenTelemetry demo scenarios replayed into one source; five mixed and five random runs; extraction on GPT-5.4 mini, judged by Gemini 3.1 Flash-Lite for architectural completeness. The diverse arm in this experiment was an earlier version than today's stratified sampler, so it shows mixed beat random in that setup; it is not a benchmark of the exact production sampler.

The category row shows the mechanism: mixed sampling saw every message shape on every run, random never more than a third. The differentiator the judge kept returning to was dependency edges. On the one run where feature counts tied at twenty, the mixed run named four service-to-service edges and the random run had a generic "downstream gRPC dependency". The counts were equal and the knowledge was not.

Stratified sampling for rare log patterns

The sampler has kept changing. On one high-volume internal source, about six million documents fell in the extraction window and a single benign request-completed pattern accounted for roughly 92 percent of them. Several lifecycle messages that fire on every rollout, which the service owner confirmed matter, appeared 56 to 64 times each. At one in a hundred thousand, those patterns are invisible to any sampler that draws by frequency. A "diverse" arm that picks representatives from the most frequent categories inherits the same bias, since it is diverse only across the head of the distribution. That observation produced the current stratified sampler. It categorizes in two passes, first the patterns above one percent of the stream and then the remainder with those patterns excluded, so rare patterns are counted instead of rounded away. It then takes one pattern per frequency band, rotating across iterations, so the tail gets read.

How to stop the same service being extracted twice

Onboarding runs again, on a schedule once continuous onboarding is enabled, re-describing the same services from fresh samples. Left alone, a model renames things: checkout becomes checkout-service, frontend becomes frontend-v2.3.1, and each rename is a new KI that gives the generator a duplicate to work from.

Two mechanisms handle this: make the same component land on the same ID by construction, and catch what slips past. The model writes a slug id; the storage id is a UUID derived from the source and the lowercased slug, so the same slug from the same source always lands on the same document. Versions, build hashes, and image tags never belong in the id. The model is shown a compact inventory of the source's live IDs, bounded by the context window, and is instructed to reuse an exact ID when it sees the same component and to check anything else with its similarity-search tool.

When the result comes back, code matches each returned feature against the known ones in three tiers, never on name similarity:

1. Exact slug. The slug the model wrote matches a stored one.
2. Same type, version-like suffixes stripped. frontend-v2.3.1 lands on frontend.
3. Same type, subtype, and properties. The fingerprint the documentation describes as the deduplication key.

A match merges into the stored KI and a miss creates one.

How Knowledge Indicators expire, and what you can override

A KI is a claim about a system that changes, so an inventory that only grows becomes wrong. KIs the system extracts are expiring: each one lapses 30 days after it was last observed, a grace period since a single run can miss a component that is still there. KIs you create or mark durable never expire; only you remove them. Excluding a KI keeps it out of later runs.

Where to review and exclude Knowledge Indicators

In Elastic nightshift, onboarding a source from the Sources page triggers a run. The Knowledge Indicators tab under Knowledge lists every KI with the source it came from, and is where you exclude or restore a KI and toggle it between durable and expiring. Continuous onboarding across enabled sources is a separate setting that is off by default; see the documentation.

What log sampling still cannot solve

Three things are open:

• Rare-pattern coverage at billion-document scale. No sampler solves this. A message that appears once in a billion documents needs a different mechanism.
• Cross-source identity. The same service in two sources is two KIs today, deliberately, and we have not explored merging them.
• Cost gating. Deciding how much extraction to run based on how much is left to discover is still maturing.

The agent-driven alternative we are benchmarking

The larger experiment is an agent-driven discovery loop: an agent with tools to pull its own samples, look up what is known, and write KIs directly, benchmarked against the loop described here on the same datasets. That moves the sampling and identity guarantees from code into the agent's judgment, and it has to earn its way in through the same measurements.

What part 2 will cover

Part 2 picks up the KIs extracted above and shows the queries the generator produces because they exist: scoped to the calling service by its filter, aware of the dependency that failed, matching the failure in the field this schema carries, and grounded so they retire when their sources do. Part 2 generates only from what part 1 found, and part 1 found only what it sampled.