We recently contributed to the mastra-ai/mastra open source project by adding support for Elasticsearch as a vector database. With this new feature, you can use Elasticsearch natively in Mastra to store embeddings. In addition to vectors, Elasticsearch provides a suite of advanced features to address all your context engineering requirements. (for example, hybrid search and reranking).

This article details the creation of an agent to implement a retrieval augmented generation (RAG) architecture using Elasticsearch. We’ll showcase a demo project where an agentic approach is used to interact with a corpus of sci-fi movie data stored within Elasticsearch. The project is available at elastic/mastra-elasticsearch-example.

Mastra

Mastra is a TypeScript framework to create agentic AI applications.

A project structure in Mastra looks as follows:

src/
├── mastra/
│   ├── agents/
│   │   └── weather-agent.ts
│   ├── tools/
│   │   └── weather-tool.ts
│   ├── workflows/
│   │   └── weather-workflow.ts
│   ├── scorers/
│   │   └── weather-scorer.ts
│   └── index.ts
├── .env.example
├── package.json
└── tsconfig.json

In Mastra, you can build agents, tools, workflows, and scores.

An agent is a class that accepts a message in input and produces a response as output. An agent can use tools, large language models (LLMs), and a memory (figure 1).

An agent's tools allow it to interact with the "external world," such as communicating with a web API or performing an internal operation, like querying Elasticsearch. The memory component is crucial for storing the history of conversations, including past inputs and outputs. This stored context enables the agent to provide more informed and relevant responses to future questions by using its past interactions.

Workflows let you define complex sequences of tasks using clear, structured steps rather than relying on the reasoning of a single agent (figure 2). They give you full control over how tasks are broken down, how data moves between them, and what gets executed when. Workflows run using the built-in execution engine by default or can be deployed to workflow runners.

In Mastra, you can also define scores, which are automated tests that evaluate agent outputs using model-graded, rule-based, and statistical methods. Scorers return scores: numerical values (typically between 0 and 1) that quantify how well an output meets your evaluation criteria. These scores enable you to objectively track performance, compare different approaches, and identify areas for improvement in your AI systems. Scorers can be customized with your own prompts and scoring functions.

Elasticsearch

For running the demo project, we need to have an Elasticsearch instance running. You can activate a free trial on Elastic Cloud or install it locally using the start-local script:

curl -fsSL https://elastic.co/start-local | sh

This will install Elasticsearch and Kibana on your computer and generate an API key to be used for configuring the Mastra integration.

The API key will be shown as output of the previous command and stored in a .env file in the elastic-start-local folder.

Install and configure the demo

We created an elastic/mastra-elasticsearch-example repository containing the source code of the demo project. The example reported in the repository illustrates how to create an agent in Mastra that implements a RAG architecture for retrieving documents from Elasticsearch.

We provided a dataset for the demo about sci-fi movies. We extracted 500 movies from the IMDb dataset on Kaggle.

The first step is to install the dependencies of the project with npm, using the following command:

npm install

Then we need to configure the .env file that will contain the settings. We can generate this file copying the structure from the .env.example file, using the following command:

cp .env.example .env

Now we can edit the .env, adding the missing information:

OPENAI_API_KEY=
ELASTICSEARCH_URL=
ELASTICSEARCH_API_KEY=
ELASTICSEARCH_INDEX_NAME=scifi-movies

The name of the Elasticsearch index is scifi-movies. If you want, you can change it using the env variable ELASTICSEARCH_INDEX_NAME.

We used OpenAI as embedding service, which means that you need to provide an API key for OpenAI in the OPENAI_API_KEY env variable.

The embedding model used in the example is openai/text-embedding-3-small, with an embedding dimension of 1536.

To generate the final answer, we used the openai/gpt-5-nano model to reduce the costs.

The RAG architecture allows you to use a less powerful (and typically less expensive) final LLM model because the heavy lifting of grounding the answer is done by the retrieval component (Elasticsearch in this case).

The smaller LLM is only responsible for two main tasks:

• Rephrasing/embedding the query: Converting the user's natural language question into a vector embedding for semantic search.
• Synthesizing the answer: Taking the highly relevant, retrieved context chunks (documents/movies) and synthesizing them into a coherent, final, human-readable answer, following the provided prompt instructions.

Since the RAG process provides the exact factual context needed for the answer, the final LLM doesn't need to be massive or highly complex and it doesn’t need to possess all the required knowledge within its own parameters (which is where large, expensive models excel). It essentially acts as a sophisticated text summarizer and formatter for the context provided by Elasticsearch, rather than as a full-fledged knowledge base itself. This enables the use of models like gpt-5-nano for cost and latency optimization.

After the configuration of the .env file, you can ingest the movies to Elasticsearch using the following command:

npx tsx src/utility/store.ts

You should see an output as follows:

🚀 Starting ingestion of 500 movies from 500_scifi_movies.jsonl...
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 1/500 (0%) | ok:1 | fail:0 | chunks:1 | eta:19m 33s | current:Capricorn One
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 2/500 (0%) | ok:2 | fail:0 | chunks:2 | eta:10m 32s | current:Doghouse
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 3/500 (1%) | ok:3 | fail:0 | chunks:3 | eta:7m 33s | current:Dinocroc
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 4/500 (1%) | ok:4 | fail:0 | chunks:7 | eta:6m 10s | current:Back to the Future           
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 5/500 (1%) | ok:5 | fail:0 | chunks:9 | eta:5m 14s | current:The Projected Man            
Ingesting ░░░░░░░░░░░░░░░░░░░░░░░░ 6/500 (1%) | ok:6 | fail:0 | chunks:11 | eta:4m 41s | current:I, Robot
...
✅ Ingestion complete in 1m 46s. Success: 500, Failed: 0, Chunks: 693.

The mapping of the scifi-movies index contains the following fields:

• embedding, dense_vector with 1536 dimension, cosine similarity.
• description, text containing the description of the movie.
• director, text containing the name of the director.
• title, text containing the title of the movie.

We generated the embeddings using the title + description. Since the title and the description are two separate fields, the concatenation of both ensures that the resulting embedding vector captures both the specific, unique identity (title) and the rich, descriptive context (description) of the movie, leading to more accurate and comprehensive semantic search results. This combined input gives the embedding model a better single representation of the document's content for similarity matching.

Run the demo

You can run the demo with the following command:

npm run dev

This command will start a web application at localhost:4111 to access Mastra Studio (figure 3).

Mastra Studio offers an interactive UI for building and testing your agents, along with a REST API that exposes your Mastra application as a local service. This lets you start building right away without worrying about integration.

We provided an Elasticsearch Agent that uses the createVectorQueryTool by Mastra as a tool for executing semantic search using Elasticsearch. This agent uses the RAG approach to search for relevant documents (that is, movies) to answer the user’s question.

This agent uses the following prompt:

You are a helpful assistant that answers questions based on the provided context.
Follow these steps for each response:

1. First, carefully analyze the retrieved context chunks and identify key information.
2. Break down your thinking process about how the retrieved information relates to the query.
3. Draw conclusions based only on the evidence in the retrieved context.
4. If the retrieved chunks don't contain enough information, explicitly state what's missing.

Format your response as:
THOUGHT PROCESS:
- Step 1: [Initial analysis of retrieved chunks]
- Step 2: [Reasoning based on chunks]

FINAL ANSWER:
[Your concise answer based on the retrieved context]

Important: When asked to answer a question, please base your answer only on the context provided in the tool. 
If the context doesn't contain enough information to fully answer the question, please state that explicitly and stop it.
Do not add more information than what is present in the retrieved chunks.
Remember: Explain how you're using the retrieved information to reach your conclusions.

If you click on the Mastra Studio > Agents menu and select Elasticsearch Agent, you can test the agent using a chat system. For instance, you can ask information regarding sci-fi movies with a question as follows:

Find 5 movies or TV series about UFOs.

You’ll notice that the agent will execute the vectorQueryTool. You can click on the invoked tool to have a look at the input and the output. At the end of execution, the LLM will reply to your question, given the context coming from the scifi-movies index of Elasticsearch (figure 4).

Mastra executes the following steps internally:

1. Vector conversion: The user's question, Find 5 movies or TV series about UFOs, is converted into a vector embedding using OpenAI's openai/text-embedding-3-small model.
2. Vector search: This embedding is then used to query Elasticsearch via a vector search.
3. Result retrieval: Elasticsearch returns a set of 10 movies highly relevant to the query (that is, those with vectors closest to the user's query vector).
4. Answer generation: The retrieved movies and the original user question are sent to the LLM, specifically openai/gpt-5-nano. The LLM processes this information and generates a final answer, ensuring that the user's request for five results is met.

The Elasticsearch Agent

Here we reported the source code of Elasticsearch Agent.

import { Agent } from "@mastra/core/agent";
import { ElasticSearchVector } from '@mastra/elasticsearch';
import { createVectorQueryTool } from '@mastra/rag';
import { ModelRouterEmbeddingModel } from "@mastra/core/llm";
import { Memory } from "@mastra/memory";

const es_url = process.env.ELASTICSEARCH_URL;
const es_apikey = process.env.ELASTICSEARCH_API_KEY;
const es_index_name = process.env.ELASTICSEARCH_INDEX_NAME;
const prompt = 'insert here the previous prompt';

const esVector = new ElasticSearchVector({
  id: 'elasticsearch-vector',
  url: es_url,
  auth: {
    apiKey : es_apikey
  }
});

const vectorQueryTool = createVectorQueryTool({
  vectorStore: esVector,
  indexName: es_index_name,
  model: new ModelRouterEmbeddingModel("openai/text-embedding-3-small")
});

export const elasticsearchAgent = new Agent({
  id: "elasticsearch-agent",
  name: "Elasticsearch Agent",
  instructions: prompt,
  model: 'openai/gpt-5-nano',
  tools: { vectorQueryTool },
  memory: new Memory(),
});

The vectorQueryTool is the tool that’s invoked to implement the retrieval part of the RAG example. It uses the ElasticSearchVector implementation that Elastic contributed to Mastra.

The agent is an object of the agent class that consumes the vectorQueryTool, the prompt, and a memory. As you can see, the code that we need to put in place for connecting Elasticsearch to an agent is very minimal.

Conclusion

This article demonstrated the simplicity and power of integrating Elasticsearch with the Mastra framework to build sophisticated agentic AI applications. Specifically, we walked through creating a RAG agent capable of performing semantic search over a corpus of sci-fi movie data indexed in Elasticsearch.

A key takeaway is the direct contribution by Elastic to the Mastra open source project, providing native support for Elasticsearch as a vector store. This integration significantly lowers the barrier to entry, as evidenced by the Elasticsearch Agent source code. Using the ElasticSearchVector and createVectorQueryTool, the complete setup for connecting Elasticsearch to your agent requires only a minimal number of lines of configuration code.

Elasticsearch provides several advanced features to enhance result relevance. For example, hybrid search significantly boosts accuracy by combining lexical search with vector search. Another interesting feature is reranking using the latest Jina models that can be applied at the end of hybrid search. To learn more about these techniques, consult the following articles from Elasticsearch Labs:

• Elasticsearch hybrid search by Valentin Crettaz
• An introduction to Jina models, their functionality, and uses in Elasticsearch by Scott Martens

We also encourage you to explore the provided example and begin building your own data-powered agents with Mastra and Elasticsearch. For more information about Mastra, you can have a look at the official documentation here.

Why this matters

• Security and support: The last Elasticsearch 6.x patch release was on January 13, 2022. ECK lets you upgrade at your own pace, with a supported operator from the creators of Elasticsearch. Remaining on an old Elasticsearch version exposes you to supportability risks or well-known security issues.
• Features you’ve been missing: Autoscaling, data tiers, machine learning (ML) jobs, searchable snapshots. None of these are available in the legacy operator.

Future-proof operations: ECK ships day-and-date with every new Elastic release, so you’re never stuck waiting again.

High-level plan

Feel free to bookmark this list. Each milestone is small, reversible, and validated before you move on.

0. Preflight checks

A. Health first: Run /_cat/health and make sure you’re green.

B. Disk watermarks: Keep at least 20% free before starting a migration.

C. Final snapshot: S3, GCS, NFS: It doesn’t matter, as long as you can mount the same repo in the new cluster.

1. If you don’t have object storage handy in your environment, you can use this solution-post by Red Hat to snapshot your data to local storage on the OpenShift cluster.

D. Review the documentation: Elastic provides thorough documentation for migrating data between Elasticsearch clusters.

1. Installing ECK 2.16.1 (your “bridge” operator)

ECK 2.16.1 is the last release that still accepts spec.version: 6.8.x, which makes it the ideal bridge between past and future Elasticsearch versions.

helm repo add elastic https://helm.elastic.co
helm repo update
oc create namespace elastic-system 

helm install elastic-operator elastic/eck-operator --version=2.16.1 -n elastic-system --create-namespace

You can keep the Red Hat operator in place; the two watch different Custom Resource Definitions (CRDs), so they won’t step on each other’s toes.

Keep in mind that with OpenShift, ECK might display some Transport Layer Security (TLS) errors in its logs as OpenShift tries to connect to its healthcheck webhook endpoint via HTTP, but ECK allows TLS communication only. This is a well-known issue and shouldn’t pose a problem.

You can refer further to Elastic documentation, in case you need to make a local namespaced installation.

2. Launching a 6.x cluster under ECK

Below is a starter Kubernetes manifest that balances resiliency (separate masters) with cost (three hot-tier data nodes). Swap storage class names, resources, and snapshot credentials to match your environment.

Note: The syntax used below is a bit different than what it would be for newer Elasticsearch versions on ECK.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: es-logs
  namespace: elastic # Create this namespace prior, or use another namespace
spec:
  version: 6.8.23
  nodeSets:
    - name: hot
      count: 3
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: gp3-csi   # adjust if needed
            resources:
              requests:
                storage: 100Gi # Storage may vary depending on  
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.attr.data: hot
        cluster.routing.allocation.awareness.attributes: data
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 16Gi
                  cpu: 2
                limits:
                  memory: 16Gi
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elastic
spec:
  version: 6.8.23
  count: 1
  elasticsearchRef:
    name: es-logs
  podTemplate:
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
            limits:
              memory: 4Gi

Deploy it, watch pods come up, and you’re ready for data.

3. Moving the data

To move data from one Elasticsearch cluster to another, you can also further consult this guide in the Elastic documentation. For the purpose of this post, we’re assuming that snapshot and restore are used.

Snapshot and restore are quickest:

# on the old cluster, take a snapshot
PUT _snapshot/log-backups
{
  "type": "s3",
  "settings": { ... }
}

PUT _snapshot/log-backups/final-snap-2025-08-07

# on the new cluster (readonly!)
PUT _snapshot/log-backups
{
  "type": "s3",
  "settings": {
    "readonly": true,
    ...
  }
}

# Perform the restore operation
POST _snapshot/log-backups/final-snap-2025-08-07/_restore

Can’t share an object store? Use remote re-index (slower, but works everywhere; has drawbacks in terms of not migrating index templates, component templates, and more) or pump logs through a one-off Logstash job.

4. Configuring ClusterLogging operator

First, we’ll need to decommission our Red Hat operator–managed Elasticsearch cluster. We’ll modify our ClusterLogging like so:

oc edit clusterlogging instance -n openshift-logging 
---------
 logStore:
    elasticsearch:
      nodeCount: 0 # scale down node count, previously > 0 
      redundancyPolicy: ZeroRedundancy
    type: elasticsearch
  managementState: Managed # this needs to be kept, as it will manage the fluentd instance for us.
  visualization:
    kibana:
      replicas: 0 # scale down kibana as well 
    type: kibana

Then we’ll define a ClusterLogForwarder to direct the logs from fluentd to our newly built Elasticsearch 6.x cluster managed by ECK. We’ll need to create a secret with the Elasticsearch credentials:

oc create secret generic eck-es-credentials \
  -n openshift-logging \
  --from-literal=username=elastic \
  --from-literal=password=$(oc get secret es-logs-es-elastic-user -n elastic -o jsonpath='{.data.elastic}' | base64 -d)

For configuring TLS (as recommended), you’ll need to create a ConfigMap for ClusterLogForwarder to trust the ECK ca certificates. Further guidance can be found here. We’ll run the command: 

oc -n elastic get secret es-logs-es-http-certs-public \
-o go-template='{{index .data "tls.crt" | base64decode}}' > ca.crt

oc -n openshift-logging create configmap eck-es-ca \
--from-file=ca-bundle.crt=ca.crt

To create the certificate secret, and then we’ll reference it in the ClusterLogging CRD:

apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: instance
  namespace: openshift-logging
spec:
  outputs:
    - name: eck-es
      type: elasticsearch
      url: https://es-logs-es-http.elastic.svc:9200
      secret:
        name: eck-es-credentials # this secret needs to be created first
      tls:
        # insecureSkipVerify: true # can be used for lab testing purposes
        ca:
          name: eck-es-ca
  pipelines:
    - name: send-to-eck
      inputRefs:
        - application
        - infrastructure
        - audit
      outputRefs:
        - eck-es

⚠️ If you’re troubleshooting connectivity issues, you can temporarily set tls.insecureSkipVerify: true, but this shouldn’t be used long term.

Because we’re restoring legacy indices into a fresh ECK-managed cluster, OpenShift Logging will not automatically recreate the legacy index layout or aliases. You must ensure that write aliases exist and point to writable indices. In my case, I needed to verify that I have proper aliases, set up as:

• app-write
• infra-write
• audit-write

Pointing to indices with dynamic mappings (not recommended) for minimizing errors and troubleshooting steps. 

# Forward ES port to local machine
oc -n elastic port-forward svc/es-logs-es-http 9200:9200

PASS="$(oc -n elastic get secret es-logs-es-elastic-user -o jsonpath='{.data.elastic}' | base64 -d)"

# Make sure the write alias points to the correct backing index
curl -s -k -u "elastic:${PASS}" -XPOST "https://localhost:9200/_aliases" \
  -H 'Content-Type: application/json' \
  -d '{
    "actions": [
      { "add": { "index": "infra-000002", "alias": "infra-write", "is_write_index": true } }
    ]
  }'

Repeat for app-write and audit-write with their respective backing indices.We should see data start flowing now toward our new ECK managed cluster.

5. Rolling upgrade to 7.17.29, and verify

Now you can finally leave 6.x behind.

A. Check _xpack/migration/deprecations?pretty using curl against Elasticsearch, to tackle deprecations. This API will return either warnings or critical things to attend to before you upgrade.

B. Patch the CRD to upgrade it to the latest 7.x version. I’m using 7.17.29.

oc -n elastic patch elasticsearch es-logs --type=merge -p '{"spec":{"version":"7.17.29"}}'

C. ECK restarts nodes one at a time. Your cluster should be online throughout.

D. Give cluster tasks and shard recoveries time to settle before pressing on.

E. Don’t forget to upgrade Kibana in the same way.

oc -n elastic patch kibana kibana --type=merge -p '{"spec":{"version":"7.17.29"}}'

Once complete, check your Elasticsearch version and Kibana version, as well as the health state:

oc -n elastic get elasticsearch es-logs
oc -n elastic get kibana kibana

6. Operator upgrade: ECK 2.16.1 → 3.3.1

ECK upgrades are pleasantly boring:

helm upgrade elastic-operator elastic/eck-operator -n elastic-system --version 3.3.1

Watch the operator pod roll. Your Elasticsearch cluster keeps running; only the controller restarts.

Verify that the upgrade is successful by looking at the operator logs and ensuring that no major errors appear:

oc logs -n elastic-system sts/elastic-operator

And then verifying the new version of the operator (will now be 3.3.1):

helm -n elastic-system list

7. Your roadmap to 8.x and 9.x (when you’re ready)

You’re now on:

• ECK Operator: 3.3.1
• Elastic Stack: 7.17.29

That pair is fully supported and serves as the official launchpad for 8.x. It’s important to first go through the Elastic upgrade documentation.

We’ll again go through the procedure of checking for any hard-breaking changes between our 7.17.29 and the latest 8 version (8.19.9):

GET _migration/deprecations?pretty

It's important to look through the result of this query carefully and to go through necessary steps, like re-indexing indices and changing mappings, among others.

Once you’ve addressed all required changes from 7.17.29 to 8.x:

oc -n elastic patch elasticsearch es-logs --type=merge -p '{"spec":{"version":"8.19.9"}}'
oc -n elastic patch kibana kibana --type=merge -p '{"spec":{"version":"8.19.9"}}'

ECK will handle the rest. Just remember to upgrade Beats, Logstash pipelines, and client libraries in lockstep to avoid wire-protocol surprises.

Repeat the process again to migrate to the latest 9.x version.

8. Cleanup

• Remove the Red Hat Elasticsearch operator.

Now that you’re no longer using the Red Hat Elasticsearch operator, you can remove it from your cluster. You can do that via the following steps:

A. In the OpenShift Console, go to Operators and then to Installed Operators.

B. In the Filter By Name field, enter “Elasticsearch” to find the installed Red Hat Elasticsearch operator.

C. On the Operator Details page, select Uninstall Operator from the Actions list.

D. On the Uninstall Operator? dialog box, select Uninstall. This removes the operator, the operator deployments, and the pods. After this step, the operator stops running and will no longer receive updates.

All of these steps can be found in this link from Red Hat OpenShift documentation.

Wrapping up

By installing ECK 2.16.1 as a bridge, snapshot-restoring into a new cluster, and stepping cleanly through 7.x before landing on ECK 3.3, you’ve transformed an aging, unsupported logging back end into a modern, secure, first-class Elastic deployment, without surprises or downtime.

EIS provides managed, GPU-accelerated inference tightly integrated with Elasticsearch. With EIS, you don’t need to host, scale, or maintain infrastructure for embedding models.

Semantic search retrieves results based on meaning. Text is converted into vector embeddings so queries can match related concepts, even when the exact words differ.

The semantic_text field type simplifies this entire workflow, with automatic chunking, embedding generation at index time, and seamless querying via the semantic query, without building custom pipelines or managing separate model inference.

The jina-embeddings-v5-text model family just launched on EIS, giving developers powerful multilingual embeddings accessible as part of the core semantic_text workflow. So now your semantic search works across languages out of the box, and global datasets, such as support articles, product descriptions, user reviews, and multilingual websites, work without extra configuration.

This default opens up broad, globe-spanning semantic retrieval with no operational overhead.

jina-embeddings-v5-text

The jina-embeddings-v5-text models represent the latest generation of compact, high-performance multilingual embedding models on EIS.

• State-of-the-art multilingual performance: Top scores on MMTEB benchmarks across hundreds of languages. jina-embeddings-v5-text-nano leads models under 500M parameters, and jina-embeddings-v5-text-small outperforms significantly larger alternatives.
• Multiple task capabilities: Spanning across retrieval, semantic matching, clustering, and classification.
• Flexible choices to fit your use case: Two model sizes (small, nano) let you balance speed, cost, and quality.
• Long-context support: Embed long texts efficiently, ideal for document collections with extended context.

Get started

1. Create index

Define a semantic_text field with no additional configuration. Embeddings will be generated automatically at index time using the default model. For production workloads, explicitly specify the model to ensure consistent behavior and results.

PUT /multilingual-reviews
{
  "mappings": {
    "properties": {
      "product": { "type": "keyword" },
      "review": { "type": "semantic_text" },
      "language": { "type": "keyword" }
    }
  }
}

2. Index multilingual documents

Add product reviews in six different languages. Each document’s review field is automatically embedded at ingest time, with no separate pipeline or preprocessing needed.

POST /multilingual-reviews/_bulk?refresh=wait_for
{ "index": { "_id": "1" } }
{ "product": "wireless-headphones", "review": "Amazing noise cancellation and the battery lasts all day. Perfect for long flights.", "language": "en" }
{ "index": { "_id": "2" } }
{ "product": "wireless-headphones", "review": "La cancelación de ruido es impresionante. Muy cómodos incluso después de horas de uso.", "language": "es" }
{ "index": { "_id": "3" } }
{ "product": "wireless-headphones", "review": "ノイズキャンセリングが素晴らしく、長時間つけていても耳が痛くなりません。", "language": "ja" }
{ "index": { "_id": "4" } }
{ "product": "wireless-headphones", "review": "Réduction de bruit excellente et très confortable pour les longs trajets en avion.", "language": "fr" }
{ "index": { "_id": "5" } }
{ "product": "wireless-headphones", "review": "Hervorragende Geräuschunterdrückung. Ideal für Pendler und Vielflieger.", "language": "de" }
{ "index": { "_id": "6" } }
{ "product": "wireless-headphones", "review": "O cancelamento de ruído é excelente e a bateria dura o dia todo.", "language": "pt" }

3. Search across languages with a query in English

GET /multilingual-reviews/_search
{
  "query": {
    "match": {
      "review": "comfortable for long flights"
    }
  }
}

The results show all six reviews ranked by semantic relevance to the English query:

{
  "took": 83,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 6,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.8275735,
    "hits": [
      {
        "_index": "multilingual-reviews",
        "_id": "4",
        "_score": 0.8275735,
        "_source": {
          "product": "wireless-headphones",
          "review": "Réduction de bruit excellente et très confortable pour les longs trajets en avion.",
          "language": "fr"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "1",
        "_score": 0.7616198,
        "_source": {
          "product": "wireless-headphones",
          "review": "Amazing noise cancellation and the battery lasts all day. Perfect for long flights.",
          "language": "en"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "5",
        "_score": 0.72122526,
        "_source": {
          "product": "wireless-headphones",
          "review": "Hervorragende Geräuschunterdrückung. Ideal für Pendler und Vielflieger.",
          "language": "de"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "2",
        "_score": 0.6867013,
        "_source": {
          "product": "wireless-headphones",
          "review": "La cancelación de ruido es impresionante. Muy cómodos incluso después de horas de uso.",
          "language": "es"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "3",
        "_score": 0.66513836,
        "_source": {
          "product": "wireless-headphones",
          "review": "ノイズキャンセリングが素晴らしく、長時間つけていても耳が痛くなりません。",
          "language": "ja"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "6",
        "_score": 0.61658823,
        "_source": {
          "product": "wireless-headphones",
          "review": "O cancelamento de ruído é excelente e a bateria dura o dia todo.",
          "language": "pt"
        }
      }
    ]
  }
}

Notice that the French review ranks first, even above the English one. That's because "très confortable pour les longs trajets en avion" ("very comfortable for long trips by plane") is a closer semantic match to the query than the English review, which splits its focus across noise cancellation, battery life, and flights. This demonstrates the jina-embeddings-v5-text-small ability to rank by meaning, not language.

4. Search across languages with a Japanese query

GET /multilingual-reviews/_search
{
  "query": {
    "match": {
      "review": "長時間のフライトに最適"
    }
  }
}

The results show all six reviews ranked by semantic relevance to the Japanese query (“Ideal for long-haul flights”):

{
  "took": 89,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 6,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 6,
      "relation": "eq"
    },
    "max_score": 0.7556782,
    "hits": [
      {
        "_index": "multilingual-reviews",
        "_id": "4",
        "_score": 0.7556782,
        "_source": {
          "product": "wireless-headphones",
          "review": "Réduction de bruit excellente et très confortable pour les longs trajets en avion.",
          "language": "fr"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "1",
        "_score": 0.7395687,
        "_source": {
          "product": "wireless-headphones",
          "review": "Amazing noise cancellation and the battery lasts all day. Perfect for long flights.",
          "language": "en"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "5",
        "_score": 0.68835545,
        "_source": {
          "product": "wireless-headphones",
          "review": "Hervorragende Geräuschunterdrückung. Ideal für Pendler und Vielflieger.",
          "language": "de"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "3",
        "_score": 0.6487931,
        "_source": {
          "product": "wireless-headphones",
          "review": "ノイズキャンセリングが素晴らしく、長時間つけていても耳が痛くなりません。",
          "language": "ja"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "6",
        "_score": 0.6241487,
        "_source": {
          "product": "wireless-headphones",
          "review": "O cancelamento de ruído é excelente e a bateria dura o dia todo.",
          "language": "pt"
        }
      },
      {
        "_index": "multilingual-reviews",
        "_id": "2",
        "_score": 0.6183049,
        "_source": {
          "product": "wireless-headphones",
          "review": "La cancelación de ruido es impresionante. Muy cómodos incluso después de horas de uso.",
          "language": "es"
        }
      }
    ]
  }
}

The ranking is nearly identical to the English query: French and English still lead because they're the most semantically relevant to "perfect for long flights," regardless of query language. The Japanese review didn't get artificially boosted just because the query was in Japanese. It ranks fourth because it discusses wearing comfort, not flights. Semantic relevance takes priority over language matching.

Note: For English-only use cases

If you prefer a sparse representation or would like to continue to use Elastic Learned Sparse EncodeR (ELSER) for English workloads, ELSER remains available and fully supported as an option for semantic_text.

You can explicitly choose ELSER by specifying inference_id: ".elser-2-elastic in your mappings when creating an index.

Conclusion: Semantic search without borders

With semantic_text now defaulting to the jina-embeddings-v5-text family on Elastic Inference Service, multilingual semantic search becomes the standard developer experience in Elasticsearch. This means developers can build search, retrieval augmented generation (RAG), and AI applications that work across global datasets without stitching pipelines together.

Create a semantic_text field, index your data, and start searching. All Elastic Cloud trials have access to Elastic Inference Service. Try it now on Elastic Cloud Serverless or Elastic Cloud Hosted, or use EIS via Cloud Connect with your self-managed cluster.

The problem

Imagine you have two indices, index-a (source) and index-b (target), and you want to find all documents that exist in index-a but are missing from index-b.

A naive approach, querying both indices and comparing results in memory, won't scale. Elasticsearch is designed to handle millions of documents, and loading them all at once isn’t practical.

There are two scenarios:

1. IDs are stable: Both indices use the same _id for the same document (for example, emp_no as the document ID). This is the easy case.
2. IDs are generated: Documents were ingested through different pipelines that assigned random or sequential IDs. You can't compare by _id; you need to match on content.

Let's walk through both.

Step 0 — A lighter CLI for Elasticsearch

All the examples in this post use escli, a small l Rust command line interface (CLI) that wraps the Elasticsearch REST API. It reads your cluster URL and credentials from environment variables, so you don’t have to repeat authentication headers on every command.

To see why that matters, here's a typical _search call with raw curl:

curl -X GET \
  -H "Authorization: ApiKey $ELASTIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":{"term":{"user.id":"kimchy"}}}' \
  "$ELASTICSEARCH_URL/my-index-000001/_search"

With escli, the same request becomes:

./escli search --index my-index-000001 <<< '{"query":{"term":{"user.id":"kimchy"}}}'

The credentials live in a .env file that escli sources automatically — no -H "Authorization: ..." on every call, no risk of leaking secrets in shell history. The request body is passed via stdin (<<<), which makes it easy to pipe in multi-line JSON built dynamically with jq.

Step 1 — Count documents in both indices

Before doing a full scan, get a quick count of each index. If the counts match, the indices are likely in sync, and there’s no need to scan at all.

./escli count --index index-a
./escli count --index index-b

The _count API returns:

{ "count": 1000000 }

f the counts differ, proceed to the full comparison.

Step 2 — When IDs mean something: Use op_type=create

If both indices use the same _id for the same document, for example, because you indexed documents using a functional business key like emp_no rather than a generated UUID, you can find and fix missing documents in a single _reindex call.

Why functional IDs matter

Using a meaningful field as _id (instead of a random UUID) is a best practice when the data has a natural key. It means:

• The same document always gets the same _id, regardless of which pipeline ingested it.
• You can easily update or delete documents by ID.
• You can use op_type=create to skip documents that already exist in the target.
• No client-side scanning or comparison is needed.

The op_type=create trick

_reindex with op_type=create tries to create each document from the source in the target. If a document with the same _id already exists, Elasticsearch reports it as a version_conflict and moves on. It doesn’t overwrite the existing document. Setting conflicts=proceed tells the API to continue instead of aborting on the first conflict.

./escli reindex <<< '{
  "source": { "index": "index-a" },
  "dest":   { "index": "index-b", "op_type": "create" },
  "conflicts": "proceed"
}'

The response tells you exactly what happened:

{
  "total": 1000000,
  "created": 49594,
  "version_conflicts": 950406,
  "failures": []
}

• created: Documents that were missing from index-b and have now been added.
• version_conflicts: Documents that already existed in index-b and were left untouched.

No scanning, no client-side comparison, no intermediate file. Everything happens server-side in about six seconds on a 1M-document dataset.

Step 3 — When IDs are not stable: Business-key comparison

Sometimes you can't rely on _id. A document pipeline that generates IDs at ingestion time will assign a different _id each time the same record is processed. If index-a and index-b were populated by two such pipelines, the same employee record might have _id: "abc123" in one index and _id: "xyz789" in the other, even though the underlying data is identical.

In this case, you need to match documents by content rather than by ID. The key is to identify a set of fields that together form a unique business key.

For an employee dataset, a reasonable business key is (first_name, last_name, birth_date). A document in index-a is "missing" from index-b if no document in index-b has the same combination of those three fields.

3a — Scan the source with PIT + search_after

Open a point in time (PIT) on the source index to get a consistent snapshot, and then paginate through it, fetching only the business-key fields:

./escli open_point_in_time index-a 5m
# → { "id": "46ToAwMDaWR..." }

./escli search <<< '{
  "size": 10000,
  "_source": ["first_name", "last_name", "birth_date"],
  "pit": { "id": "46ToAwMDaWR...", "keep_alive": "5m" },
  "sort": [{ "_shard_doc": "asc" }]
}'

The sort key _shard_doc is the most efficient sort for full-index pagination: it uses the internal Lucene document order with no overhead. Repeat with search_after until the response contains zero hits. Always close the PIT when done:

./escli close_point_in_time <<< '{"id": "46ToAwMDaWR..."}'

3b — Check each page against the target via _msearch

For each page of source documents, build one _msearch request with one subquery per document. Each subquery uses a bool/must on the three business-key fields and requests size: 0; we only need to know whether a match exists, we don’t need to retrieve the document itself.

./escli msearch << 'EOF'
{"index": "index-b"}
{"size":0,"query":{"bool":{"must":[{"term":{"first_name.keyword":"Alice1"}},{"term":{"last_name.keyword":"Smith"}},{"term":{"birth_date":"1985-03-12"}}]}}}
{"index": "index-b"}
{"size":0,"query":{"bool":{"must":[{"term":{"first_name.keyword":"Bob2"}},{"term":{"last_name.keyword":"Jones"}},{"term":{"birth_date":"1990-07-24"}}]}}}
EOF

The response contains one entry per subquery, in the same order:

{
  "responses": [
    { "hits": { "total": { "value": 1 } } },
    { "hits": { "total": { "value": 0 } } }
  ]
}

total.value == 0 means no document in index-b matches that business key; the document is missing. Collect the corresponding _id from the source page.

Note on .keyword subfields: term queries require exact (keyword) matching. The first_name and last_name fields must have a .keyword subfield in the index mapping. The demo's mapping.json includes this.

3c — Speed it up with split-by-date

If the business key includes a date field, you can partition the source into date slices and run each slice as an independent job. Each slice opens its own PIT with a range filter on birth_date, runs its own msearch loop, and writes its results to a separate file. The parent script launches all slices in parallel and aggregates the results when they’re all done.

But depending on your use case, you might want to partition by a different field; for example, if you have a team field, you could run one slice per team. The key is to find a field that allows you to split the data into reasonably even chunks that can be processed in parallel.

[compare] Launching 5 slices in parallel...

  → Slice 1: 1960-01-01 → 1969-12-31 ✅ — 244408 checked, 12207 missing
  → Slice 2: 1970-01-01 → 1979-12-31 ✅ — 243624 checked, 12212 missing
  → Slice 3: 1980-01-01 → 1989-12-31 ✅ — 243551 checked, 11921 missing
  → Slice 4: 1990-01-01 → 1999-12-31 ✅ — 243895 checked, 11991 missing
  → Slice 5: 2000-01-01 → 2009-12-31 ✅ — 24522 checked, 1263 missing

Performance on a 1M dataset

To validate the approaches, the demo generates 1,000,000 documents in index-a and deliberately skips ~5% in index-b (49,594 missing documents), and then runs the full compare → reindex cycle.

Results on a MacBook M3 Pro:

Comparison (compare-indices.sh):

The op_type=create approach is fastest because everything is server-side and requires no client-side scanning. The split-by-date strategy cuts the business-key duration from 1m 38s down to 36s through parallelism: not bad for a comparison across two 1M-document indices.

Decision tree

Are _id values stable between both indices?
├── Yes → _reindex with op_type=create          (6s, server-side)
└── No  → Do you have a reliable business key?
          ├── Yes, simple scan is fast enough → business-key   (1m 42s)
          └── Yes, and you need more speed    → split-by-date  (36s, parallel)

Conclusion

Elasticsearch doesn't offer a native index diff command, but the right strategy depends on your data model:

• Use functional _ids (a natural business key like emp_no) whenever possible. It unlocks the simplest and fastest approach: _reindex with op_type=create finds and fills gaps in one server-side call.
• When IDs are unstable, match by business key using PIT + _msearch. Partition by a field and run slices in parallel to recover most of the performance. If you find yourself doing this regularly, consider computing a hash of your business key fields and using it as _id at ingestion time. You get the best of both worlds: stable IDs and efficient lookups.

The complete demo, including dataset generation, comparison scripts, and reindex scripts, is available at https://github.com/dadoonet/blog-compare-indices/.

Elastic Workflows is a built-in automation engine inside Kibana that lets you define multistep processes using a simple YAML configuration. Each workflow can be triggered on a schedule or event or as a tool in Elastic Agent Builder, and each step can call Kibana APIs, query Elasticsearch, or transform data.

We’ll use dashboard view counts as a concrete example, but the same pattern applies to any metric exposed through the Kibana saved objects API.

Prerequisites

• Elastic Cloud or self-managed cluster running 9.3
• Workflows enabled (Advanced settings)

Step 1: Explore the raw data in Dev Tools

Before building anything, let's understand what data we have. Kibana stores most of its configuration and metadata as saved objects in a dedicated internal index. One of the things Kibana tracks this way is dashboard view counts, using a special saved object type called usage counters. You can query them directly from Dev Tools:

GET kbn:/api/saved_objects/_find?type=usage-counter&filter=usage-counter.attributes.domainId:"dashboard"%20and%20usage-counter.attributes.counterType:"viewed"&per_page=10000

The response looks like this:

{
  "page": 1,
  "per_page": 10000,
  "total": 1,
  "saved_objects": [
    {
      "type": "usage-counter",
      "id": "dashboard:346f3c64-ebca-484d-9d57-ec600067d596:viewed:server:20260310",
      "attributes": {
        "domainId": "dashboard",
        "counterName": "346f3c64-ebca-484d-9d57-ec600067d596",
        "counterType": "viewed",
        "source": "server",
        "count": 1
      },
      ...
    }
  ]

The counterName field is the dashboard ID, and count is the cumulative view count for that dashboard on that specific day. Kibana creates one counter object per dashboard per day; you can see the date suffix in the object ID (...viewed:server:20260310). The count grows throughout the day as users open the dashboard.

Rather than replicating this daily-document model in our index, we’ll create one document per workflow execution. Each document records how many views that dashboard had accumulated for the day at the moment of capture.

Step 2: Create the destination index

We need an index to store our dashboard view snapshots. The following command creates it with explicit mappings so we can aggregate and visualize later. Run this in Dev Tools:

PUT dashboard-views
{
  "mappings": {
    "properties": {
      "captured_at": {
        "type": "date"
      },
      "dashboard_id": {
        "type": "keyword"
      },
      "dashboard_name": {
        "type": "keyword"
      },
      "view_count": {
        "type": "integer"
      }
    }
  }
}

Using keyword mappings for IDs and names allows aggregations. Using integer for view_count is a safe default, since Kibana resets the counter daily, reaching the 32-bit limit (more than 2 billion views in a single day) isn’t a realistic concern. It still supports numeric operations, like max, avg, and min among others.

Step 3: Create the workflow

Go to Stack Management > Workflows > New Workflow, and paste the following workflow YAML configuration:

name: dashboard-views-ingestion
triggers:
  - type: scheduled
    with:
      every: 30m

steps:
  - name: fetch_dashboard_views
    type: kibana.request
    with:
      method: GET
      path: >-
        /api/saved_objects/_find?type=usage-counter&per_page=10000&filter=usage-counter.attributes.domainId:"dashboard"%20and%20usage-counter.attributes.counterType:"viewed"

  - name: index_each_dashboard
    type: foreach
    foreach: "{{ steps.fetch_dashboard_views.output.saved_objects }}"
    steps:
      - name: fetch_dashboard_name
        type: kibana.request
        with:
          method: GET
          path: /api/saved_objects/dashboard/{{ foreach.item.attributes.counterName }}
        on-failure:
          continue: true

      - name: index_doc
        type: elasticsearch.request
        with:
          method: POST
          path: /dashboard-views/_doc
          body:
            dashboard_id: "{{ foreach.item.attributes.counterName }}"
            dashboard_name: "{{ steps.fetch_dashboard_name.output.attributes.title }}"
            view_count: "${{ foreach.item.attributes.count | plus: 0 }}"
            captured_at: "{{ execution.startedAt | date: '%Y-%m-%dT%H:%M:%SZ' }}"

In the next section, let's break down the workflow step by step.

How the workflow works

Triggers

The workflow runs on a scheduled trigger every 30 minutes. This gives us time-series data without hammering the API.

fetch_dashboard_views

Uses kibana.request to call the Kibana saved objects API. No authentication setup is needed: The workflow engine automatically attaches the correct headers based on the execution context.

index_each_dashboard (foreach)

Iterates over the saved_objects array returned by the previous step. The current item in each iteration is available as foreach.item. Inside the loop, we run two nested steps for each dashboard.

1. fetch_dashboard_name:

Resolves the human-readable dashboard title by calling GET /api/saved_objects/dashboard/{id}. We add on-failure: continue: true so that if a dashboard was deleted but still has view counters, the loop continues instead of failing the whole execution.

2. index_doc:

Indexes each document using POST /dashboard-views/_doc (without an explicit ID), which lets Elasticsearch auto-generate IDs. This creates a new document on every run, building a history of view counts over time rather than overwriting the previous snapshot.

Two things worth noting:

• The captured_at field uses the date filter to format the timestamp as ISO 8601. Without it, the value comes out as a JavaScript date string, like Tue Mar 10 2026 05:03:47 GMT+0000, which Elasticsearch won't map as a date.
• The view_count uses ${{ }} syntax with | plus: 0 to preserve the numeric type. Using {{ }} would render it as a string, which would prevent math operations in the dashboard.

The UI allows you to nicely debug each of the workflow steps.

Step 4: Build the stats dashboard

Once the workflow has run a few times and data is collected, create a new dashboard in Kibana using the dashboard-views data view.

Some panels to start with:

• Top dashboards by views: Use a Bar chart with dashboard_name on the X axis and last_value(view_count) on the Y axis. This shows the current daily view count per dashboard.
• Views over time: Use a Line chart with captured_at on the X axis and last_value(view_count) on the Y axis, broken down by dashboard_name. Since each run appends a new document, use last value to get the peak count per time bucket rather than summing duplicates.
• Current snapshot: Use a Data table with the latest captured_at to show the most recent view counts across all dashboards.

Since each workflow creates a new document, you can filter by time range to analyze activity in specific periods, compare week over week, or build alerts when a dashboard drops below a view threshold.

Conclusion

Elastic Workflows is a good fit for this kind of periodic data collection because both the source (Kibana API) and the destination (Elasticsearch) are native, which means zero credential management. The workflow engine handles authentication automatically for kibana.request and elasticsearch.request steps, so the only thing you write is the logic.

Resources

• Elastic Workflows
• Kibana API

Elasticsearch was rejecting late-arriving metrics. Those rejections caused the pipeline to fall behind, resulting in more late data, which triggered even more rejections. Eventually, the pipeline stalled completely.

We had to restore from snapshot, reindex the data, and redesign the ingestion pipeline to recover.

The root cause wasn't index lifecycle management (ILM) itself. It was time series data streams (TSDS) and how they enforce time‑bound backing indices.

TSDS can reduce storage requirements for metrics by 40–70%, but the architectural changes that make TSDS efficient also alter how indices behave over time. Those changes matter when designing ILM policies or when your ingestion pipelines may produce late‑arriving data.

TL;DR

When using TSDS:

• Backing indices only accept documents within a specific time window.
• If late data arrives after an index moves to cold or frozen, Elasticsearch rejects those documents or routes them to the failure store, if configured.

Design rule:

warm_min_age > rollover_max_age + maximum_expected_lateness

What is a time series data stream?

A time series data stream (TSDS) is a specialized data stream optimized for metrics data. Data is routed so that related documents are located within the same shards, optimizing them for query and retrieval. Here’s how Elasticsearch does it:

Each document contains:

• A timestamp.
• Dimension fields identifying the time series.
• Metric fields representing measured values.

Examples include:

• CPU usage per host.
• Request latency per service.
• Temperature readings per sensor.

Dimensions identify what we want to measure, while metrics represent values that change over time.

Dimensions

Dimensions describe the measured entity.

Examples:

host.name
service.name
container.id

We define them in mappings with:

time_series_dimension: true

Metrics

Metrics represent numeric values and are defined using:

time_series_metric

Common metric types:

• Gauge: Values that rise and fall.
• Counter: Values that increase until reset.

Elastic Agent primarily collects metrics and logs data, so even if you haven’t enabled any TSDS indices by hand, you may still have them in your cluster.

The _tsid field

Elasticsearch internally generates a _tsid value from dimension fields. This allows documents with identical dimensions to be routed to the same shard, improving:

• Compression.
• Query locality.
• Aggregation performance.

The key difference: Time‑bound backing indices

Traditional data streams always write to the most recent backing index, called the write index, but TSDS behaves differently.

Each TSDS backing index has a defined time window and only accepts documents with @timestamp values that fall in that window:

GET _data_stream/my-metrics-data-stream
{
  "index_mode": "time_series",
  "time_series": {
    "temporal_ranges": [
      {
        "start": "2026-01-15T14:35:50.000Z",
        "end": "2026-03-16T11:34:40.000Z"
      }
    ]
  }
}

When a document is indexed, Elasticsearch routes it to the backing index responsible for that timestamp, meaning that, unlike traditional indices, a TSDS may write to multiple backing indices simultaneously.

For example:

• Real‑time data → newest index.
• Late data → earlier index covering that time range.

Designing for late‑arriving data

Real ingestion pipelines rarely deliver metrics perfectly on time. Metrics can be delayed by network outages, backlogs along the way, batch ingestion, and loss of edge devices, which reconnect and start to catch up.

Traditional indices quietly absorb those delays. TSDS does not.

If a document's timestamp falls outside the range of writable backing indices, Elasticsearch rejects it, meaning your ILM policy must account for late data.

The critical constraint

Backing indices must remain writable long enough to accept delayed data.

In practical terms:

time_until_readonly > maximum_expected_lateness

Because ILM measures ages from rollover, the operational rule becomes:

warm_or_cold_min_age > rollover_max_age + maximum_expected_lateness

For example, if metrics may arrive up to six hours late, indices must remain writable at least six hours after rollover.

Failing to account for this constraint was exactly what caused the ingestion failure described earlier. Late-arriving data was directed to an earlier index, which was already in the cold tier and therefore write-blocked.

Handling rejected documents

When TSDS rejects a document, Elasticsearch returns an error, indicating that the timestamp doesn’t fall within the range of writable indices. How your ingestion pipeline handles that error determines whether you lose data or stall ingestion.

The primary mechanism for handling rejected documents is the failure store.

Failure store (recommended in Elasticsearch 9.1+)

Elasticsearch 9.1 introduced the failure store, which automatically captures rejected documents. Instead of returning errors to clients, Elasticsearch writes failed documents to a dedicated failure index inside the data stream.

You can inspect failures using:

GET metrics-myapp::failures/_search

Using the failure store prevents ingestion pipelines from choking on rejection errors while preserving failed data for analysis or reindexing.

Monitoring for rejection issues

Late‑arrival problems usually appear first as ingestion anomalies. You may notice them first as:

• Sudden drops in indexing rate.
• Spikes in rejected documents.
• A growing number of failure store entries.
• Mismatches between pipeline input and output counts.

Alerting on these signals allows operators to detect issues before pipelines stall. Workflows, machine learning jobs, and other mechanisms can be used to automate detection and notification.

Migration checklist for TSDS + ILM

If you're migrating a metrics cluster to TSDS, introducing ILM tiering, or upgrading to an Elasticsearch version where metrics are TSDS by default, review these items first.

1. Measure ingestion latency

Before changing ILM policies, determine:

• Normal ingestion delay.
• Worst-case delay during incidents.
• Delays caused by batch pipelines.

Your ILM design must accommodate the maximum realistic delay.

2. Verify index time windows

Inspect your TSDS backing indices:

GET _data_stream/

Look for:

• time_series.start_time
• time_series.end_time

These bounds determine which indices can accept documents. Understanding these windows can help you determine how late data can be before it’s rejected.

3. Size the hot tier for late arrivals

Ensure backing indices remain writable long enough for delayed data.

Operational rule:

• warm_min_age > rollover_max_age + maximum_expected_lateness

Remember, indices must remain writable for at least six hours if metrics may arrive six hours late.

4. Decide how to handle rejected documents

Choose a strategy before enabling TSDS:

• Failure store (recommended in Elasticsearch 9.1+).
• Logstash dead letter queue.
• Fallback index for late arrivals.
• Accepting limited data loss.

5. Monitor ingestion health

Add alerts for:

• Indexing rate drops.
• Rejected documents.
• Failure store growth.
• Pipeline input/output mismatches.

Late data issues often appear first as ingestion anomalies.

Summary

Time series data streams provide major storage and performance improvements for metrics workloads, but they introduce an important architectural change: Backing indices are time‑bound, which affects how ILM behaves.

When using TSDS:

• Indices must remain writable long enough to accept delayed data.
• Ingestion pipelines should handle rejected documents safely.

The key rule to remember is:

warm_min_age > rollover_max_age + maximum_expected_lateness

If you design ILM policies around that constraint, TSDS works extremely well for metrics workloads.

Ignore it, though, and your ingestion pipeline may discover those time boundaries the hard way.

Your first query

Start by defining a plain old CLR object (POCO) that maps to your Elasticsearch index. Property names are resolved to ES|QL column names through standard System.Text.Json attributes, like [JsonPropertyName], or through a configured JsonNamingPolicy. The same source serialization rules that apply across the rest of the client apply here as well.

using System.Text.Json.Serialization;

public class Product
{
    [JsonPropertyName("product_id")]
    public string Id { get; set; }

    public string Name { get; set; }

    public string Brand { get; set; }

    [JsonPropertyName("price_usd")]
    public double Price { get; set; }

    [JsonPropertyName("in_stock")]
    public bool InStock { get; set; }
}

With the type in place, a query looks like this:

var minPrice = 100.0;
var brand = "TechCorp";

await foreach (var product in client.Esql.QueryAsync(q => q
    .From("products")
    .Where(p => p.InStock && p.Price >= minPrice && p.Brand == brand)
    .OrderByDescending(p => p.Price)
    .Take(10)))
{
    Console.WriteLine($"{product.Name}: ${product.Price}");
}

The provider translates this into the following ES|QL:

FROM products
| WHERE (in_stock == true AND price_usd >= ?minPrice AND brand == ?brand)
| SORT price_usd DESC
| LIMIT 10

A few details to note:

• Property name resolution: p.Price becomes price_usd because of the [JsonPropertyName] attribute, and p.Brand becomes brand following the default camelCase naming policy.
• Parameter capturing: The C# variables minPrice and brand are captured as named parameters (?minPrice, ?brand). They’re sent separately from the query string in the JSON payload, which prevents injection and enables server-side query plan caching.
• Streaming: QueryAsync<T> returns IAsyncEnumerable<T>. Rows are materialized one at a time as they arrive from Elasticsearch.

You can also inspect the generated query and its parameters without executing it:

var query = client.Esql.CreateQuery()
    .Where(p => p.InStock && p.Price >= minPrice && p.Brand == brand)
    .OrderByDescending(p => p.Price)
    .Take(10);

Console.WriteLine(query.ToEsqlString());
// FROM products | WHERE (in_stock == true AND price_usd >= 100) | SORT price_usd DESC | LIMIT 10

Console.WriteLine(query.ToEsqlString(inlineParameters: false));
// FROM products | WHERE (in_stock == true AND price_usd >= ?minPrice AND brand == ?brand) | SORT price_usd DESC | LIMIT 10

var parameters = query.GetParameters();
// { "minPrice": 100.0, "brand": "TechCorp" }

How does this work? A quick LINQ refresher

The mechanism that makes LINQ providers possible is the distinction between IEnumerable<T> and IQueryable<T>.

When you call .Where(p => p.Price > 100) on an IEnumerable<T>, the lambda compiles to a Func<Product, bool>, a regular delegate that the runtime executes in-process. This is LINQ-to-Objects.

When you call the same method on an IQueryable<T>, the C# compiler wraps the lambda in an Expression<Func<Product, bool>> instead. This is a data structure that represents the structure of the code rather than its executable form. The expression tree can be inspected, analyzed, and translated into another language at runtime.

// IEnumerable: the lambda is a compiled delegate
IEnumerable local = products.Where(p => p.Price > 100);

// IQueryable: the lambda is an expression tree, a data structure
IQueryable remote = queryable.Where(p => p.Price > 100);

The IQueryProvider interface is the extension point. Any provider can implement CreateQuery<T> and Execute<T> to translate these expression trees into a target language. Entity Framework uses this to emit SQL. The LINQ to ES|QL provider uses it to emit ES|QL.

The expression tree for the query above looks like this:

Expression tree for the example query.

The tree is nested inside out: Take wraps OrderByDescending, which wraps Where, which wraps From, which wraps the root EsqlQueryable<Product> constant. The Where predicate is itself a subtree of BinaryExpression nodes for the &&, >=, and == operators, with MemberExpression leaves for property accesses and closure captures for the minPrice and brand variables. This is the data structure that the provider walks to produce the final ES|QL.

Under the hood: The translation pipeline

The path from a LINQ expression to query results follows a six-stage pipeline:

Translation pipeline overview.

1. Expression tree capture

When you chain .Where(), .OrderBy(), .Take() and other operators on an IQueryable<T>, the standard LINQ infrastructure builds an expression tree. EsqlQueryable<T> implements IQueryable<T> and delegates to EsqlQueryProvider.

2. Translation

When the query is executed (by enumerating, calling ToList(), or using await foreach), the EsqlExpressionVisitor walks the expression tree inside out. It dispatches each LINQ method call to a specialized visitor:

During translation, C# variables referenced in expressions are captured as named parameters.

3. Query model

The visitors don’t produce strings directly. Instead, they produce QueryCommand objects, an immutable intermediate representation. A FromCommand, a WhereCommand, a SortCommand, and a LimitCommand, each representing one ES|QL processing command. These are collected into an EsqlQuery model.

Query model and command pattern.

This intermediate model is decoupled from both the expression tree and the output format. It can be inspected, intercepted (via IEsqlQueryInterceptor), or modified before formatting.

4. Formatting

EsqlFormatter visits each QueryCommand in order and produces the final ES|QL string. Each command becomes one line, separated by the pipe (|) operator that ES|QL uses to chain processing commands. Identifiers containing special characters are automatically escaped with backticks.

5. Execution

The formatted ES|QL string and captured parameters are sent to Elasticsearch’s /_query endpoint as a JSON payload. The IEsqlQueryExecutor interface abstracts the transport layer, which is where the layered package architecture comes into play.

6. Materialization

EsqlResponseReader streams the JSON response without buffering the entire result set into memory. A ColumnLayout tree, precomputed once per query, maps flat ES|QL column names (like address.street, address.city) to nested POCO properties. Each row is assembled into a T instance and yielded one at a time via IEnumerable<T> or IAsyncEnumerable<T>.

The layered architecture

The LINQ to ES|QL functionality is split across three packages:

Package architecture.
Elastic.Esql is the pure translation engine. It has zero HTTP dependencies and contains the expression visitors, query model, formatter, and response reader. You can use it stand alone to build and inspect ES|QL queries without an Elasticsearch connection, which is useful for testing, query logging, or building your own execution layer.

// Translation-only: no Elasticsearch connection needed
var provider = new EsqlQueryProvider();
var query = new EsqlQueryable(provider)
    .From("products")
    .Where(p => p.InStock)
    .OrderByDescending(p => p.Price);

Console.WriteLine(query.ToEsqlString());
// FROM products | WHERE in_stock == true | SORT price_usd DESC

Elastic.Clients.Esql is a lightweight stand-alone ES|QL client. It adds HTTP execution on top of Elastic.Esql via Elastic.Transport. If your application only needs ES|QL and none of the other Elasticsearch APIs, this is the minimal dependency option.

Elastic.Clients.Elasticsearch is the full Elasticsearch .NET client. It also builds on Elastic.Esql and exposes the LINQ provider through the client.Esql namespace. This is the recommended entry point for most applications.

Both execution-layer packages provide their own implementation of IEsqlQueryExecutor, the strategy interface that bridges translation and transport.

All three packages are compatible with Native AOT when used with a source-generated JsonSerializerContext. For the full client, see the Native AOT documentation.

Beyond the basics

The example above covered filtering, sorting, and pagination. The provider supports a broader set of operations.

Aggregations

GroupBy, combined with aggregate functions in Select, translates to ES|QL STATS ... BY:

var stats = client.Esql.Query(q => q
    .GroupBy(p => p.Brand)
    .Select(g => new
    {
        Brand = g.Key,
        Count = g.Count(),
        AvgPrice = g.Average(p => p.Price),
        MaxPrice = g.Max(p => p.Price)
    }));

// -> FROM products | STATS COUNT(*), AVG(price_usd), MAX(price_usd) BY brand

Projections

Select, with anonymous types generates EVAL, KEEP, and RENAME commands:

var query = client.Esql.CreateQuery()
    .Select(p => new { ProductName = p.Name, p.Price, p.InStock });

// -> FROM products | KEEP name, price_usd, in_stock | RENAME name AS ProductName

Rich function library

Over 80 ES|QL functions are available through the EsqlFunctions class, covering date/time, string, math, IP, pattern matching, and scoring. Standard Math.* and string.* methods are also translated:

.Where(p => p.Name.Contains("Pro"))       // -> WHERE name LIKE "*Pro*"
.Where(p => EsqlFunctions.CidrMatch(      // -> WHERE CIDR_MATCH(ip, "10.0.0.0/8")
    p.IpAddress, "10.0.0.0/8"))

LOOKUP JOIN

Cross-index lookups translate to ES|QL LOOKUP JOIN:

var enriched = client.Esql.Query(q => q
    .LookupJoin(
        "category-lookup-index",
        product => product.Id,
        category => category.CategoryId,
        (product, category) => new { product.Name, category!.CategoryLabel }));

Raw ES|QL escape hatch

For ES|QL features not yet covered by the LINQ provider, you can append raw fragments:

var results = client.Esql.Query(q => q
    .Where(p => p.InStock)
    .RawEsql("| EVAL discounted = price_usd * 0.9"));

Server-side async queries

For long-running queries, submit them for background processing on the server:

await using var asyncQuery = await client.Esql.SubmitAsyncQueryAsync(
    q => q.Where(p => p.InStock),
    asyncQueryOptions: new EsqlAsyncQueryOptions
    {
        WaitForCompletionTimeout = TimeSpan.FromSeconds(5),
        KeepAlive = TimeSpan.FromMinutes(10)
    });

await asyncQuery.WaitForCompletionAsync();
await foreach (var product in asyncQuery.AsAsyncEnumerable())
    Console.WriteLine(product.Name);

Server-side async queries are especially useful for long-running analytical queries / large dataset processing that might exceed typical timeout thresholds, or in timeout-sensitive environments with load balancers, API gateways, or proxies that enforce strict HTTP timeouts. Async queries avoid connection drops by decoupling submission from result retrieval.

Getting started

LINQ to ES|QL is available starting from:

• Elastic.Clients.Elasticsearch v9.3.4 (9.x branch)
• Elastic.Clients.Elasticsearch v8.19.18 (8.x branch)

Install from NuGet:

dotnet add package Elastic.Clients.Elasticsearch

The entry points are on client.Esql:

For the full feature reference, including query options, multifield access, nested objects, and multivalue field handling, see the LINQ to ES|QL documentation.

Conclusion

LINQ to ES|QL brings the full expressiveness of C# LINQ to Elasticsearch's ES|QL query language, letting you write strongly typed, composable queries without handcrafting query strings. With automatic parameter capturing, streaming materialization, and a layered package architecture that scales from stand-alone translation to the full Elasticsearch client, it fits naturally into .NET applications of any size. Install the latest client, point your LINQ expressions at an index, and let the provider handle the rest.

If judgment lists answer the question, “How good is my ranking?,” Learning To Rank (LTR) answers, “How do I systematically make it better?”

In this article, we take the next step: using those judgment lists to train an LTR model using XGBoost, Eland, and Elasticsearch. We’ll focus on understanding the process rather than on implementation details. For the complete code, refer to the companion notebook.

What is LTR?

LTR uses machine learning (ML) to build a ranking function for your search engine. Instead of manually tuning query weights, you provide examples of proper rankings (your judgment list) and let the model learn what makes documents relevant. In Elasticsearch, LTR works as a second-stage reranker following retrieval of documents from Elasticsearch:

• First stage: A standard query (BM25, vector, or hybrid) retrieves candidate documents quickly.
• Second stage: The LTR model reranks the top results using multiple signals it learned to combine.

For a deeper introduction, see Introducing Learning To Rank (LTR) in Elasticsearch.

The journey from judgment list to model

A judgment list tells us which documents should rank highly for a given query. But the model cannot learn directly from document IDs. It needs numerical signals that explain why certain documents are potentially relevant.

The process works like this:

1. Start with judgments. Query-document pairs with relevance grades, so you define that doc1 is a good match for “DiCaprio performance” search terms.
2. Extract features. For each query-document pair, compute numerical signals, some about the document alone (for example, popularity), and others about how the query and document interact (for example, BM25 score).
3. Train the model. The model learns which feature patterns predict high grades.
4. Deploy. Deploy the trained model to your Elasticsearch cluster.
5. Query. Use the model to rerank search results.

The key insight is that features must capture what your judgments are measuring. If your judgment list rewards popular thriller movies but your features only include text-matching scores, the model has no way to learn what makes those documents relevant.

What are features?

Features are numerical values that describe a query-document pair. In Elasticsearch, we define features using queries that return scores. There are three types:

• Query-document features measure how well a query matches a document. Eland provides the QueryFeatureExtractor utility to define these features, which computes the BM25 relevance score for each query-document pair:

QueryFeatureExtractor(
    feature_name="title_bm25",
    query={"match": {"title": "{{query}}"}}
)

This extracts the BM25 score from the title field for each document relative to the query.

• Document features are properties of the document that don’t depend on the query. You can extract these using script_score or function_score:

QueryFeatureExtractor(
    feature_name="popularity",
    query={
        "script_score": {
            "query": {"exists": {"field": "popularity"}},
            "script": {"source": "return doc['popularity'].value;"}
        }
    }
)

• Query features describe the query itself, like the number of terms. These are less common but can help the model handle different query types.

Designing your feature set

Choosing features isn’t random. Each feature should capture a signal that might explain why users prefer certain documents. Let's look at the features from the LTR notebook and understand the reasoning:

Notice the strategy here:

• Multiple signals for the same concept. We have both title_bm25 (lenient) and title_all_terms_bm25 (strict). The lenient version scores any document where at least one query term matches the title, and the strict version requires all the terms to be present. For short queries, the lenient match might be enough; whereas for longer, more specific queries, strict matching might be more important. The model can learn when to rely on each.
• Text features plus quality features. Text matching alone can return irrelevant documents that happen to contain the right words. The popularity feature lets the model boost well-known, quality content when text scores are similar.
• Coverage for different query types. Some queries target titles ("star wars"), and others target actors ("dicaprio movies"). Having features for both means that the model can handle diverse searches.

When designing your own features, ask yourself, "What signals would a human use to decide if this document is relevant?" Those are your candidate features.

Building the training dataset

Once features are defined, we extract them for every query-document pair in our judgment list. The result is a training dataset where each row contains:

• The query identifier.
• The document identifier.
• The relevance grade (from our judgment list).
• All feature values.

Here’s a simplified example:

A few things to notice:

NaN values are normal. When a query doesn’t match a field, the feature returns no score. The movie Star Wars has a high title_bm25 but no actors_bm25 because the query "star wars" doesn’t match any actor names.

Queries are grouped during training. The query_id column tells the model which documents to compare against each other. For "star wars", it learns that document 11 (grade 4) should rank above document 278427 (grade 1).

But here’s the important part: The model doesn’t memorize these specific queries. Instead, it learns general patterns, like "documents with high title_bm25 AND high popularity tend to have high grades." When presented with a new query, the model applies these learned patterns to rank the results.

Features must explain grade differences. Look at qid:1: The grade 4 document has a higher title_bm25 and higher popularity than the grade 1 document. These patterns are what the model learns.

Training the LTR model

With the training dataset prepared, we train an XGBoost model with a ranking objective. The model builds decision trees that learn patterns like:

• "If title_bm25 > 10 and popularity > 50, predict high relevance."
• "If title_bm25 is missing but actors_bm25 > 12, still predict moderate relevance."

Here's how the training process works in practice:

from xgboost import XGBRanker
from sklearn.model_selection import GroupShuffleSplit

# Create the ranker model:
ranker = XGBRanker(
    objective="rank:ndcg",
    eval_metric=["ndcg@10"],
    early_stopping_rounds=20,
)

# Shaping training and eval data in the expected format.
X = judgments_with_features[ltr_config.feature_names]
y = judgments_with_features["grade"]
groups = judgments_with_features["query_id"]

# Split the dataset in two parts respectively used for training and evaluation of the model.
group_preserving_splitter = GroupShuffleSplit(n_splits=1, train_size=0.7).split(
    X, y, groups
)
train_idx, eval_idx = next(group_preserving_splitter)

train_features, eval_features = X.loc[train_idx], X.loc[eval_idx]
train_target, eval_target = y.loc[train_idx], y.loc[eval_idx]
train_query_groups, eval_query_groups = groups.loc[train_idx], groups.loc[eval_idx]

# Training the model
ranker.fit(
    X=train_features,
    y=train_target,
    group=train_query_groups.value_counts().sort_index().values,
    eval_set=[(eval_features, eval_target)],
    eval_group=[eval_query_groups.value_counts().sort_index().values],
    verbose=True,
)

During training, the model tries different combinations of these rules and measures how well the resulting rankings match your judgment grades. It uses a metric called Normalized Discounted Cumulative Gain (NDCG) to score itself. A perfect NDCG of 1.0 means that the model's ranking exactly matches your judgments. Lower scores mean that some relevant documents are ranking below where they should be.

The training also uses a technique called early stopping. If the model's score stops improving for several rounds, training halts automatically. This prevents the model from memorizing the training data too closely, which would hurt its ability to generalize to new queries.

The companion notebook contains the complete training code.

Understanding what your LTR model learned

After training, XGBoost can show you which features the model relied on most. You can generate a feature importance chart using XGBoost's built-in visualization:

from xgboost import plot_importance

plot_importance(ranker, importance_type="weight")

The importance_type="weight" parameter shows how often each feature was used in tree splits. Here’s the resulting chart:

The F score counts how many times each feature was used to make split decisions across all trees in the model. Higher values mean that the model relied on that feature more often.

In this example:

• popularity (2178): The most important feature. The model frequently uses popularity to separate relevant from nonrelevant documents.
• title_bm25 (1642): Second-most important. Title matches matter a lot for movie searches.
• actors_bm25 (565): Moderately important. This is useful for queries that mention actors.
• title_all_terms_bm25 (211): Occasionally useful. The stricter matching helps for some queries.
• actors_all_terms_bm25 (63): Rarely used. The model found this feature less predictive.

This chart helps you iterate on your feature set. If a feature that you expected to be important shows near-zero importance, investigate why. Maybe the feature extraction is not working as intended, or maybe that signal doesn’t actually predict relevance in your judgment data.

Deploying and using the LTR model

Once trained, upload the model to Elasticsearch using Eland:

MLModel.import_ltr_model(
    es_client=es_client,
    model=ranker,
    model_id="ltr-model-xgboost",
    ltr_model_config=ltr_config,
    es_if_exists="replace",
)

Once uploaded, the model can be used as a rescorer retriever to be combined with other retrievers for multistage search pipelines:

GET movies/_search
{
  "retriever": {
    "rescorer": {
      "rescore": {
        "window_size": 50,
        "learning_to_rank": {
          "model_id": "ltr-model-xgboost",
          "params": {
            "query": "star wars"
          }
        }
      },
      "retriever": {
        "standard": {
          "query": {
            "multi_match": {
              "fields": ["title", "overview", "actors", "director", "tags", "characters"],
              "query": "star wars"
            }
          }
        }
      }
    }
  }
}

Response (simplified):

 "hits": {
    "total": {
      "value": 852,
      "relation": "eq"
    },
    "max_score": 25.165691,
    "hits": [
      {
        "_index": "movies",
        "_id": "11",
        "_score": 25.165691,
        "_source": {
          "title": "Star Wars"
        }
      },
      {
        "_index": "movies",
        "_id": "12180",
        "_score": 25.092865,
        "_source": {
          "title": "Star Wars: The Clone Wars"
        }
      },
      {
        "_index": "movies",
        "_id": "181812",
        "_score": 23.456198,
        "_source": {
          "title": "Star Wars: The Rise of Skywalker"
        }
      },
      {
        "_index": "movies",
        "_id": "140607",
        "_score": 23.320757,
        "_source": {
          "title": "Star Wars: The Force Awakens"
        }
      },
...

The first-stage query retrieves candidates using BM25. The LTR model then reranks the top 50 results using all the features it learned to weight.

For the sake of the example, the multi_match query alone would return some less relevant results on the first positions that LTR helped to fix:

{
  "hits": [
    {
      "_index": "movies",
      "_id": "11",
      "_score": 10.971989,
      "_source": {
        "title": "Star Wars"
      }
    },
    {
      "_index": "movies",
      "_id": "12180",
      "_score": 9.923633,
      "_source": {
        "title": "Star Wars: The Clone Wars"
      }
    },
    {
      "_index": "movies",
      "_id": "1022100",
      "_score": 8.9880295,
      "_source": {
        "title": "Andor: A Disney+ Day Special Look"
      }
    },
    {
      "_index": "movies",
      "_id": "278427",
      "_score": 8.845748,
      "_source": {
        "title": "Family Guy Presents: It's a Trap!"
      }
    },
    ...
  ]
}

Conclusion

The path from judgment lists to a working LTR model involves three key steps: designing features that capture relevance signals, building a training dataset that pairs those features with your judgment grades, and training a model that learns the patterns.

Our previous article becomes the starting point for this process. Your grades define what "relevant" means and how to measure it, and your features give the model the signals to predict it.

For the complete implementation with a dataset of 9,750 movies and 384,755 judgment rows, see the LTR notebook. For advanced use cases, like personalized search, see Personalized search with LTR.

Prerequisites

• Elasticsearch 8.15+ (for :: cast operator support; core ES|QL features available from 8.11)

Runtime fields versus ES|QL

Runtime fields were introduced in Elasticsearch 7.11 as a way to define fields at query time. Instead of reindexing data, you could write a Painless script that computes values on the fly:

PUT my-index/_mapping
{
  "runtime": {
    "full_address": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['address'].value + ':' + doc['port'].value)"
      }
    }
  }
}

This works, but comes with trade-offs:

• Painless scripting overhead: Every runtime field requires scripting knowledge, and the syntax is Java-like, not query-like.
• Performance cost: Runtime fields evaluate per document at query time. Elasticsearch classifies them as "expensive queries" that can be rejected by cluster settings.
• Isolated computation: Each runtime field computes independently. There’s no way to chain transforms or use the output of one field in another within the same query.

ES|QL changes the equation. It has its own execution engine (not translated to Query DSL), runs queries concurrently across nodes, and provides a complete toolkit for field computation: EVAL, GROK, DISSECT, type casting, and pipeline chaining.

Let's see how each runtime field pattern maps to ES|QL.

Setting up the example data

All the code snippets in this article can be executed in the Kibana Dev Tools console.

To follow along, create a sample index with data that exercises all five patterns. This simulates a server logs scenario with mixed field types, raw messages, and some intentional data quality issues:

PUT server-logs
{
  "mappings": {
    "properties": {
      "host": { "type": "keyword" },
      "port": { "type": "keyword" },
      "raw_message": { "type": "text" },
      "response_time": { "type": "keyword" },
      "status_code": { "type": "keyword" },
      "region": { "type": "keyword" }
    }
  }
}

Now index some sample documents:

POST _bulk
{ "index": { "_index": "server-logs" } }
{ "host": "web-01", "port": "8080", "raw_message": "2024-01-15 INFO user=alice action=login duration=230ms", "response_time": "145", "status_code": "200", "region": "us-east" }
{ "index": { "_index": "server-logs" } }
{ "host": "web-02", "port": "443", "raw_message": "2024-01-15 ERROR user=bob action=upload duration=1200ms", "response_time": "not_available", "status_code": "500", "region": "eu-west" }
{ "index": { "_index": "server-logs" } }
{ "host": "api-01", "port": "3000", "raw_message": "2024-01-15 WARN user=charlie action=query duration=890ms", "response_time": "890", "status_code": "200", "region": "us-east" }
{ "index": { "_index": "server-logs" } }
{ "host": "api-02", "port": "3000", "raw_message": "2024-01-16 INFO user=diana action=export duration=3400ms", "response_time": "3400", "status_code": "200", "region": "ap-south" }
{ "index": { "_index": "server-logs" } }
{ "host": "web-01", "port": "8080", "raw_message": "2024-01-16 ERROR user=eve action=login duration=50ms", "response_time": "50", "status_code": "401", "region": "US-EAST" }

Notice that response_time is stored as a keyword (a common real-world mistake), and the last document has "US-EAST" instead of "us-east" (a data quality issue we’ll fix later).

Pattern 1: Field concatenation

A common runtime field use case is combining two fields into one. For example, creating a host:port identifier.

The runtime field approach

You can define it inline at query time. Query-time approach avoids modifying the mapping, but you still need Painless scripting, scoping it to a single search request:

GET server-logs/_search
{
  "runtime_mappings": {
    "endpoint": {
      "type": "keyword",
      "script": {
        "source": "emit(doc['host'].value + ':' + doc['port'].value)"
      }
    }
  },
  "fields": ["endpoint"],
  "_source": false
}

The ES|QL approach

You can run ES|QL queries using the _query API endpoint:

POST _query
{
  "query": """
    FROM server-logs
    | EVAL endpoint = CONCAT(host, ":", port)
    | KEEP host, port, endpoint
    | LIMIT 1
  """
}

Response:

{
  "columns": [
    { "name": "host", "type": "keyword" },
    { "name": "port", "type": "keyword" },
    { "name": "endpoint", "type": "keyword" }
  ],
  "values": [
    ["web-01", "8080", "web-01:8080"]
  ]
}

CONCAT accepts two or more arguments and always returns a keyword.

Note: For brevity, the remaining ES|QL examples in this article show just the query. Wrap them in POST _query { "query": "..." } to run them in Kibana Dev Tools.

When to use

If you need endpoint to persist across all queries and be available in Kibana dashboards, use a mapping-level runtime field. If you need it for a single search request within Query DSL, use a query-time runtime field. If you need it for ad-hoc analysis or exploratory work, ES|QL is simpler.

Pattern 2: Data extraction from unstructured text

Extracting structured data from raw log messages is another classic runtime field pattern.

The runtime field approach

Painless uses Java's regex Matcher class:

GET server-logs/_search
{
  "runtime_mappings": {
    "log_user": {
      "type": "keyword",
      "script": {
        "source": "def matcher = /user=(\\w+)/.matcher(params._source['raw_message']); if (matcher.find()) { emit(matcher.group(1)); }"
      }
    }
  },
  "fields": ["log_user"],
  "_source": false
}

This is verbose. You need to know Painless regex syntax, handle the Matcher object, and call emit() correctly.

The ES|QL approach: GROK

ES|QL provides two purpose-built commands for text extraction. GROK uses regex-based patterns:

FROM server-logs
| GROK raw_message "%{WORD:timestamp_date} %{WORD:log_level} user=%{WORD:user} action=%{WORD:action} duration=%{WORD:duration}"
| KEEP user, log_level, action, duration

Response:

{
  "columns": [
    { "name": "user", "type": "keyword" },
    { "name": "log_level", "type": "keyword" },
    { "name": "action", "type": "keyword" },
    { "name": "duration", "type": "keyword" }
  ],
  "values": [
    ["alice", "INFO", "login", "230ms"], ...
  ]
}

GROK uses the %{SYNTAX:SEMANTIC} pattern format. It extracts multiple fields in a single and readable command.

The ES|QL approach: DISSECT

For structured data with consistent delimiters, DISSECT is faster because it doesn’t use regular expressions:

FROM server-logs
| DISSECT raw_message "%{timestamp_date} %{log_level} user=%{user} action=%{action} duration=%{duration}"
| KEEP user, log_level, action, duration

The syntax is nearly identical to GROK, but DISSECT works by splitting on delimiters rather than matching regex patterns. This makes it faster for data that follows a consistent format.

When to use GROK vs DISSECT

Use DISSECT when your data has a predictable structure (same delimiters, same field order). Use GROK when you need regex flexibility, for example when fields may be optional or formats vary.

Pattern 3: Dynamic type conversion

When a field is mapped as keyword but contains numeric data (a surprisingly common scenario), runtime fields can cast it at query time.

The runtime field approach

GET server-logs/_search
{
  "runtime_mappings": {
    "response_time_long": {
      "type": "long",
      "script": {
        "source": """
          def val = doc['response_time'].value;
          if (val != 'not_available') {
            emit(Long.parseLong(val));
          }
        """
      }
    }
  },
  "fields": ["response_time_long"],
  "_source": false
}

You need to handle parsing exceptions manually. If Long.parseLong fails on an unexpected value, the script throws an error.

The ES|QL approach

ES|QL provides explicit conversion functions and a shorthand cast operator:

FROM server-logs
| EVAL response_ms = TO_LONG(response_time)
| KEEP host, response_time, response_ms

Or with the :: cast operator (available since 8.15):

FROM server-logs
| EVAL response_ms = response_time::long
| KEEP host, response_time, response_ms

Response:

{
  "columns": [
    { "name": "host", "type": "keyword" },
    { "name": "response_time", "type": "keyword" },
    { "name": "response_ms", "type": "long" }
  ],
  "values": [
    ["web-01", "145", 145]
  ]
}

Both produce the same result. The key difference from Painless: Failed conversions return null instead of throwing exceptions. The document with "not_available" simply gets null for response_ms, and ES|QL emits a warning.

Common conversion functions include:

The :: operator works with all these types (for example, field::double, field::datetime).

When to use

ES|QL's graceful null handling makes it safer for dirty data. Runtime fields with Painless give you fine-grained control over error handling but require more code. For type conversion specifically, ES|QL is almost always the better choice.

Pattern 4: Dynamic field handling

Runtime fields support "dynamic": "runtime" in mappings, which prevents mapping explosion by creating all new fields as runtime fields instead of indexed fields:

{
  "mappings": {
    "dynamic": "runtime",
    "properties": {
      "timestamp": { "type": "date" }
    }
  }
}

Any new field sent to this index becomes a runtime field automatically. This is useful when you ingest semi-structured data with unpredictable field names.

Where ES|QL fits

ES|QL provides query-time flexibility, but it still needs fields to be visible in the mapping. This is where runtime fields and ES|QL complement each other rather than compete.

If a field exists in _source but isn’t mapped, ES|QL cannot access it directly. The current workaround is to define a runtime field to make the unmapped field visible:

PUT dynamic-logs/_mapping
{
  "runtime": {
    "custom_field": {
      "type": "keyword",
      "script": {
        "source": "emit(params._source['custom_field'])"
      }
    }
  }
}

Once defined, ES|QL can query it:

FROM dynamic-logs
| WHERE custom_field == "some_value"
| KEEP timestamp, custom_field

This is one scenario where runtime fields remain essential. They act as a bridge, making unmapped data accessible to ES|QL.

Pattern 5: Field shadowing for error correction

Runtime fields can shadow (override) indexed fields by defining a runtime field with the same name as an existing field. This is useful for correcting data without reindexing.

The runtime field approach

Remember our data quality issue, where region has inconsistent casing ("US-EAST" versus "us-east")?

GET server-logs/_search
{
  "runtime_mappings": {
    "region": {
      "type": "keyword",
      "script": {
        "source": "emit(params._source['region'].toLowerCase())"
      }
    }
  },
  "fields": ["region"],
  "_source": false
}

This overrides the indexed region field for all queries. Every search, aggregation, and Kibana visualization will see the lowercase version.

FROM server-logs
| EVAL region = TO_LOWER(region)
| KEEP host, port, region

When you use EVAL with an existing column name, ES|QL drops the original column and replaces it with the computed value. This is the exact equivalent of field shadowing, but scoped to the current query.

You can also chain multiple corrections in a pipeline:

FROM server-logs
| EVAL region = TO_LOWER(region)
| EVAL region = CASE(region == "us-east", "US East", region == "eu-west", "EU West", region == "ap-south", "AP South", region)
| KEEP host, region

When to use

If the correction should apply to all queries and Kibana dashboards, use runtime field shadowing. If you need to correct data for a specific analysis, ES|QL is more flexible since you can apply different transformations in different queries without modifying the mapping.

The ES|QL pipeline advantage: Going beyond runtime fields

This is where ES|QL fundamentally surpasses runtime fields. Runtime fields are isolated: each one computes independently, and you cannot use the output of one runtime field as input for another in the same query.

ES|QL pipelines chain transforms. Here’s a single query that combines multiple patterns:

FROM server-logs
| GROK raw_message "%{WORD:log_date} %{WORD:log_level} user=%{WORD:user} action=%{WORD:action} duration=%{INT:duration_raw}ms"
| EVAL duration_ms = duration_raw::long
| EVAL region = TO_LOWER(region)
| WHERE log_level == "ERROR" AND duration_ms > 100
| STATS avg_duration = AVG(duration_ms), error_count = COUNT(*) BY region

This single query:

• Extracts fields from raw text (GROK).
• Converts the duration to a number (EVAL with cast).
• Normalizes region casing (EVAL with TO_LOWER).
• Filters for errors with high duration (WHERE).
• Aggregates by region (STATS).

To achieve the same result with runtime fields, you would need to define at least three separate runtime fields (for extraction, conversion, and normalization) and then write a Query DSL query with filters and aggregations. The ES|QL version is a single, readable pipeline.

You can even use expressions directly inside aggregations:

FROM server-logs
| EVAL response_ms = response_time::long
| STATS
    avg_response = AVG(response_ms),
    p95_response = PERCENTILE(response_ms, 95),
    slow_count = COUNT(CASE(response_ms > 1000, 1, null))
  BY host

Conclusion

What we covered:

• ES|QL provides a full toolkit (EVAL, GROK, DISSECT, type casting with ::) that replaces most runtime field patterns without any Painless scripting.
• Failed type conversions in ES|QL return null instead of throwing exceptions, making it safer for real-world data.
• Pipeline processing (chaining GROK into EVAL into WHERE into STATS) goes beyond what runtime fields can do in isolation.
• Runtime fields remain valuable for persistent computed fields, field shadowing across all queries, and as a bridge for unmapped data in ES|QL.

One important caveat: Both runtime fields and ES|QL compute values at query time, which means they pay the cost on every query. If you find yourself applying the same transformation repeatedly (type corrections, field extraction, data normalization), consider using ingest pipelines to fix the data at index time instead. Ingest pipelines let you parse, enrich, and transform documents before they’re stored, so queries can work with clean, properly typed fields directly. Runtime fields and ES|QL are great for exploration and ad-hoc analysis, but for production workloads, indexing the right data from the start is almost always the better choice.

The key takeaway: Runtime fields aren’t deprecated, and they aren’t going away. But for most query-time computation patterns, ES|QL offers a simpler, more powerful, and more performant approach. And when the transformation is known up front, an ingest pipeline is the most efficient option of all.

Next steps

• ES|QL documentation
• Runtime fields reference
• ES|QL timeline of improvements
• Getting started with runtime fields
• ES|QL processing data with DISSECT and GROK

In this article, we’ll explore the benefits of building a custom Elasticsearch MCP server and show how to create one in TypeScript that connects Elasticsearch to LLM-powered applications.

Why build a custom Elasticsearch MCP server?

Elastic provides some alternatives for MCP servers:

• Elastic Agent Builder MCP server for Elasticsearch 9.2+
• Elasticsearch MCP server for older versions (Python)

If you need more control over how your MCP server interacts with Elasticsearch, building your own custom server gives you the flexibility to tailor it exactly to your needs. For example, Agent Builder's MCP endpoint is limited to Elasticsearch Query Language (ES|QL) queries, while a custom server allows you to use the full Query DSL. You also gain control over how results are formatted before being passed to the LLM and can integrate additional processing steps, like the OpenAI-powered summarization we'll implement in this tutorial.

By the end of this article, you’ll have an MCP server in TypeScript that searches for information stored in an Elasticsearch index, summarizes it, and provides citations. We'll use Elasticsearch for retrieval, OpenAI's gpt-4o-mini model to summarize and generate citations, and Claude Desktop as the MCP client and UI to take in user queries and give responses. The end result is an internal knowledge assistant that helps engineers discover and synthesize best practices across their organization’s technical docs.

Prerequisites:

• Node.js 20 +
• Elasticsearch
• OpenAI API key
• Claude Desktop

What is MCP?

MCP is an open standard, created by Anthropic, that provides secure, bidirectional connections between LLMs and external systems, like Elasticsearch. You can read more about the current state of MCP in this article.

The MCP landscape is evolving every day, with servers available for a wide range of use cases. On top of that, it’s easy to build your own custom MCP server, as we’ll show in this article.

MCP clients

There’s a long list of available MCP clients, each with its own characteristics and limitations. For simplicity and popularity, we’ll use Claude Desktop as our MCP client. It will serve as the chat interface where users can ask questions in natural language, and it will automatically invoke the tools exposed by our MCP server to search documents and generate summaries.

Creating an Elasticsearch MCP server

Using the TypeScript SDK, we can easily create a server that understands how to query our Elasticsearch data based on a user query input.

Here are the steps in this article to integrate the Elasticsearch MCP server with the Claude Desktop client:

1. Configure MCP server for Elasticsearch.
2. Load the MCP server into Claude Desktop.
3. Test it out.

Configure MCP server for Elasticsearch

To begin, let's initialize a node application:

npm init -y

This will create a package.json file, and with it, we can start installing the necessary dependencies for this application.

npm install @elastic/elasticsearch @modelcontextprotocol/sdk openai zod && npm install --save-dev ts-node @types/node typescript

• @elastic/elasticsearch will give us access to the Elasticsearch Node.js library.
• @modelcontextprotocol/sdk provides the core tools to create and manage an MCP server, register tools, and handle communication with MCP clients.
• openai allows interaction with OpenAI models to generate summaries or natural language responses.
• zod helps define and validate structured schemas for input and output data in each tool.

ts-node, @types/node, and typescript will be used during development to type the code and compile the scripts.

Set up the dataset

To provide the data that Claude Desktop can query using our MCP server, we’ll use a mock internal knowledge base dataset. Here’s what a document from this dataset will look like:

{
    "id": 5,
    "title": "Logging Standards for Microservices",
    "content": "Consistent logging across microservices helps with debugging and tracing. Use structured JSON logs and include request IDs and timestamps. Avoid logging sensitive information. Centralize logs in Elasticsearch or a similar system. Configure log rotation to prevent storage issues and ensure logs are searchable for at least 30 days.",
    "tags": ["logging", "microservices", "standards"]
}

To ingest the data, we prepared a script that creates an index in Elasticsearch and loads the dataset into it. You can find it here.

MCP server

Create a file named index.ts and add the following code to import the dependencies and handle environment variables:

// index.ts
import { z } from "zod";
import { Client } from "@elastic/elasticsearch";
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import OpenAI from "openai";

const ELASTICSEARCH_ENDPOINT =
  process.env.ELASTICSEARCH_ENDPOINT ?? "http://localhost:9200";
const ELASTICSEARCH_API_KEY = process.env.ELASTICSEARCH_API_KEY ?? "";
const OPENAI_API_KEY = process.env.OPENAI_API_KEY ?? "";
const INDEX = "documents";

Also, let’s initialize the clients to handle the Elasticsearch and OpenAI calls:

const openai = new OpenAI({
  apiKey: OPENAI_API_KEY,
});

const _client = new Client({
  node: ELASTICSEARCH_ENDPOINT,
  auth: {
    apiKey: ELASTICSEARCH_API_KEY,
  },
});

To make our implementation more robust and ensure structured input and output, we'll define schemas using zod. This allows us to validate data at runtime, catch errors early, and make the tool responses easier to process programmatically:

const DocumentSchema = z.object({
  id: z.number(),
  title: z.string(),
  content: z.string(),
  tags: z.array(z.string()),
});

const SearchResultSchema = z.object({
  id: z.number(),
  title: z.string(),
  content: z.string(),
  tags: z.array(z.string()),
  score: z.number(),
});

type Document = z.infer;
type SearchResult = z.infer;

Learn more about structured outputs here.

Now let’s initialize the MCP server:

const server = new McpServer({
  name: "Elasticsearch RAG MCP",
  description:
    "A RAG server using Elasticsearch. Provides tools for document search, result summarization, and source citation.",
  version: "1.0.0",
});

Defining the MCP tools

With everything configured, we can start writing the tools that will be exposed by our MCP server. This server exposes two tools:

• search_docs: Searches for documents in Elasticsearch using full-text search.
• summarize_and_cite: Summarizes and synthesizes information from previously retrieved documents to answer a user question. This tool also adds citations referencing the source documents.

Together, these tools form a simple “retrieve-then-summarize” workflow, where one tool fetches relevant documents and the other uses those documents to generate a summarized, cited response.

Tool response format

Each tool can accept arbitrary input parameters, but it must respond with the following structure:

• Content: This is the response of the tool in an unstructured format. This field is usually used to return text, images, audio, links, or embeddings. For this application, it will be used to return formatted text with the information generated by the tools.
• structuredContent: This is an optional return used to provide the results of each tool in a structured format. This is useful for programmatic purposes. Although it isn't used in this MCP server, it can be useful if you want to develop other tools or process the results programmatically.

With that structure in mind, let’s dive into each tool in detail.

Search_docs tool

This tool performs a full-text search in the Elasticsearch index to retrieve the most relevant documents based on the user query. It highlights key matches and provides a quick overview with relevance scores.

server.registerTool(
  "search_docs",
  {
    title: "Search Documents",
    description:
      "Search for documents in Elasticsearch using full-text search. Returns the most relevant documents with their content, title, tags, and relevance score.",
    inputSchema: {
      query: z
        .string()
        .describe("The search query terms to find relevant documents"),
      max_results: z
        .number()
        .optional()
        .default(5)
        .describe("Maximum number of results to return"),
    },
    outputSchema: {
      results: z.array(SearchResultSchema),
      total: z.number(),
    },
  },
  async ({ query, max_results }) => {
    if (!query) {
      return {
        content: [
          {
            type: "text",
            text: "Query parameter is required",
          },
        ],
        isError: true,
      };
    }

    try {
      const response = await _client.search({
        index: INDEX,
        size: max_results,
        query: {
          bool: {
            must: [
              {
                multi_match: {
                  query: query,
                  fields: ["title^2", "content", "tags"],
                  fuzziness: "AUTO",
                },
              },
            ],
            should: [
              {
                match_phrase: {
                  title: {
                    query: query,
                    boost: 2,
                  },
                },
              },
            ],
          },
        },
        highlight: {
          fields: {
            title: {},
            content: {},
          },
        },
      });

      const results: SearchResult[] = response.hits.hits.map((hit: any) => {
        const source = hit._source as Document;

        return {
          id: source.id,
          title: source.title,
          content: source.content,
          tags: source.tags,
          score: hit._score ?? 0,
        };
      });

      const contentText = results
        .map(
          (r, i) =>
            `[${i + 1}] ${r.title} (score: ${r.score.toFixed(
              2,
            )})\n${r.content.substring(0, 200)}...`,
        )
        .join("\n\n");

      const totalHits =
        typeof response.hits.total === "number"
          ? response.hits.total
          : (response.hits.total?.value ?? 0);

      return {
        content: [
          {
            type: "text",
            text: `Found ${results.length} relevant documents:\n\n${contentText}`,
          },
        ],
        structuredContent: {
          results: results,
          total: totalHits,
        },
      };
    } catch (error: any) {
      console.log("Error during search:", error);

      return {
        content: [
          {
            type: "text",
            text: `Error searching documents: ${error.message}`,
          },
        ],
        isError: true,
      };
    }
  }
);

We configure fuzziness: “AUTO” to have a variable typo tolerance based on the length of the token that’s being analyzed. We also set title^2 to increase the score of the documents where the match happens on the title field.

summarize_and_cite tool

This tool generates a summary based on documents retrieved in the previous search. It uses OpenAI’s gpt-4o-mini model to synthesize the most relevant information to answer the user’s question, providing responses derived directly from the search results. In addition to the summary, it also returns citation metadata for the source documents used.

server.registerTool(
  "summarize_and_cite",
  {
    title: "Summarize and Cite",
    description:
      "Summarize the provided search results to answer a question and return citation metadata for the sources used.",
    inputSchema: {
      results: z
        .array(SearchResultSchema)
        .describe("Array of search results from search_docs"),
      question: z.string().describe("The question to answer"),
      max_length: z
        .number()
        .optional()
        .default(500)
        .describe("Maximum length of the summary in characters"),
      max_docs: z
        .number()
        .optional()
        .default(5)
        .describe("Maximum number of documents to include in the context"),
    },
    outputSchema: {
      summary: z.string(),
      sources_used: z.number(),
      citations: z.array(
        z.object({
          id: z.number(),
          title: z.string(),
          tags: z.array(z.string()),
          relevance_score: z.number(),
        })
      ),
    },
  },
  async ({ results, question, max_length, max_docs }) => {
    if (!results || results.length === 0 || !question) {
      return {
        content: [
          {
            type: "text",
            text: "Both results and question parameters are required, and results must not be empty",
          },
        ],
        isError: true,
      };
    }

    try {
      const used = results.slice(0, max_docs);

      const context = used
        .map(
          (r: SearchResult, i: number) =>
            `[Document ${i + 1}: ${r.title}]\\n${r.content}`
        )
        .join("\n\n---\n\n");

      // Generate summary with OpenAI
      const completion = await openai.chat.completions.create({
        model: "gpt-4o-mini",
        messages: [
          {
            role: "system",
            content:
              "You are a helpful assistant that answers questions based on provided documents. Synthesize information from the documents to answer the user's question accurately and concisely. If the documents don't contain relevant information, say so.",
          },
          {
            role: "user",
            content: `Question: ${question}\\n\\nRelevant Documents:\\n${context}`,
          },
        ],
        max_tokens: Math.min(Math.ceil(max_length / 4), 1000),
        temperature: 0.3,
      });

      const summaryText =
        completion.choices[0]?.message?.content ?? "No summary generated.";

      const citations = used.map((r: SearchResult) => ({
        id: r.id,
        title: r.title,
        tags: r.tags,
        relevance_score: r.score,
      }));

      const citationText = citations
        .map(
          (c: any, i: number) =>
            `[${i + 1}] ID: ${c.id}, Title: "${c.title}", Tags: ${c.tags.join(
              ", ",
            )}, Score: ${c.relevance_score.toFixed(2)}`,
        )
        .join("\n");

      const combinedText = `Summary:\\n\\n${summaryText}\\n\\nSources used (${citations.length}):\\n\\n${citationText}`;

      return {
        content: [
          {
            type: "text",
            text: combinedText,
          },
        ],
        structuredContent: {
          summary: summaryText,
          sources_used: citations.length,
          citations: citations,
        },
      };
    } catch (error: any) {
      return {
        content: [
          {
            type: "text",
            text: `Error generating summary and citations: ${error.message}`,
          },
        ],
        isError: true,
      };
    }
  }
);

Finally, we need to start the server using stdio. This means the MCP client will communicate with our server by reading and writing to its standard input and output streams. stdio is the simplest transport option and works well for local MCP servers launched as subprocesses by the client. Add the following code at the end of the file:

const transport = new StdioServerTransport();
server.connect(transport);

Now compile the project using the following command:

npx tsc index.ts --target ES2022 --module node16 --moduleResolution node16 --outDir ./dist --strict --esModuleInterop

This will create a dist folder, and inside it, an index.js file.

Load the MCP server into Claude Desktop

Follow this guide to configure the MCP server with Claude Desktop. In the Claude configuration file, we need to set the following values:

{
  "mcpServers": {
    "elasticsearch-rag-mcp": {
      "command": "node",
      "args": [   "/Users/user-name/app-dir/dist/index.js"
      ],
      "env": {
        "ELASTICSEARCH_ENDPOINT": "your-endpoint-here",
        "ELASTICSEARCH_API_KEY": "your-api-key-here",
        "OPENAI_API_KEY": "your-openai-key-here"
      }
    }
  }
}

The args value should point to the compiled file in the dist folder. You also need to set the environment variables in the configuration file with the exact same names defined in the code.

Test it out

Before executing each tool, click on Search and Tools to make sure that the tools are enabled. Here you can also enable or disable each one:

Finally, let’s test the MCP server from the Claude Desktop chat and start asking questions:

For the question “Search for documents about authentication methods and role-based access control”, the search_docs tool is executed and returns the following results:

Most Relevant Documents:
Access Control and Role Management (highest relevance) - This document covers role-based access control (RBAC) principles, including ensuring users only have necessary permissions, regular auditing of user roles, revoking inactive accounts, and implementing just-in-time access for sensitive operations.
User Authentication with OAuth 2.0 - This document explains OAuth 2.0 authentication, which enables secure delegated access without credential sharing. It covers configuring identity providers, token management with limited scope and lifetime, and secure storage of refresh tokens.
Container Security Guidelines - While primarily about container security, this document touches on access control aspects like running containers as non-root users and avoiding embedded credentials.
Incident Response Playbook - This mentions role assignment during incidents (incident commander, communications lead, etc.), which relates to access control in emergency scenarios.
Logging Standards for Microservices - This document includes guidance on avoiding logging sensitive information, which is relevant to authentication security.

The response is, “Great! I found 5 relevant documents about authentication methods and role-based access control. Here's what was found:”

The tool call returns the source documents as part of its response payload, which are later used to generate citations.

It’s also possible to chain multiple tools in a single interaction. In this case, Claude Desktop analyzes the user's question and determines that it needs to first call search_docs to retrieve relevant documents and to then pass those results to summarize_and_cite to generate the final answer, all without requiring separate prompts from the user:

In this case, for the query “What are the main recommendations to improve authentication and access control across our systems? Include references.”, we obtained the following results:

Based on the documentation, here are the main recommendations to improve authentication and access control across your systems:

Key Recommendations

1. Implement Role-Based Access Control (RBAC)
Ensure users have only the permissions necessary for their job functions. This principle of least privilege minimizes security risks by limiting access to what's truly needed. [1]

2. Regular Access Audits
Conduct regular audits of user roles and promptly revoke access for inactive accounts. This helps prevent unauthorized access from dormant or compromised accounts. [1]

3. Just-in-Time (JIT) Access
Implement just-in-time access for sensitive operations, where permissions are granted only when needed and for a limited duration. This reduces the attack surface by minimizing standing privileges. [1]

4. OAuth 2.0 for Secure Authentication
Use OAuth 2.0 to enable secure delegated access without sharing user credentials. This protocol allows third-party applications to access resources on behalf of users without exposing passwords. [2]

5. Token Security and Management
Configure your identity provider to issue tokens with limited scope and lifetime

Store refresh tokens securely
Validate access tokens consistently to maintain security [2]
References

Access Control and Role Management (Tags: security, access-control)
User Authentication with OAuth 2.0 (Tags: authentication, oauth)
These recommendations work together to create a defense-in-depth approach, where multiple security layers protect your systems from unauthorized access.

As in the previous step, we can see the response from each tool for this question:

Note: If a submenu appears asking whether you approve the use of each tool, select Always allow or Allow once.

Conclusion

MCP servers represent a significant step toward standardizing LLM tools for both local and remote applications. Though full compatibility is still in the works, we’re moving fast in that direction.

In this article, we learned how to build a custom MCP server in TypeScript that connects Elasticsearch to LLM-powered applications. Our server exposes two tools: search_docs for retrieving relevant documents using Query DSL; and summarize_and_cite for generating summaries with citations via OpenAI models and Claude Desktop as client UI.

The future of compatibility between different client and server providers looks promising. Next steps include adding more functionalities and flexibility to your agent. There’s a practical article on how you can parameterize your queries using search templates to gain precision and flexibility.

That’s exactly why we built read-only dashboards. It’s the control you’ve been asking for. Share dashboards with confidence, without worrying that the next person with edit access will change or break them.

Note: Read-only permissions are available in Elastic Cloud Serverless and from version 9.3 for Elastic Cloud Hosted and Elastic Self-Μanaged.

When “everyone can edit” gets in the way

In Kibana, sharing has usually meant space-level permissions. If someone can create dashboards in a space, they can also edit or delete anyone else’s. That’s great for collaboration until it isn’t. One accidental edit can ripple into wrong decisions, lost trust, and a lot of cleanup.

We’ve heard the workarounds: “We put ‘read-only’ in the dashboard name and hope people notice.” Or: “We tag them and cross our fingers.” Hope isn’t a permission model. You needed a real way to lock a dashboard without locking everyone out of the space.

What actually goes wrong

Deb and Kevin both have edit access to the log monitoring dashboard within the Operations space. Kevin makes some changes to the charts. When Deb comes back, the numbers don’t match what she presented. She has to track down what changed (often from memory), fix it, and wonder how many reports went out with bad data.

Read-only dashboards: Ownership and control that make sense

Read-only dashboards fix this by giving you control to decide whether other users can edit the dashboard. When you share a dashboard, you choose: edit (default, same as today) or view. In view mode, only you (and Kibana admins) can change or delete it. Everyone else can open it, use it, and trust it, but they can’t modify it.

What you get

• Dashboard integrity: In view mode, other users with edit access in the space cannot modify or delete the dashboard. If they try, they’re told it’s locked. Your charts and logic stay as you left them.
• You stay in control: You’re the owner. You can always edit, refine, and update. Sharing as view-only doesn’t lock you out; it locks in the version everyone else sees.
• Flexible lifecycle: You can switch a dashboard back to “can edit” anytime. And Kibana admins can still manage all dashboards (for example, if the owner leaves). No dead ends.

You can share finalized, mission-critical dashboards widely and know they’ll stay consistent. This is available in all Elastic tiers and offerings, including Serverless.

Who can do what?

Quick reference by role:

• Dashboard owner: You created it; you have full edit access.
• Kibana admin: Can manage all dashboards.
• User with space edit: Can create and edit their dashboards; can’t edit or delete view-only dashboards.
• User with space view: Can only view (and list) dashboards.

How to turn on read-only

You can set view-only when you save a new dashboard or later from the share menu.

When saving a new dashboard

• Build your dashboard, and click Save.
• In the “Save as new dashboard” modal, find Permissions.
• Change from Can edit to Can view.
• Click Save. Done. It’s read-only for everyone else.

For a dashboard you already own

• Open the dashboard.
• Open the Share dashboard menu.

• In the sharing modal, find Permissions and switch to Can view. The change applies immediately; other users in the space can no longer edit or delete it.

• You can mouse over the Share action to see what type of permissions a given dashboard has.

Seeing which dashboards are locked

On the main Dashboards list, dashboards you can’t edit or delete have a disabled selection checkbox. This provides an easy way to spot what’s view-only.

In the dashboard, you will also find that the Edit action is disabled and a tooltip will appear, explaining that the dashboard has been set as view-only.

Try it

Read-only dashboards are available now. Create a dashboard, flip it to Can view, and share it. Your team gets a single source of truth, and you get peace of mind. No more “please don’t edit” in the title.

We’d love to hear how you use read-only dashboards. Share your feedback in our community forum.

This post refocuses on the question,What are the right search interfaces an agent needs to build its own context? It first covers the trade-offs between shell tools and dedicated database tools. From there, it offers a practical framework for finding the right interfaces for your agent's needs.

What does "building context" actually mean for an agent?

In early retrieval augmented generation (RAG) pipelines, the developer engineered a fixed retrieval pipeline, and the large language model (LLM) was a passive recipient of the context. This was a fundamental limitation: Context was retrieved on every query, whether or not it was needed, with no check that it actually helped.

With the shift to agentic RAG, the agents now have access to a set of search tools to build their own context. For example, both Claude Code [1] and Cursor [2] let the agent choose between different search tools and even combine them for chained queries, depending on what the task actually requires.

What search interfaces exist for context engineering?

Context can live in different locations, such as on the web, in a local filesystem, or in a database. An agent can interact with each of these out-of-context data sources through different tools:

• Shell tools can execute shell commands and have access to the local filesystem. Some examples of built-in shell tools are Claude API's bash tool, OpenClaw's exec tool, and LangChain's shell tool.
• Dedicated database tools, such as tools from a Model Context Protocol (MCP) server (for example, the Elastic Agent Builder MCP server) or custom tools (for example, run_esql(query) or db_list_index()), can query databases.
• Dedicated file search tools can search and read local (or uploaded) files (without full shell access). Some examples of built-in file search tools are Gemini API’s File Search Tool or OpenAI’s File Search Tool.
• Web search tools can retrieve information from the web.
• Memory tools store and recall from long-term memory (regardless of how it’s stored).

As you can see, the shell tool is versatile and can be used to retrieve context from different data sources, including:

• Filesystem: The agent explores the directory structure (ls, find), searches for relevant content (grep, cat), and repeats until it has built sufficient context.
• Database: The agent can use database command line interface (CLI) tools (for example, elasticsearch-sql-cli), call HTTP APIs via curl, or run scripts, which is especially useful in combination with agent skills, which are reusable, documented examples injected into the agent's context to guide correct tool usage (for example, Elastic Agent Skills for Elasticsearch).
• Web: The agent can execute web searches via a curl command through a search provider’s API.

However, the shell tool provides direct system access and therefore requires safety measures, such as running in an isolated sandbox environment and logging all executed commands.

When to use which search interfaces

The right search interface depends on your data, your query patterns, and your use case. This section serves as a practical starting point.

Filesystems aren’t making databases obsolete

The filesystems-versus-databases discussion is not about the storage layer. For example, LangChain explains that its memory system doesn’t actually store memory in a real filesystem. Instead, it stores memory in a database and represents it as a set of files to the agent [3].

Filesystems are a natural fit for file-native use cases, such as coding agents. They also work well as a temporary scratch pad or working memory and for single-user or single-agent scenarios where concurrency isn't a concern. In these cases, a physical filesystem or representing the data as a filesystem gives you flexibility before committing to a purpose-built interface.

But filesystem storage has real downsides, such as weak concurrency, manual schema enforcement, and atomic transactions. These become more apparent when your application needs to scale or move to a multi-agent scenario. Anyone who ignores these downsides is doomed to painfully reinvent worse databases without the decades of engineering behind transaction safety or access control that production databases already provide. Additionally, in most enterprise contexts, you don't choose whether to use a database since it's already there, storing business-critical data.

Shell tool + filesystem

A shell tool is the natural starting point for filesystem search. Currently, coding agents are driving a lot of progress in the field. Because they work with code in local files, they’re naturally file-heavy use cases. Therefore, LLMs are fine-tuned in the post-training stage for coding tasks. That’s why many LLMs are not only good at writing code but also at using shell commands and navigating filesystems.

Using a shell tool with built-in CLIs, like ls and grep, to find files is effective. With grep, a query like "Find all files that import matplotlib" is fast, precise, and cheap. But when the agent needs to handle conceptual queries, such as "How does our app handle failed authentication?", pattern matching with grep can hit a ceiling quickly. Several alternatives that bring semantic search capabilities to the command line have emerged to fill this gap, including jina-grep.

However, grep and many of its semantic search alternatives run in O(n) over the corpus. For use cases over codebases, this might be fine. However, if your data grows, latency will become noticeable. In this case, an indexed datastore becomes necessary to maintain performance.

Shell tool + database

Another way to add more search capabilities, such as semantic or hybrid search, over your data is to store it in a database, as Cursor does, for example. Additionally, when data requires complex relational joins or aggregations, a database interface is nonnegotiable.

When the data lives in a database rather than on the filesystem, a shell tool can serve as a lightweight database interface for certain use cases. If your queries are simple enough for a CLI or a curl call, a dedicated database tool may add unnecessary complexity.

This approach is also suitable in early exploration stages, when you don't yet know what query patterns your agent will actually develop. In this case, Agent Skills can give the agent enough structure to query correctly without committing to a purpose-built tool. However, when the agent requires many iterations to figure out the right way to query the database for repeated tasks, the token overhead of using a shell tool as the interface no longer justifies the simplicity benefit of avoiding an extra tool.

Dedicated database tool

Especially when repeated query patterns are structured or analytical, dedicated database tools become necessary. A blog post from Vercel and Braintrust compared agents with different sets of search tools for real-world retrieval tasks over semi-structured data, such as customer support tickets and sales call transcripts (for example, “How many open issues mention 'security'?" or "Find issues where someone reported a bug and later someone submitted a PR claiming to fix it?") [4].

Agents with dedicated database tools used fewer tokens, were faster, and made fewer mistakes than agents with only a shell tool and filesystem. The lesson is that direct database tools are the right choice when the query requires analytical reasoning over semi-structured data.

Combining search interfaces

No single search interface handles every query well. For example, Cursor combines shell tools (for searches via grep) and semantic search tools and lets the agent select the right tool based on the user’s prompt. They report that the agent chooses grep for matching specific symbols or strings, semantic search for conceptual or behavior questions, and both for exploratory tasks.

The Vercel experiment reports the same: Its hybrid agent with access to both a shell tool and a dedicated database tool achieved the best performance out of all tested agents by first using the dedicated database tools and then verifying the results by grepping through the filesystem. However, this approach uses more tokens and time for reasoning about tool choice and verification.

The pattern across both examples is the same: Composition beats any single interface, but composition comes at the trade-off of added cost and latency.

Practical recommendations for finding the right set of tools

The right set of search interfaces is small, purposeful, and specific to your agent's actual query patterns. The current best practice is to have an agent with as few tools as possible instead of having an agent with hundreds of MCP tools. This is because the downside of exposing all possible tools up front is that it bloats the context window and confuses the agent about which tool to actually use. For example, Claude Code reportedly only has about 20 tools.

Instead, the idea of progressive disclosure is to start with a minimal set of tools and let the agent discover additional capabilities only when needed. Research from Anthropic [5] and Cursor [6] has shown that this approach yields a token savings between 47%–85%. Claude Code, for example, implements this directly, allowing the agent to incrementally discover how to query an API or a database, without that knowledge consuming context on every LLM call.

Once you’re familiar with the agent's query patterns, you can revisit the set of search tools that the agent has access to by default. A useful way to think about this trade-off is the "low floor, high ceiling" principle for deciding which tools make the cut. High-ceiling tools don't limit the agent's potential. For example, a versatile shell tool lets the agent write full database queries, including ambiguous ones, but at the cost of reasoning overhead, higher latency, and lower reliability.

Low-floor tools are the opposite. They’re specialized tools that wrap specific queries and are immediately accessible to the agent with minimal reasoning overhead, producing lower cost and higher reliability. But they need upfront engineering, can't cover every possible query, and can make it harder for the agent to choose the right tool.

Think of each tool on a spectrum: Low-floor tools are easy for the agent to use correctly but narrow in scope. High-ceiling tools are versatile but demand more reasoning to use well.

Most agents need a mix of different search tools. But each tool needs to earn its addition. We recommend starting with an all-purpose search tool (for example a search_database() tool or a shell tool). Then reuse the command logs you're already keeping for security purposes to track what your agent actually does, including tool calls, retries, and number of calls per user query. And, when you see a query pattern repeating or failing, that's the signal to build a purpose-built tool for it.

Summary

The filesystem-versus-database debate is distracting from the actual question that engineers need to be asking: What are the right search interfaces an agent needs to build its own context? The answer is most likely, Not a single one.

A shell tool is a versatile tool to interact with different out-of-context sources and thus a good starting point. But it’s less efficient and accurate for use cases with structured analytical queries than dedicated database tools.

The goal is to find the minimal set of search tools that handles your agent's actual query patterns well. Start with a shell tool, and log what your agent actually does. When you see a query pattern repeating and failing, it’s time to engineer specialized tools.

References

1. Thariq (Anthropic). Lessons from Building Claude Code: Seeing like an Agent (2026).

2. Cursor: Documentation. Semantic & agentic search (2026).

3. Harrison Chase (LangChain). How we built Agent Builder’s memory system (2026).

4. Ankur Goyal (Braintrust) and Andrew Qu (Vercel). Testing if "bash is all you need" (2026).

5. Anthropic. Introducing advanced tool use on the Claude Developer Platform (2025).

6. Cursor. Dynamic context discovery (2026).

The party is getting crowded

You're hosting a pizza party. You've got a few friends helping you serve, each stationed at different spots around the room. You give each friend a pizza, and they start handing out slices to hungry guests as they arrive.

At first, things run smoothly. A few guests trickle in, your friends serve slices, everyone's happy. But then word spreads about your sourdough pizzas. The doorbell keeps ringing. Guests pour in. Soon, there's a crowd forming around one of your friends, the one holding the pepperoni pizza, which everyone seems to want.

Your friend with the pepperoni pizza is overwhelmed. Guests are waiting, getting impatient, and a large queue has formed. Meanwhile, your friend holding the margherita pizza is standing around with barely anyone asking for a slice.

What do you do?

You order a couple more pepperoni pizzas and hand them to other friends. Now three friends are holding pepperoni instead of one. The crowd spreads out, and suddenly you can serve three times as many guests at once.

A few things become clear as you host more parties:

• Not all pizzas are equally popular. Some are in high demand, others have fewer takers. You don't need extra "copies" of the unpopular ones. You need extras of the ones with queues.
• Order more pizzas before the queue gets too long. If you wait until your friend is completely overwhelmed and guests are leaving angry, you've waited too long. Better to get an extra pizza when you see a crowd forming.
• Don't throw away pizzas too quickly. Just because the crowd around the pepperoni thinned out for five minutes doesn't mean the rush is over. Maybe they're just refilling drinks, or even talking among themselves (is that still a thing?). Keep the extra pizzas ready. If the lull continues for a while, then you can put them away.
• You can only hand out as many pizzas as you have friends who are helping. If you've only got four friends helping, ten pizzas won’t change the outcome. Only four can be served at once. Match your pizza count to your available hands.
• When a friend leaves, take their pizza. If one of your friends needs to head out, grab their pizza immediately. You can't have pizzas sitting unattended. Hand it to someone else, or put it away.

From pizzas to replicas

Let's map this back to Elasticsearch.

In our analogy, pizzas are replicas (copies of your index shards), your friends helping serve are search nodes, hungry guests are search queries, and that popular pizza with a crowd around it is a hot index with high search load.

When search traffic increases on a particular index, we create additional replicas and distribute them across your search nodes. Any replica can serve any query for that index, just like any friend holding pepperoni can hand out pepperoni slices. More replicas means higher throughput: Three replicas can handle three times the queries per second of a single replica.

Measuring the hunger

Before we decide how many pizzas to order, we need to know how hungry the crowd is.

Elasticsearch tracks the search load for every shard. It's a metric that captures how much search activity a shard is handling. We aggregate this across all shards of an index to understand the total search demand.

What matters most is the relative search load: What proportion of your project's total search traffic is hitting each index? If one index is receiving 60% of all searches while another gets 5%, we know where to add capacity.

The math behind the pizzas

We calculate the optimal number of replicas following this formula:

desired_replicas = min(ceil(L × N / (S × X)), N)

Where:

• L = the index's relative search load (between 0 and 1).
• N = the number of desired search nodes in your project.
• S = the number of shards in the index.
• X = a threshold to avoid hot spots (default: 0.5).

An example: four search nodes, one index with two primary shards receiving 80% of search traffic:

desired_replicas = min(ceil(0.8 × 4 / (2 × 0.5)), 4)
                 = min(4, 4)
                 = 4

This hot index gets four replicas distributed across the search nodes.

The threshold X (defaulting to 0.5) is important. We don't wait until a replica is completely overwhelmed; we scale up when it's at half capacity. Hand out the extra pizza when you see the crowd forming, not when guests are already leaving.

Scale up fast, scale down slow

When search load increases, we add replicas immediately. No reason to make users wait.

When search load drops, we wait a bit before taking any action. We need to see consistent low demand for about 30 minutes before reducing replicas. (This is to deal with spiky traffic where a quiet moment doesn't mean the party is over.)

This matters because adding a replica has a cost. The new replica copies data and warms its caches before serving queries efficiently. Removing replicas too eagerly means constantly paying this startup cost as traffic naturally fluctuates.

Respecting topology bounds

Replicas can never exceed the number of search nodes. Having more replicas than nodes provides no benefit (you can only serve as many pizzas as you have friends who are helping to serve slices).

When nodes are removed from your project, we reduce replicas immediately to match. No waiting for the cooldown, as you can't have unassigned replicas. The moment a friend leaves, we remove their pizza.

The bigger Serverless picture

Replicas for search load balancing works alongside other autoscaling systems:

• Search autoscaling adjusts the number of search nodes (how many friends are helping).
• Replicas for search load balancing distribute traffic by adjusting replica counts per index (how many pizzas of each kind we need).
• Data stream autosharding optimizes shard counts for writes (how to slice each pizza, covered in the previous post).

An important design principle: Replicas for load balancing don't directly trigger search autoscaling. Instead, by distributing search requests across more replicas, it enables increasing resource utilization across your search nodes. This higher utilization then triggers our existing autoscaling logic to add capacity if needed. Replicas for load balancing enables autoscaling to do its job, making sure your search nodes are actually being used, rather than having all traffic bottlenecked on a single replica while other nodes sit idle.

What this means for you

You don't need to predict which indices will be popular. You don't need to manually adjust replicas when traffic patterns change. You don't need to wake up at 3 a.m. because a surge overwhelmed your busiest index.

The system watches where queues are forming and orders more pizzas for those spots. Cold indices don't waste resources on unnecessary replicas. Hot indices get the capacity they need. Your budget goes where it matters.

Conclusion

In the autosharding post, we made sure your pizzas are sliced right. Now, with replicas for search load balancing, we make sure you have enough pizzas, in the right hands, when the hungry crowds arrive.

Try Elastic Cloud Serverless and let us handle the pizza logistics.

Prerequisites

• Elasticsearch 9.3 or Elastic Cloud Serverless: You can create a cloud deployment following these instructions, or you can use the start-local quickstart instead.
• Python 3.12: Download Python here.
• Hugging Face access token.

Chat completions using a Hugging Face inference endpoint

First, we’ll build a practical example that connects Elasticsearch to a Hugging Face inference endpoint to generate AI-powered recommendations from a collection of blog posts. For the app knowledge base, we’ll use a dataset of company blog articles, which contains valuable but often hard-to-navigate information.

With this endpoint, semantic search retrieves the most relevant articles for a given query, and a Hugging Face LLM generates short, contextual recommendations based on those results.

Let’s take a look at a high-level overview of the information flow we’re going to build:

In this article, we’ll test SmolLM3-3B capacity to combine its compact size with strong multilingual reasoning and tool-calling capabilities. Based on a search query, we’ll send all the matching content (in English and Spanish) to the LLM to generate a list of recommended articles with a custom-made description based on the search query and results.

Here’s what the UI of an article site with an AI recommendations generation system could look like.

You can find the full implementation of this application in the linked notebook.

Configuring Elasticsearch inference endpoints

To use the Elasticsearch Hugging Face inference endpoint, we need two important elements: a Hugging Face API key and a running Hugging Face endpoint URL. It should look like this:

PUT _inference/chat_completions/hugging-face-smollm3-3b
{
    "service": "hugging_face",
    "service_settings": {
        "api_key": "hugging-face-access-token", 
        "url": "url-endpoint" 
    }
}

The Hugging Face inference endpoint in Elasticsearch supports different task types: text_embedding, completion, chat_completion, and rerank. In this blog post, we use chat_completion because we need the model to generate conversational recommendations based on the search results and a system prompt.This endpoint allows us to perform chat completions directly from Elasticsearch in a simple way using the Elasticsearch API:

POST _inference/chat_completion/hugging-face-smollm3-3b/_stream
{
  "messages": [
      { "role": "user", "content": "" }
  ]
}

This will serve as the core of the application, receiving the prompt and the search results that will pass through the model. With the theory covered, let’s start implementing the application.

Setting up ​​inference endpoint on Hugging Face

To deploy the Hugging Face model, we’re going to use Hugging Face one-click deployments, an easy and fast service for deploying model endpoints. Keep in mind that this is a paid service, and using it may incur additional costs. This step will create the model instance that will be used to generate the recommendations of the articles.

You can pick a model from the one-click catalog:

Let’s pick the SmolLM3-3B model:

From here, grab the Hugging Face endpoint URL:

As mentioned in the Elasticsearch Hugging Face inference endpoints documentation, text generation requires a model that’s compatible with the OpenAI API. For that reason, we need to append the /v1/chat/completions subpath to the Hugging Face endpoint URL. The final result will look like this:

https://j2g31h0futopfkli.us-east-1.aws.endpoints.huggingface.cloud/v1/chat/completions

With this in place, we can start coding in a Python notebook.

Generating Hugging Face API key

Create a Hugging Face account, and obtain an API token by following these instructions. You can choose between three token types: fine-grained (recommended for production, as it provides access only to specific resources); read (for read-only access); or write (for read and write access). For this tutorial, a read token is sufficient, since we only need to call the inference endpoint. Save this key for the next step.

Setting up Elasticsearch inference endpoint

First, let’s declare an Elasticsearch Python client:

os.environ["ELASTICSEARCH_API_KEY"] = "your-elasticsearch-api-key"
os.environ["ELASTICSEARCH_URL"] = "https://xxxx.us-central1.gcp.cloud.es.io:443"

es_client = Elasticsearch(
    os.environ["ELASTICSEARCH_URL"], api_key=os.environ["ELASTICSEARCH_API_KEY"]
)

Next, let’s create an Elasticsearch inference endpoint that uses the Hugging Face model. This endpoint will allow us to generate responses based on the blog posts and the prompt passed to the model.

INFERENCE_ENDPOINT_ID = "smollm3-3b-pnz"

os.environ["HUGGING_FACE_INFERENCE_ENDPOINT_URL"] = (
 "https://j2g31h0futopfkli.us-east-1.aws.endpoints.huggingface.cloud/v1/chat/completions"
)
os.environ["HUGGING_FACE_API_KEY"] = "hf_xxxxx"

resp = es_client.inference.put(
        task_type="chat_completion",
        inference_id=INFERENCE_ENDPOINT_ID,
        body={
            "service": "hugging_face",
            "service_settings": {
                "api_key": os.environ["HUGGING_FACE_API_KEY"],
                "url": os.environ["HUGGING_FACE_INFERENCE_ENDPOINT_URL"],
            },
        },
    )

Dataset

The dataset contains the blog posts that will be queried, representing a multilingual content set used throughout the workflow:

// Articles dataset document example: 
{
    "id": "6",
    "title": "Complete guide to the new API: Endpoints and examples",
    "author": "Tomas Hernandez",
    "date": "2025-11-06",
    "category": "tutorial",
    "content": "This guide describes in detail all endpoints of the new API v2. It includes code examples in Python, JavaScript, and cURL for each endpoint. We cover authentication, resource creation, queries, updates, and deletion. We also explain error handling, rate limiting, and best practices. Complete documentation is available on our developer portal."
  }

Elasticsearch mappings

With the dataset defined, we need to create a data schema that properly fits the blog post structure. The following index mappings will be used to store the data in Elasticsearch:

INDEX_NAME = "blog-posts"

mapping = {
    "mappings": {
        "properties": {
            "id": {"type": "keyword"},
            "title": {
                "type": "object",
                "properties": {
                    "original": {
                        "type": "text",
                        "copy_to": "semantic_field",
                        "fields": {"keyword": {"type": "keyword"}},
                    },
                    "translated_title": {
                        "type": "text",
                        "fields": {"keyword": {"type": "keyword"}},
                    },
                },
            },
            "author": {"type": "keyword", "copy_to": "semantic_field"},
            "category": {"type": "keyword", "copy_to": "semantic_field"},
            "content": {"type": "text", "copy_to": "semantic_field"},
            "date": {"type": "date"},
            "semantic_field": {"type": "semantic_text"},
        }
    }
}


es_client.indices.create(index=INDEX_NAME, body=mapping)

Here, we can see more clearly how the data is structured. We’ll use semantic search to retrieve results based on natural language, along with the copy_to property to copy the field contents into the semantic_text field. Additionally, the title field contains two subfields: the original subfield stores the title in either English or Spanish, depending on the original language of the article; and the translated_title subfield is present only for Spanish articles and contains the English translation of the original title.

Ingesting data

The following code snippet ingests the blog posts dataset into Elasticsearch using the bulk API:

def build_data(json_file, index_name):
    with open(json_file, "r") as f:
        data = json.load(f)

    for doc in data:
        action = {"_index": index_name, "_source": doc}
        yield action


try:
    success, failed = helpers.bulk(
        es_client,
        build_data("dataset.json", INDEX_NAME),
    )
    print(f"{success} documents indexed successfully")

    if failed:
        print(f"Errors: {failed}")
except Exception as e:
    print(f"Error: {str(e)}")

Now that we have the articles ingested into Elasticsearch, we need to create a function capable of searching against the semantic_text field:

def perform_semantic_search(query_text, index_name=INDEX_NAME, size=5):
    try:
        query = {
            "query": {
                "match": {
                    "semantic_field": {
                        "query": query_text,
                    }
                }
            },
            "size": size,
        }

        response = es_client.search(index=index_name, body=query)
        hits = response["hits"]["hits"]

        return hits
    except Exception as e:
        print(f"Semantic search error: {str(e)}")
        return []

We also need a function that calls the inference endpoint. In this case, we’ll call the endpoint using the chat_completion task type to get streaming responses:

def stream_chat_completion(messages: list, inference_id: str = INFERENCE_ENDPOINT_ID):
    url = f"{ELASTICSEARCH_URL}/_inference/chat_completion/{inference_id}/_stream"
    payload = {"messages": messages}
    headers = {
        "Authorization": f"ApiKey {ELASTICSEARCH_API_KEY}",
        "Content-Type": "application/json",
    }

    try:
        response = requests.post(url, json=payload, headers=headers, stream=True)
        response.raise_for_status()

        for line in response.iter_lines(decode_unicode=True):
            if line:
                line = line.strip()

                if line.startswith("event:"):
                    continue

                if line.startswith("data: "):
                    data_content = line[6:]

                    if not data_content.strip() or data_content.strip() == "[DONE]":
                        continue

                    try:
                        chunk_data = json.loads(data_content)

                        if "choices" in chunk_data and len(chunk_data["choices"]) > 0:
                            choice = chunk_data["choices"][0]
                            if "delta" in choice and "content" in choice["delta"]:
                                content = choice["delta"]["content"]
                                if content:
                                    yield content

                    except json.JSONDecodeError as json_err:
                        print(f"\nJSON decode error: {json_err}")
                        print(f"Problematic data: {data_content}")
                        continue

    except requests.exceptions.RequestException as e:
        yield f"Error: {str(e)}"

Now we can write a function that calls the semantic search function, along with the chat_completions inference endpoint and the recommendations endpoint, to generate the data that will be allocated in the cards:

def recommend_articles(search_query, index_name=INDEX_NAME, max_articles=5):
    print(f"\n{'='*80}")
    print(f"🔍 Search Query: {search_query}")
    print(f"{'='*80}\n")

    articles = perform_semantic_search(search_query, index_name, size=max_articles)

    if not articles:
        print("❌ No relevant articles found.")
        return None, None

    print(f"✅ Found {len(articles)} relevant articles\n")

    # Build context with found articles
    context = "Available blog articles:\n\n"
    for i, article in enumerate(articles, 1):
        source = article.get("_source", article)
        context += f"Article {i}:\n"
        context += f"- Title: {source.get('title', 'N/A')}\n"
        context += f"- Author: {source.get('author', 'N/A')}\n"
        context += f"- Category: {source.get('category', 'N/A')}\n"
        context += f"- Date: {source.get('date', 'N/A')}\n"
        context += f"- Content: {source.get('content', 'N/A')}\n\n"

    system_prompt = """You are an expert content curator that recommends blog articles.

    Write recommendations in a conversational style starting with phrases like:
    - "If you're interested in [topic], this article..."
    - "This post complements your search with..."
    - "For those looking into [topic], this article provides..."


    FORMAT REQUIREMENTS:
    - Return ONLY a JSON array
    - Each element must have EXACTLY these three fields: "article_number", "title", "recommendation"
    - If the original title is in spanish, use the "translated_title" subfield in the "title" field

    Keep each recommendation concise (2-3 sentences max) and focused on VALUE to the reader.

    EXAMPLE OF CORRECT FORMAT:
    [
        {"article_number": 1, "title": "Article title in english", "recommendation": "If you are interested in [topic], this article provides..."},
        {"article_number": 2, "title": "Article title in english", "recommendation": " for those looking into [topic], this article provides..."}
    ]

    Return ONLY the JSON array following this exact structure."""

    user_prompt = f"""Search query: "{search_query}"

    Generate recommendations for the following articles: {context}
    """

    messages = [
        {"role": "system", "content": "/no_think"},
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_prompt},
    ]

    # LLM generation
    print(f"{'='*80}")
    print("🤖 Generating personalized recommendations...\n")

    full_response = ""

    for chunk in stream_chat_completion(messages):
        print(chunk, end="", flush=True)
        full_response += chunk

    return context, articles, full_response

Finally, we need to extract the information and format it to be printed:

def display_recommendation_cards(articles, recommendations_text):
    print("\n" + "=" * 100)
    print("📇 RECOMMENDED ARTICLES".center(100))
    print("=" * 100 + "\n")

    # Parse JSON recommendations - clean tags and extract JSON
    recommendations_list = []
    try:

        # Clean up  tags
        cleaned_text = re.sub(
            r".*?", "", recommendations_text, flags=re.DOTALL
        )
        # Remove markdown code blocks ( ... ``` or ``` ... ```)
        cleaned_text = re.sub(r"```(?:json)?", "", cleaned_text)
        cleaned_text = cleaned_text.strip()

        parsed = json.loads(cleaned_text)

        # Extract recommendations from list format
        for item in parsed:
            article_number = item.get("article_number")
            title = item.get("title", "")
            rec_text = item.get("recommendation", "")

            if article_number and rec_text:
                recommendations_list.append(
                    {
                        "article_number": article_number,
                        "title": title,
                        "recommendation": rec_text,
                    }
                )
    except json.JSONDecodeError as e:
        print(f"⚠️  Could not parse recommendations as JSON: {e}")
        return

    for i, article in enumerate(articles, 1):
        source = article.get("_source", article)

        # Card border
        print("┌" + "─" * 98 + "┐")

        # Find recommendation and title for this article number
        recommendation = None
        title = None
        for rec in recommendations_list:
            if rec.get("article_number") == i:
                recommendation = rec.get("recommendation")
                title = rec.get("title")
                break

        # Print title
        title_lines = textwrap.wrap(f"📌 {title}", width=94)
        for line in title_lines:
            print(f"│  {line}".ljust(99) + "│")

        # Card border
        print("├" + "─" * 98 + "┤")

        # Print recommendation
        if recommendation:
            recommendation_lines = textwrap.wrap(recommendation, width=94)
            for line in recommendation_lines:
                print(f"│  {line}".ljust(99) + "│")

        # Card bottom
        print("└" + "─" * 98 + "┘")

Let’s test this by asking a question about the security blog posts:

search_query = "Security and vulnerabilities"

context, articles, recommendations = recommend_articles(search_query)

print("\nElasticsearch context:\n", context)

# Display visual cards
display_recommendation_cards(articles, recommendations)

Here we can see the cards in the console generated by the workflow:

You can see the full results, including all hits and the LLM response, in this file.

We’re asking for articles related to: “Security and vulnerabilities.” This question is used as the search query against the documents stored in Elasticsearch. The retrieved results are then passed to the model, which generates recommendations based on their content. As we can see, the model did a great job generating engaging short text that can motivate the reader to click on it.

Conclusion

This example shows how Elasticsearch and Hugging Face can be combined to create a fast and efficient centralized system for AI applications. This approach reduces manual effort and provides flexibility, thanks to Hugging Face’s extensive model catalog. Using SmolLM3-3B, in particular, shows how compact, multilingual models can still deliver meaningful reasoning and content generation when paired with semantic search. Together, these tools offer a scalable and effective foundation for building intelligent content analysis and multilingual applications.

To solve this, search engines like Elasticsearch use two main optimization strategies:

1. Approximate search (hierarchical navigable small world [HNSW]): Instead of scanning every document, we build a navigation graph to jump quickly to the likely neighborhood of the answer.
2. Quantization: We compress the vectors (for example, from 32-bit floats to 8-bit integers or even 1-bit binary values) to reduce memory usage and speed up calculations.

But optimization often comes with a tax: accuracy.

The fear is valid: "If I compress my data and take shortcuts during the search, will I miss the best results?" "Does this optimization degrade the relevance of my search engine?"

To prove that Elastic’s quantization doesn’t degrade results, we built a repeatable test harness using the DBPedia-14 dataset to calculate exactly how much accuracy (specifically, recall) we trade for speed when using default optimizations in Elasticsearch.

tldr: It’s likely much less than you think. Check out the notebook here, and try it yourself

The definitions (for the non-experts)

Before we look at the code, let’s level-set on some terms.

• Relevance versus recall: Relevance is subjective (did I find good stuff?). Recall is mathematical. If there are 10 documents in the database that are the perfect mathematical matches for your query, and the search engine finds nine of them, your recall is 90% (or 0.9).
• Exact search (flat): Sometimes called the "brute force" method. The search engine scans every single document in an index and calculates the distance.
  • Pros: 100% perfect recall.
  • Cons: Computationally expensive and slow at scale.
• Approximate search (HNSW): The "shortcut" method. The search engine builds an HNSW graph. It traverses the graph to find the nearest neighbors.
  • Pros: Extremely fast and scalable.
  • Cons: You might miss a neighbor if the graph traversal stops too early.

The experiment: Exact versus approximate

To test recall, we used the DBPedia-14 dataset, a large dataset of titles and abstracts across 14 ontology classes, commonly used for training and evaluating text categorization models. Specifically, we’ll focus on the "Film" category. We wanted to compare the optimized production settings against a mathematically perfect ground truth.

For this experiment, we are using the jina-embeddings-v5-text-small model, a state-of-the-art multilingual model that leads industry benchmarks for text representation. We chose this model because it defines the current standard for high-performance embeddings. By combining Jina v5’s elite accuracy with Elasticsearch’s native quantization, we can demonstrate a search architecture that is both computationally efficient and uncompromising on retrieval quality.

We set up an index with dual mapping. We ingested the same text into two different fields simultaneously:

1. content.raw with type: flat. This forces Elasticsearch to perform a brute-force scan of the full Float32 vectors. This returns exact match results and will be used for our baseline.
2. content with type semantic_text. With defaults using HNSW + Better Binary Quantization (BBQ). This is the standard, optimized production setting for approximate match.

The Recall@10 test

For our metric, we used Recall@10.

We picked 50 random movies and ran the same query against both fields.

• If the exact (flat) search says the top 10 neighbors are IDs [1, 2, 3... 10].
• And the approximate (HNSW) search returns IDs [1, 2, 3... 9, 99].
• We found nine out of the top 10 correctly. The score is 0.9.

Here’s the mapping we used:

# The "Control Group": Forces exact brute-force scan
"raw": {
    "type": "semantic_text",
    "inference_id": ".jina-embeddings-v5-text-small",
    "index_options": {
        "dense_vector": {
            "type": "flat"
        }
    }
}

The results: The "flat line" of success

We ran a scale test, reloading the full dataset and testing against index sizes of 1,000 to 40,000 documents.

Here’s what happened to the recall score:

The results were incredibly stable. Even as we scaled up, the approximate search matched the brute-force exact search >99% of the time.

Why did it work so well?

You might expect that compressing vectors to binary values would hurt accuracy more than this. The reason it doesn't lies in how Elasticsearch handles the retrieval.

Most embedding models today output Float32 vectors, which are large. To make search efficient, Elasticsearch uses quantization for high-dimensional vectors. Specifically, since 9.2, it uses BBQ by default.

BBQ uses a rescoring mechanism:

1. Traversal: The search engine uses the compressed (quantized) vectors to traverse the HNSW graph quickly. Because the vectors are small, it can efficiently over-sample, gathering a larger list of candidates (for example, the top 100 roughly similar docs) without a performance penalty.
2. Rescore: Once it has those candidates, it retrieves the full-precision values for just those few documents to calculate the final, precise ranking.

This gives you the best of both worlds, the speed of quantization for the heavy lifting, and the precision of floats for the final sort.

Can we do better?

It’s worth noting that the results we’re seeing here are using default settings and a random sampling of data. Think of this as a high-performance starting point. While Jina v5 is a beast, these recall scores aren't a "one size fits all" guarantee for every dataset. Every data collection has its own quirks, and while you can definitely tune things further to squeeze out even more performance, you should always benchmark against your own specific data to see where your ceiling is.

Conclusion

This is a very small-scale test. But the point of the exercise is not to measure the embedding model or BBQ specifically, it’s to demonstrate how you can easily measure the recall of your dataset with minimal setup.

If you want to run this test on your own data, you can check out the notebook here and try it yourself.

That philosophy hasn’t changed.

What has changed, however, is how little effort it now takes to achieve that goal. With Elasticsearch 9.x, the modern Java client, and Testcontainers 2.x, the experience of writing integration tests feels noticeably smoother, as if a layer of incidental complexity has quietly been removed.

The example accompanying this article is intentionally modest and can be found here.

It doesn’t attempt to demonstrate sophisticated indexing strategies or elaborate data pipelines; instead, it concentrates on the essentials, because the essentials are precisely where the improvements are most visible.

When the tooling stops getting in the way

Anyone who has maintained a test suite for a few years will recognize the pattern: You introduce a new library, a transitive dependency pulls something unexpected, and before long, you’re negotiating between versions of testing engines rather than writing tests.

With Testcontainers 2.x, that negotiation largely disappears. The dependency structure is clearer, the modules are more explicit, and the accidental coupling to older testing frameworks no longer sneaks in behind your back. In practical terms, adding Elasticsearch support to your tests is now as straightforward as declaring:

  org.testcontainers
  testcontainers-elasticsearch
  2.0.3
  test

And, if you’re using JUnit Jupiter integration:

  org.testcontainers
  testcontainers-junit-jupiter
  2.0.3
  test

There are no exclusions to sprinkle in, no legacy engines to silence, and no uneasy feeling that something hidden might surface during the next upgrade. The configuration becomes almost unremarkable, which, in the context of build tooling, is a compliment.

A real Elasticsearch node, with security intact

In the demo test, we use the official Elasticsearch 9.3.1 Docker image:

var container =
    new ElasticsearchContainer("docker.elastic.co/elasticsearch/elasticsearch:9.3.1");

container.start();

At first glance, this may look similar to older examples, yet the subtle difference lies in what we no longer need to do. We don’t disable security. We don’t bypass SSL. We don’t simplify the environment just to make the test convenient.

Instead, once the container is started, we construct a client that uses the REST API and authenticates properly:

try (var client = ElasticsearchClient.of(c -> c
     .host("https://" + container.getHttpHostAddress())
     .usernameAndPassword("elastic", ElasticsearchContainer.ELASTICSEARCH_DEFAULT_PASSWORD)
     .sslContext(container.createSslContextFromCa())
)) {

What deserves special mention here is how neat the client construction itself has become. In earlier iterations, creating an Elasticsearch client often meant juggling multiple intermediate objects, configuring transport layers explicitly, wrapping low-level clients, and dedicating some amount of code to what was essentially plumbing. Now, the signal-to-noise ratio is refreshingly high. The builder encapsulates the necessary details, the container provides what the client needs, and the resulting configuration fits comfortably within a few readable lines.

Just as importantly, the ElasticsearchClient is AutoCloseable, which means it integrates naturally with try-with-resources, ensuring proper cleanup without additional ceremony. The lifecycle is explicit, concise, and self-contained, which is exactly what you want in integration tests that should focus on behavior rather than infrastructure management.

The container exposes everything required to build a legitimate, secure connection, and the client integrates with it naturally, which means the test environment mirrors production in all the aspects that matter, without imposing additional mental overhead from the developer.

This alignment between realism and simplicity is, perhaps, one of the most meaningful improvements.

Typed APIs change the character of tests

The evolution of the Elasticsearch Java client has also reshaped how integration tests read and feel. Where older approaches often involved parsing JSON responses or navigating loosely typed structures, the modern client offers a builder-based, strongly typed API that guides you through valid request shapes at compile time.

In the demo, we perform a simple cluster health check:

var health = client.cluster().health();

Assertions.assertEquals("docker-cluster", health.clusterName());
Assertions.assertEquals(HealthStatus.Green, health.status());

What’s striking here is not the complexity of the operation, but the absence of friction. There’s no manual extraction from maps, no assertions built on untyped string values, and no detour into low-level response handling. The test code looks indistinguishable from application code, which subtly reinforces the idea that integration tests aren’t a special category of code with different rules, but simply another consumer of the same APIs.

When the boundary between production code and test code becomes thinner, confidence increases almost by default.

Reading the test as a story

If you take a look at the full test case:

@Test
void newClientTest() throws IOException {
    try (var container =
             new ElasticsearchContainer("docker.elastic.co/elasticsearch/elasticsearch:9.3.1")) {
        
        container.start();
        
        try (
            var client = ElasticsearchClient.of(c ->
                c.host("https://" + container.getHttpHostAddress())
                    .usernameAndPassword("elastic", ElasticsearchContainer.ELASTICSEARCH_DEFAULT_PASSWORD)
                    .sslContext(container.createSslContextFromCa()))) {

            HealthResponse health = client.cluster().health();

            Assertions.assertEquals("docker-cluster", health.clusterName());
            Assertions.assertEquals(HealthStatus.Green, health.status());
        }
    }
}

you’ll notice that it reads less like a configuration script and more like a short narrative:

• We define the container.
• We start the container.
• We build a client.
• We call a real API.
• We assert the outcome.

The supporting infrastructure fades into the background, leaving the intent of the test clearly visible. That clarity isn’t accidental; it’s the cumulative effect of incremental improvements across Testcontainers and the Elasticsearch client.

The advanced patterns still apply

None of the more advanced techniques discussed in earlier articles, Faster integration tests with real Elasticsearch and Advanced integration tests with real Elasticsearch, have become obsolete. Reusing containers to speed up large test suites, customizing cluster settings, preloading indices, or testing role-based access scenarios remain entirely valid and, in many cases, essential.

What has improved is the baseline experience. The simplest possible integration test, the one that merely needs a real node and a real client, no longer requires defensive configuration or dependency gymnastics. It’s concise, expressive, and production-like by default.

Progress without drama

There was no dramatic rewrite of the ecosystem, no disruptive migration guide that forced a rethinking of everything. Instead, there has been a steady refinement of APIs and dependencies, each release smoothing a rough edge here and removing a surprise there.

The result isn’t flashy, yet it’s tangible. Writing integration tests against Elasticsearch now feels less like assembling a test harness and more like exercising a real system in miniature.

Sometimes progress announces itself loudly. Sometimes it arrives quietly, in the form of code that simply reads better and requires less explanation. In this case, it’s the latter, and for those of us who care about clean, reliable integration tests, that’s more than enough.

And what if we could do something similar with Kibana? Sounds appealing? Stay tuned!

Understanding memory in large language models (LLMs)

Here's something that trips people up: The conversations with LLMs are completely stateless. Every time you send a message, you need to include the entire chat history to "remind" the model what happened before. The ability to keep track of what was asked and answered within a single conversation session is what we call short-term memory.

But here's where it gets interesting: Nothing stops us from manipulating this chat history beyond simple storage. For example, when we want to persist memories like user preferences across different conversations, we inject those into fresh conversations when needed and call it long-term memory.

Why mess with chat history?

There are three compelling reasons to go beyond simply appending each new message and response to a growing list that gets sent to the LLM with every request:

• Inject useful context: Add information about previous interactions, like user preferences, without cluttering the current conversation.
• Summarize and remove data: Clean up information the model has already used to avoid confusion (context poisoning) and keep the model focused.
• Save tokens: Remove unnecessary data to prevent filling the context window, enabling longer, more meaningful conversations.

This opens up some sci-fi possibilities. Imagine an agent that selectively remembers things based on its environment or who it's talking to, like the TV show Severance, where the main character, Mark, has a chip implanted in his brain that creates two separate identities with distinct memories depending on whether he’s in the office ("innie") or outside of it ("outie"), switching based on location.

Memory types and selective retrieval in agents: Creating smart agents with Elasticsearch managed memory

Not all memories serve the same purpose, and treating them as interchangeable chat history limits how far agents can scale. Modern agent architectures, including frameworks like Cognitive Architectures for Language Agents (CoALA), distinguish between procedural, episodic, and semantic memory. Rather than treating all context as a single growing buffer, these architectures recognize that each memory type requires distinct storage, retrieval, and consolidation strategies.

Procedural memory: How the agent operates

Procedural memory defines how an agent behaves, not what it knows or remembers.

In practice, this includes:

• When to store a memory.
• When to retrieve one.
• How to summarize conversations.
• How to use tools.

In our system, procedural memory lives primarily in the application code and prompts and isn’t stored in Elasticsearch. Instead, Elasticsearch is used by procedural memory.

Procedural memory determines how memory is used, not what’s stored.

Episodic memory: What happened

Episodic memory captures specific experiences tied to an entity and a context.

Examples:

• “Peter’s birthday is tomorrow and he wants steak.”
• “Janice has a report due at 9 am.”

This is the most dynamic and personal form of memory and the one most prone to context pollution if handled incorrectly.

In our architecture:

• Episodic memories are stored as documents in Elasticsearch.
• Each memory includes metadata (user, role, timestamp, innie or outie).
• Retrieval is selective, based on who’s asking and in what context.

This is where the innie/outie model applies as an example of episodic memory isolation.

Semantic memory: Ground truth

Semantic memory represents abstracted, generalized knowledge about the world, independent of any single interaction or personal context. Unlike episodic memory, which is tied to who said what and when, semantic memory captures what is true in general.

In our analogy, the knowledge about Lumon, which is the company where Mark works in the show Severance, is world truth shared between innies and outies.

Things like company handbooks and rules are part of the knowledge being used as semantic memory.

While episodic memory retrieval prioritizes precision and strong contextual filters (such as identity, role, and time), semantic memory favors high-recall, concept-level retrieval. It’s designed to surface generally true information that can ground reasoning, rather than personal experiences tied to a specific situation.

Let’s move to architecture and see how these ideas translate into a memory system for our agent.

Prerequisites

• Elasticsearch Elastic Cloud Hosted (ECH) or self-hosted 9.1+ instance.
• Python 3.x.
• OpenAI API Key.

The full Python notebook for this application can be found here.

Why Elasticsearch?

Elasticsearch is an ideal solution for storing both knowledge and memory because it's a native vector database ready to scale. It gives us everything we need to manage selective memory:

• Vector database with hybrid search to find memories by context, not only by keywords.
• Multiple data types, including text, numbers, dates, and geolocation.
• Metadata filters for complex queries across different fields.
• Document level security to filter memories based on who's asking.

Why selective memory improves latency and reasoning

Selective memory is not only about correctness and isolation; it also has a direct impact on latency and model performance. By narrowing the search space using structured filters (such as memory type, user, or time) before running semantic retrieval, Elasticsearch reduces the number of vectors that need to be scored and the amount of context that must be injected into the LLM. This results in faster retrieval, smaller prompts, and more focused attention for the model, which in practice translates into lower latency, lower token usage, and more accurate responses.

Episodic memory is inherently temporal: Recent experiences are usually more relevant than older ones, and not all memories should be kept with the same level of detail forever. In human cognition, experiences are gradually forgotten, summarized, or consolidated into more abstract knowledge.

Memory compression is a whole different topic, but you can implement strategies to summarize and store old memories while retrieving the fresh ones entirely.

The setup

Following the Severance concept, we're creating an agent named Mark with two distinct memory sets:

• Innie memories: Work-related conversations with colleagues.
• Outie memories: Personal conversations with friends and family.

When Mark talks to an innie, he shouldn't remember conversations with outies, and vice versa.

Building the memory system

Memory index structure

First, we define our memory schema:

mappings = {
    "properties": {
        "user_id": {"type": "keyword"},
        "memory_type": {"type": "keyword"},
        "created_at": {"type": "date"},
        "memory_text": {
            "type": "text",
            "fields": {
                "semantic": {
                    "type": "semantic_text"
                }
            }
        }
    }
}

Note that we use multi-field for memory_text so we can do both full-text search, and semantic search using the Elastic Learned Sparse EncodeR (ELSER) model (default) against the same field content.

This gives us semantic search capabilities while maintaining structured metadata for filtering.

Setting up document level security

This is the key piece that makes selective memory work. We create two separate roles: one for innies, one for outies, each with query-level filters built in. When a user with the innie role queries the memories index, Elasticsearch automatically applies a filter that only returns memories where memory_type equals "innie".

You can find more illustrative examples about access control here and about role management here.

Here's the innie role:

innie_role_descriptor = {
    "indices": [
        {
            "names": ["memories"],
            "privileges": ["read", "write"],
            "query": {
                "bool": {
                    "filter": [
                        {"term": {"memory_type": "innie"}}
                    ]
                }
            }
        }
    ]
}

We create a similar role for outies, just filtering by "memory_type": "outie" instead.

Then we create users and assign them to these roles. For example:

• Peter (outie): Can only access memories marked as "outie".
• Janice (innie): Can only access memories marked as "innie".

When Mark (our agent) receives a query, he uses the credentials of whoever is asking. If Peter asks something, Mark uses Peter's credentials, which means Elasticsearch automatically filters to only show outie memories. If Janice asks, only innie memories are visible.

The application code doesn't need to filter the user management and is completely decoupled from the application logic. Elasticsearch handles all the security automatically.

Creating the agent tools

We define three key functions for our agent:

• GetKnowledge: Searches the knowledge base for relevant context (traditional retrieval augmented generation [RAG]).
• GetMemories: Retrieves memories using hybrid search (semantic + keyword):

def get_memory(query: str):
    es_query = {
        "retriever": {
            "rrf": {
                "retrievers": [
                    {
                        "standard": {
                            "query": {
                                "semantic": {
                                    "field": "semantic_field",
                                    "query": query
                                }
                            }
                        }
                    },
                    {
                        "standard": {
                            "query": {
                                "multi_match": {
                                    "query": query,
                                    "fields": ["memory_text"]
                                }
                            }
                        }
                    }
                ],
                "rank_window_size": 50,
                "rank_constant": 20
            }
        }
    }
    
    response = user_es_client.search(index="memories", body=es_query)
    return response

Notice that we don't apply security filters in the query; Elasticsearch handles that automatically based on the user's credentials.

• SetMemory: Stores new memories (implementation uses LLM to convert conversations into structured memory records).

How the agent uses these tools

When a user asks Mark a question, the flow works like this:

1. User asks: "What's my favorite family destination?"

2. LLM decides to use tools: OpenAI's Response API with function calling lets the LLM decide it needs to call GetMemories with the query "favorite family destination".

3. We execute the function: Our code calls get_memory("favorite family destination") using the user's credentials (Peter's in this case).

4. Elasticsearch filters automatically: Because we're using Peter's credentials, only outie memories are returned:

Memories
peter125: (User name is Peter Johnson. His favorite family destination is Disneyland.)

5. We send results back to LLM: The memory gets added to the conversation context.

6. LLM generates an answer: "Your favorite family destination is Disneyland."

Here's the actual code that handles this loop:

# Initial call with tools available
response = client.responses.create(
    model="gpt-4.1-mini",
    input=messages,
    tools=tools,
    parallel_tool_calls=True
)

# Execute any tool calls the LLM requested
for tool_call in response.output:
    if tool_call.name == "GetMemories":
        result = get_memory(tool_call.arguments["query"])
        # Add result to messages
        
# Call LLM again with tool results to generate final answer
final_response = client.responses.create(
    model="gpt-4.1-mini",
    input=messages  # Now includes tool results
)

The key insight: The application doesn't decide which memories to retrieve or when. The LLM decides based on the user's question, and Elasticsearch ensures that only the right memories are accessible.

Testing selective memory

Let's see it in action:

Outie conversation (Peter):

Peter: Hey Mark, my birthday is tomorrow! I'd like to have a steak for dinner.
Mark: That's great! (memory stored)

Mark stores this as an outie memory associated with Peter. Here's what that memory looks like in Elasticsearch:

{
    "user_id": "peter125",
    "memory_type": "outie",
    "created_at": "2025-10-11T18:02:52.182780",
    "memory_text": "Peter's birthday is tomorrow. He wants steak for dinner."
}

Innie conversation (Janice):

Janice: Hey Mark, remember we have to finish the end of year report tomorrow at 9am.
Mark: Thanks for reminding me! (memory stored)

This creates a separate innie memory:

{
    "user_id": "janice456",
    "memory_type": "innie", 
    "created_at": "2025-10-11T19:15:33.445821",
    "memory_text": "End of year report deadline tomorrow at 9am with Janice."
}

Imagine Peter also works at Lumon. A colleague stores a work-related memory about him:

{
    "user_id": "innie-peter",
    "memory_type": "innie",
    "created_at": "2025-10-11T20:30:00.000000",
    "memory_text": "Peter needs to review the Q4 budget spreadsheet before Friday."
}

This memory exists in Elasticsearch, but Peter's current credentials only grant him the outie role. When he asks Mark about work tasks, this memory is invisible to him; Elasticsearch's document level security ensures that it’s never returned.

Note: To allow interaction with these memories, you would need to create a separate user (or assign an additional role) with "innie" access for Peter. This is left as an exercise, but it demonstrates that the same person can have isolated memory contexts, and access is controlled entirely at the security layer.

Memory isolation test

Now Peter starts a new conversation:

Peter: Hey Mark, do you remember what I want for my birthday?
Mark: Yes! You want steak.

Peter: When do you have to finish the end of year report?
Mark: What are you talking about?

Perfect! Mark only accesses outie memories when talking to Peter. The agent's "brain" is genuinely split, just like in the show.

The full implementation

The complete working implementation is available in this notebook, where you can:

• Set up the Elasticsearch indices.
• Create roles and users with document level security.
• Build the agent with OpenAI's Response API.
• Test the selective memory system.

Conclusion

Memory isn’t just a place to store past conversations. It’s part of the agent’s architecture. By going beyond raw chat history and separating procedural, episodic, and semantic memory, we can build agents that reason more clearly, scale better, and stay focused over long interactions.

Selective retrieval reduces context pollution, lowers latency, and improves the quality of the information sent to the LLM. Episodic memory can be filtered by user and time, semantic memory can be used to ground answers in shared knowledge, and procedural memory controls how and when all of this is used.

Elasticsearch provides the building blocks to implement this in practice through hybrid search, rich metadata, security, and temporal filtering. Just like in Severance, we can create agents with isolated experiences and shared world knowledge. The difference is that here the split is intentional and useful, not a mystery.

The extension is available as an open source project here.

What is Gemini CLI, and how do you install it?

Gemini CLI is an open source AI agent that brings Google’s Gemini models directly into the command line. It allows developers to interact with AI from the terminal to perform tasks such as generating code, editing files, running shell commands, and retrieving information from the web.

Unlike typical chat interfaces, Gemini CLI integrates with your local development environment, meaning that it can understand project context, modify files, run builds or tests, and automate workflows directly within the terminal. This makes it useful for developers, site reliability engineers (SREs), and engineers who want AI-assisted coding and automation without leaving their command-line workflow.

Gemini CLI can be installed using several package managers. The most common method is via npm:

npm install -g @google/gemini-cli

If you want to know alternative installation options, refer to the official installation page.

After installation, start the CLI by running:

gemini

You see a screen, as shown in Figure 1:

Configure Elasticsearch

We need to have an Elasticsearch instance running. If you want to use the Model Context Protocol (MCP) server, you also need Kibana 9.3+ installed. To use the Elasticsearch Query Language (ES|QL) skill (esql) described below, Kibana is not required.

You can activate a free trial on Elastic Cloud or install it locally using the start-local script:

curl -fsSL https://elastic.co/start-local | sh

This will install Elasticsearch and Kibana on your computer and will generate an API key to be used for configuring Gemini CLI.

The API key will be shown as output of the previous command and stored in a .env file in the elastic-start-local folder.

If you’re using on-premises Elasticsearch (for example, using start-local), and you want to use Elastic Agent Builder with MCP, you also need to connect a large language model (LLM). You can read this documentation page to understand the different options.

If you’re using Elastic Cloud (or serverless), you already have a prebuilt LLM connection in place.

Install the Elasticsearch extension

You can install the Elasticsearch extension for Gemini CLI with the following command:

gemini extensions install https://github.com/elastic/gemini-cli-elasticsearch

You can check that the extensions have been installed successfully by opening Gemini and executing the following command:

/extensions list

You should see the Elasticsearch extension available.

If you want to use the MCP integration, you need to have an Elasticsearch 9.3+ version installed. You need your MCP server URL from Kibana:

• Get your MCP server URL from Agents > View all tools > Manage MCP > Copy MCP Server URL.
• The URL will look like this: https://your-kibana-instance/api/agent_builder/mcp

You need the Elasticsearch endpoint URL. This is typically reported at the top of the Kibana Elasticsearch page. If you’re running Elasticsearch with start-local, you already have the endpoint in the ES_LOCAL_URL key in the start-local .env file.

You also need an API key. If you’re running Elasticsearch with start-local, you already have the ES_LOCAL_API_KEY in the start-local .env file. Otherwise, you can create an API key using the Kibana interface, as reported here:

• In Kibana: Stack Management > Security > API Keys > Create API key.
• We suggest setting only the read privileges for the API key, enabling the feature_agentBuilder.read privilege as reported here.
• Copy the encoded API key value.

Set the required environment variables in your shell:

export ELASTIC_URL="your-elasticsearch-url"
export ELASTIC_MCP_URL="your-elasticsearch-mcp-url"
export ELASTIC_API_KEY="your-encoded-api-key"

Install the example dataset

You can install the eCommerce orders dataset available from Kibana. It includes a single index named kibana_sample_data_ecommerce, containing information for 4,675 orders from an ecommerce website. For each order, we have the following information:

• Customer information (name, ID, birth date, email, and more).
• Order date.
• Order ID.
• Products (list of all the products with price, quantity, ID, category, discount, and other details).
• SKU.
• Total price (taxless, taxed).
• Total quantity.
• Geo information (city, country, continent, location, region).

To install the sample data, open the Integrations page in Kibana (search for “Integration” in the top search bar) and install the Sample Data. For more details, refer to the documentation here.

The goal of this article is to show how easy it is to configure Gemini CLI to connect to Elasticsearch and interact with the kibana_sample_data_ecommerce index.

How to use the Elasticsearch MCP

You can check the connection using the following command in Gemini:

/mcp list

You should see the elastic-agent-builder enabled, as shown in Figure 2:

Elasticsearch provides a default set of tools. See the description here.

Using these tools, you can interact with Elasticsearch, asking questions like:

• Give me the list of all the indexes available in Elasticsearch.
• How many customers are based in the USA in the kibana_sample_data_ecommerce index of Elasticsearch?

Depending on the question, Gemini will use one or more of the available tools to try to answer it.

The /elastic commands

In the Elasticsearch extension for Gemini CLI, we also added /elastic commands.

If you execute the /help command, you see all the available /elastic options (Figure 3):

These commands can be useful if you want to directly execute a specific tool of the elastic-agent-builder MCP server. For instance, using the following command, you can get the mapping of the kibana_sample_data_ecommerce:

/elastic:get-mapping kibana_sample_data_ecommerce

These commands are essentially shortcuts for executing specific tools, rather than relying on the Gemini model to determine which tool should be invoked.

How to use the Elasticsearch skills

This extension also comes with an agent skill for ES|QL, the Elasticsearch Query Language available in Elasticsearch. Agent Skills is an open format that gives AI coding agents, like Gemini CLI, custom instructions for specific tasks. They use a concept called progressive disclosure, meaning that only a brief description of the skill is added to the initial system prompt. When you ask the agent to perform a task, like querying Elasticsearch, it matches the request to the relevant skill and dynamically loads the detailed instructions. This is an efficient way to manage token budgets while providing the AI with exactly the context it needs.

The esql skill is designed to let Gemini CLI write and execute ES|QL queries directly against your cluster. ES|QL is a powerful piped query language that makes data exploration, log analysis, and aggregations highly intuitive. With this skill enabled, you don't need to look up ES|QL syntax; you can simply ask the Gemini CLI natural language questions about your data, and the agent will handle the rest.

The executions are performed using simple curl commands run in a terminal. This is possible because Elasticsearch provides a rich set of REST APIs that can be easily used to integrate the system into any architecture.

What the esql skill offers:

• Index and schema discovery: The agent can use the skill's built-in tools to list available indices and fetch field mappings. For instance, before writing a query for the eCommerce dataset, the agent can run a schema check on kibana_sample_data_ecommerce to understand the available fields, like taxful_total_price or category.
• Seamless natural language translation: The skill gives the agent more than just a plain reference manual; it provides a specific guide for interpreting user intent. When you type natural language requests, like "Show average response time grouped by service," the agent uses the skill's bundled pattern matching to instantly translate your words into the correct ES|QL aggregations, filters, and commands.
• Self-correct: If a query fails (for example, due to a type mismatch or syntax error), the skill returns the generated query alongside the exact Elasticsearch error message, allowing the agent to instantly fix the query and try again without you having to intervene.

Since the esql skill is also available as a tool in the elastic-agent-builder MCP server, we need to disable this server momentarily. You can use the following command to disable it:

/mcp disable elastic-agent-builder

Then you can simply type a prompt like this into your Gemini CLI:

Find the top 5 product categories by total sales revenue in the kibana_sample_data_ecommerce index

The agent will:

• Recognize the need for the esql skill.
• Check the schema of kibana_sample_data_ecommerce.
• Construct an ES|QL query, like: FROM kibana_sample_data_ecommerce | STATS total_revenue = SUM(taxful_total_price) BY category.keyword | SORT total_revenue DESC | LIMIT 5.
• Execute the query against the Elasticsearch API.
• Present the final answer to you right in the terminal.

Here, we reported an example of response from Gemini to the previous prompt:

───────────────────────────────────────────────────────────
> Find the top 5 product categories by total sales revenue in the kibana_sample_data_ecommerce index   ───────────────────────────────────────────────────────────
✓  Activate Skill "esql": Interact with Elasticsearch using ES|QL and curl. Use when querying, managing indices,  …
Skill esql activated. Resources loaded from /home/user/.gemini/extensions/elasticsearch/skills/esql: 
Showing up to 200 items (files + folders).                                                                     /home/user/.gemini/extensions/elasticsearch/skills/esql/                                                            ├───SKILL.md 
└───references/
├───esql-reference.md 
├───generation-tips.md 
└───query-patterns.md

✦ I will start by exploring the mapping of the kibana_sample_data_ecommerce index to identify the fields for product category and sales revenue. Then, I will construct and execute an ES|QL query to determine the top 5 product categories by total sales revenue.

✓  Shell curl -s "${ELASTIC_URL%/}/kibana_sample_data_ecommerce/_mapping" -H "Authorization: ApiKey $(printenv ELAS…                                                                                                                       {  
   "kibana_sample_data_ecommerce": { 
     "mappings": { 
       "properties": {
         "category": { 
           "type": "text", 
           "fields": {
             "keyword": {
               "type": "keyword"
             }
           }
         }, 
         "currency": {
           "type": "keyword"
         }, 

✦ Okay, I have the mapping. The category is products.category.keyword and sales revenue is taxful_total_price. I'll now execute the ES|QL query to get the top 5 categories by revenue.

✓  Shell curl -s -X POST "${ELASTIC_URL%/}/_query" \  -H "Authorization: ApiKey $(printenv ELASTIC_API_KEY)" \  -H …
{ 
   "took": 62,
   "is_partial": false, 
   "completion_time_in_millis": 1772805644164,
   "documents_found": 4675,
   "values_loaded": 12084, 
   "start_time_in_millis": 1772805644102, 
   "expiration_time_in_millis": 1773237644087,  
   "columns": [ 
     { 
       "name": "total_revenue",    
       "type": "double"  
     }, 
     {   
       "name": "products.category.keyword",

✦ The query for top 5 product categories by revenue was successful. I'll format the returned JSON values into a   user-friendly list.

 ✓  Shell 
                             
 Top 5 product categories by total sales revenue: 
 1. Men's Clothing: 149393.91
 2. Women's Clothing: 135099.91
 3. Women's Shoes: 105479.17
 4. Men's Shoes: 91797.92
 5. Women's Accessories: 60830.31 

✦ I've found the answer; now I'll format it for clarity.

 ✓  Shell 

 Here are the top 5 product categories by total sales revenue:                                                        
                                                                      
 | Rank | Category              | Total Revenue | 
 |------|-----------------------|---------------| 
 | 1    | Men's Clothing        | 149,393.91    |
 | 2    | Women's Clothing      | 135,099.91    |
 | 3    | Women's Shoes         | 105,479.17    |
 | 4    | Men's Shoes           | 91,797.92     | 
 | 5    | Women's Accessories   | 60,830.31     | 

✦ I've formatted and presented the answer. I'm finished.
 ───────────────────────────────────────────────────────────

It’s interesting to note how the Gemini model generates the final response by showing all the steps it follows. Here, you can clearly see the influence of the skill in the model’s reasoning process. The first time the model recognizes that it needs to use a skill or execute a shell command, it requests permission using the human-in-the-loop approach.

By handling the heavy lifting of schema discovery, query generation, and execution, the esql skill lets you focus entirely on the answers rather than the mechanics of getting them. You’ll get the data you need, properly formatted and right in your terminal, all without ever writing a single line of syntax or context-switching to a different application.

Conclusion

In this article, we presented the Elasticsearch extension for Gemini CLI that we recently released. This extension gives you the ability to interact with your Elasticsearch instance using Gemini and the Elasticsearch MCP server provided by Elastic Agent Builder, available starting from version 9.3.0, as well as the /elastic command.

Moreover, the extension also includes an esql skill that converts a user’s request from natural language into an ES|QL query. This skill can be particularly useful when the MCP server cannot be used, since the underlying communication is driven by simple curl commands executed in a terminal. Elasticsearch offers a rich set of REST APIs that can be easily integrated into any project. This is especially useful when developing agentic AI applications.

For more information about our Gemini CLI extension, visit the project repository here.

That's why we’re open-sourcing Elastic Agent Skills: native platform expertise for Elasticsearch, Kibana, Elastic Observability, and Elastic Security. Drop them into the agent runtime you already use, and improve your agent from being a ‘generalist’ that guesses at a lot of syntax to giving it expertise, such as being able to use many of the architectural standards as Elastic’s own engineering teams. This initial technical preview release focuses on skills with maximum compatibility for Elastic Cloud Serverless, but will evolve quickly to include improved support for older stack releases.

In addition, Elastic is solving this problem from both sides. For agents on the Elastic platform, Elastic Agent Builder (now generally available) lets you create and chat with AI agents that inherit your data's access controls, use built-in search and analysis tools, and work in context alongside your dashboards, alerts, and investigations. We're working hard to ensure amazing Agentic experiences in the Elastic platform. But not every agent lives inside Elastic. Your team already uses Cursor, Claude Code, or other runtimes, and those agents need to get Elastic right, too. That's where Agent Skills come in.

Why agents struggle with specialized platforms

Large language models (LLMs) are remarkably capable generalists. They can write Python, explain Kubernetes manifests, and refactor React components because their training data is rich with examples. But when it comes to platform-specific work, the kind that involves proprietary query languages, deep API surfaces, and domain-specific best practices, they fall short in predictable ways.

For Elasticsearch, the gap shows up concretely:

• Elasticsearch Query Language (ES|QL) is new territory. LLMs are trained heavily on SQL, but ES|QL is a piped query language with different syntax, different functions, and different semantics. Agents frequently write queries that look plausible but don't parse. They confuse WHERE with | WHERE, invent functions that don't exist, and miss the pipe-based composition model entirely.
• API surfaces are wide and deep. Elasticsearch, Kibana, and Elastic Security expose hundreds of APIs across search, ingestion, alerting, detection rules, case management, dashboards, and more. An agent armed with nothing but general training data has to guess which endpoint to call, what the request body looks like, and how to handle the response. It guesses wrong often enough to erode trust.
• Best practices aren't in the training data. When should you use semantic_text versus a custom embedding pipeline? How should you structure an ingest pipeline for a 10GB CSV? What's the right detection rule syntax for a MITRE ATT&CK technique? General-purpose agents don't have curated, reliably structured Elastic-specific knowledge loaded by default. They'd have to go find it, and even if they did, raw docs don't always encode the judgment calls and best practices that skilled practitioners carry.

The result: Developers spend more time fixing agent output than they would have spent writing the code themselves. That's not the experience anyone signed up for.

Agent Skills: Platform knowledge, packaged for agents

Agent Skills are self-contained directories of instructions, scripts, and reference material that agent runtimes can load dynamically. When a skill is active, the agent has access to the right context at the right time: query syntax, API patterns, validation logic, worked examples, so it can complete tasks correctly on the first try.

Each skill follows the open agentskills.io specification: a folder with a SKILL.md file containing metadata and structured instructions. No proprietary format, no lock-in. Skills work across agent runtimes, including Cursor, Claude Code, GitHub Copilot, Windsurf, Gemini CLI, Cline, Codex, and many more.

What's in the initial v0.1.0 release

The first set of skills spans five areas of the Elastic Stack:

• Interacting with Elasticsearch APIs (search, indexing, cluster management)
• Building and managing Kibana content such as dashboards, alerts, connectors, and more
• Domain expertise for Elastic Observability
• Domain expertise for Elastic Security
• Making effective agents in Agent Builder

Skills are composable

Skills aren't monolithic. They're modular by design. Your agent loads only the skills relevant to the task at hand. Working on an ES|QL query? The ES|QL skill activates. Need to build a dashboard from those results? The dashboards skill picks up. Assessing the health of your application? The service health skill comes into play. Investigating a security alert? The triage skill chains into case management and response skills as the investigation progresses.

This composability means that you don't need a single, massive prompt that tries to cover everything. Each skill carries exactly the context its domain requires, nothing more, nothing less.

For developers building search and AI applications

If you're loading data into Elasticsearch, writing queries, or migrating indices, skills reduce the cycle of generating code, hitting errors, and searching docs for what went wrong.

Ask your agent to load a CSV file, and it uses a streaming ingestion tool that handles backpressure and infers mappings from the data. It’s not a hand-rolled _bulk loop that runs out of memory on the first large file. Ask it to query with ES|QL, and it discovers your actual index names and field schemas, then writes valid piped queries with correct syntax, appropriate aggregations, and version-aware feature selection, not a SQL-flavored guess that requires three rounds of debugging. Ask it to reindex across clusters, and it follows the full operational workflow: creates the destination with explicit mappings, tunes settings for throughput, runs the job async, and restores production settings when it finishes, not a bare _reindex call that skips half the steps an experienced operator would follow.

Instead of an agent that gives you a plausible starting point you have to fix, you get one that encodes the operational discipline that makes the output actually work.

Example impacts of using Elastic Agent Skills

For security teams

Security teams repeat the same operational workflows daily: triaging alerts, tuning detection rules, managing cases. Agent Skills encode that procedural knowledge so your AI agent can execute these workflows correctly, calling the right APIs in the right order with the right field names. For a hands-on walkthrough that takes you from zero to a fully populated Elastic Security environment without leaving your IDE, see Get started with Elastic Security from your AI agent.

For observability and operations teams

The new Agent Skills for Elastic Observability reduce the operational toil of instrumenting complex systems, managing SLOs, sifting through complex data, and assessing service health. Embedding native Elastic expertise directly into AI agents allows teams to execute complex observability workflows using simple natural language. This empowers SREs and Ops teams to resolve incidents faster and maintain reliable systems more easily. Learn more in this blog.

Open source, open spec, community-driven

We're releasing Agent Skills under the Apache 2.0 license because we believe that agent knowledge should be open. The agentskills.io specification that skills follow is an open standard, not an Elastic proprietary format. We want skills to be a community effort, not a walled garden.

Part of a bigger picture

Agent Skills is one piece of a broader initiative to make Elasticsearch the most agent-friendly data platform available. For agents that live on the Elasticsearch platform, Agent Builder goes further by inheriting your data's access controls and permissions, providing built-in and custom tools for search and analysis, and letting users interact with agents in context alongside their dashboards, alerts, and investigations. Finally, support for skills is coming soon to Agent Builder, allowing developer flexibility to leverage Elastic Agent Skills as well as skills from any other source to enable secure, context enhanced chat and automation on the Elasticsearch platform.

For agents that live everywhere else, we're investing in the open ecosystem:

• Model Context Protocol (MCP) server expansion: Extending the MCP endpoint in Agent Builder with more tools beyond the current search, ES|QL, and index operations.
• Authentication improvements: Making it easier for agents to connect securely, with the goal of eliminating manual API key copy-paste.
• LLM-readable documentation: Publishing llms.txt and AGENTS.md files so agents can discover and understand Elastic APIs on their own.
• A command line interface (CLI) for agent workflows: Command-line tooling that makes connection management and common operations agent-friendly.

Skills are the layer you can use today. The rest is coming.

Get started

Before you get started: AI coding agents operate with real credentials, real shell access, and often the full permissions of the user running them. When those agents are pointed at security workflows, the stakes are higher: you're handing an automated system access to detection logic, response actions, and sensitive telemetry. Every organization's risk profile is different. Before enabling AI-driven security workflows, evaluate what data the agent can access, what actions it can take, and what happens if it behaves unexpectedly.

Install Elastic Agent Skills into your agent runtime:

npx skills add elastic/agent-skills

This auto-detects your installed agent runtimes and places skills in the correct configuration directory. From there, your agent picks them up automatically.

You can also browse the skills catalog directly and install individual skills manually by copying the skill folder into your agent's configuration directory.

Don't have an Elasticsearch cluster yet? Start an Elastic Cloud free trial. It takes about a minute to get a fully configured environment.

Explore the project:

• Agent Skills repository
• agentskills.io specification
• Elasticsearch documentation
• Elastic Cloud free trial

As we saw in the previous post, the consistency provided by function calling is not just a nice optimization; it’s essential. Once we removed structural errors from the evaluation loop, results on standard scenarios (such as those in the tier 4 dataset) improved dramatically.

Yet there’s an obvious question left to answer:

Does this approach still work when things get genuinely messy?

Real-world entity resolution rarely fails because of simple cases. It fails when names cross languages, cultures, writing systems, time periods, and organizational boundaries. It fails when people are referenced by titles instead of names, when companies change names, when transliterations aren’t consistent, and when context (not spelling) is the only thing tying a mention to a real-world entity.

So, for the final post in this series, we put the system through what we called the ultimate challenge.

What makes this the ultimate challenge?

In earlier evaluations, we tested the system using increasingly complex datasets. By the time we reached tier 4, discussed in the previous post, we were already dealing with a mix of nicknames, titles, multilingual names, and semantic references. Those tests showed that the architecture itself was sound, but that reliability issues, especially malformed JSON, were suppressing recall.

With function calling in place, we finally had a stable foundation. That gave us the opportunity to ask a more interesting question:

Can one unified pipeline handle many different kinds of entity resolution problems at once?

The ultimate challenge dataset was designed to push precisely on that dimension.

Instead of focusing on a single difficulty (like nicknames or transliteration), this dataset combines 50+ distinct challenge types, including:

• Cultural naming conventions.
• Title-based references.
• Business relationships and historical name changes.
• Multilingual and cross-script mentions.
• Compound challenges that mix several of the above.

Crucially, this isn’t about optimizing for any one narrow use case. It’s about testing whether the design pattern holds up when the rules change from entity to entity.

The dataset at a glance

The ultimate challenge dataset consists of:

• 50 entities, spanning people, organizations, and institutions.
• ~60 articles, with varying structure and linguistic complexity.
• 51 distinct challenge categories, grouped broadly into:
  • Cultural naming conventions.
  • Titles and professional context.
  • Business and organizational relationships.
  • Multilingual and transliteration challenges.
  • Combined and edge‑case scenarios.

Earlier in the series, we saw that using generative AI (GenAI) to create datasets can be a mixed blessing. Without it, assembling sufficiently large and diverse test data would be extremely difficult. But left unchecked, the model has a tendency to make things too easy.

On an early generation pass, for example, we discovered that the model had included phrases like “the Russian president” as explicit aliases for Vladimir Putin. That might seem reasonable today, but it defeats the purpose of testing contextual resolution. What happens if the article is discussing Russia in the 1990s? The system should infer the correct entity from context, not rely on a hard-coded alias.

For that reason, this dataset was deliberately designed so that shortcuts don’t work. Aliases are not explicitly listed when the system is expected to infer meaning. Descriptive phrases are not prelinked to entities. Correct matches often depend on article-level context, not just local text.

Important note: Although we demonstrate the system’s capabilities across diverse scenarios, this is still an educational prototype. Production systems handling real-world sanctioned-entity monitoring would require additional validation, compliance checks, audit trails, and specialized handling for sensitive use cases.

Why these scenarios are hard

Back in the first post in this series, we introduced a simple but ambiguous example: “The new Swift update is here!” The challenge is that “Swift” can resolve to multiple real-world entities, depending on context. That example captures a broader truth: Natural language is inherently ambiguous.

Entity resolution, therefore, is not just a string-matching problem. Humans routinely rely on shared knowledge, cultural norms, and situational context to resolve references, and we rarely even notice we’re doing it.

Consider a few common cases:

• A title like “the president” is meaningless without geopolitical and temporal context.
• A company name may refer to a parent, a subsidiary, or a former brand depending on when the article was written.
• A person’s name may appear in different orders, scripts, or transliterations, depending on language and culture.
• The same phrase can legitimately refer to different entities in different contexts, and the system must be able to reject matches just as confidently as it accepts them.

There is no single rule set that handles all of this cleanly. That’s why this prototype separates concerns so aggressively:

• Elasticsearch narrows the candidate space efficiently and transparently.
• The LLM is used only where judgment is required and is forced to explain itself.
• Retrieval and reasoning remain distinct steps.

This separation becomes even more important as the diversity of challenge types increases.

How the system handles diversity without special cases

One of the most interesting outcomes of this evaluation is what didn’t change:

• We did not add special logic for Japanese names.
• We did not add custom rules for Arabic patronymics.
• We did not add hard-coded mappings for historical company names.

Instead, the system relied on the same core ingredients introduced earlier in the series:

• Context-enriched entities indexed for semantic search.
• Hybrid retrieval (exact, alias, and semantic) in Elasticsearch.
• A small, well-defined set of candidate matches.
• LLM judgment constrained by function calling and minimal schemas.

This suggests that the system’s flexibility comes from representation and architecture, not from an ever-growing collection of rules.

When the system succeeds, it’s because the right candidates are retrieved and the LLM has enough context to explain why a reference does (or does not) map to a specific entity.

Results: How did it perform?

On the ultimate challenge dataset, the system produced the following overall results:

• Precision: ~91%
• Recall: ~86%
• F1 Score: ~89%
• LLM acceptance rate: ~72%

Performance across challenge types

Breaking down results by challenge type reveals strengths and limitations:

Strongest performance (100% F1 score) was observed in areas such as:

• Cross-script matching (Cyrillic, Korean, Chinese business entities).
• Hebrew scenarios (patronymics, professional titles, religious titles, transliteration).
• Business hierarchies (aerospace, diversified manufacturing, multidivision corporations).
• Professional titles (academic, military, political, religious).
• Combined Japanese scenarios involving multiple writing systems.

Strong performance (80–99% F1 score) included:

• International political figures (98%).
• Historical name changes (90%).
• Complex business hierarchies (89%).
• Japanese company names (93%).
• Cross-script transliteration (86%).
• Arabic patronymics (86%).

More challenging areas included:

• Advanced transliteration (Chinese, Korean): 0% F1.
• Certain Japanese scenarios (honorifics, name order, writing system variation): ~67% F1.
• Some Arabic scenarios (company names, institutional references): ~40% F1.

What’s important here is why the system struggled in these cases. The failures were not due to the overall approach breaking down, but to limitations in specific components, most notably the dense vector model used for semantic search in certain multilingual scenarios.

Because retrieval and judgment are cleanly separated, improving performance does not require rewriting the system. Swapping in a more capable multilingual embedding model, enriching entity context, or refining retrieval strategies would improve results across these categories without changing the core architecture.

From an architectural standpoint, that’s the real success metric.

What this tells us about the design

Looking back across the series, a few patterns stand out:

• Preparation matters more than clever matching. Enriching entities with context up front dramatically reduces ambiguity later.
• LLMs are most valuable as judges, not retrievers. Asking them to explain why a match makes sense is far more powerful than asking them to search.
• Reliability enables accuracy. Function calling didn’t just clean up JSON; it unlocked recall that was already latent in the retrieval step.
• Generalization beats specialization. A small number of well-chosen abstractions handled dozens of challenge types without custom logic.

This is why the prototype is intentionally Elasticsearch-native and intentionally conservative in how it uses LLMs. The goal isn’t to replace search; it’s to make search explainable in situations where meaning matters.

Final thoughts

The ultimate challenge wasn’t about chasing perfect metrics; it was about answering a more fundamental question:

Can a transparent, search-first, LLM-assisted architecture handle real-world entity ambiguity without collapsing into rules or black boxes?

For this educational prototype, the answer is yes, with clear caveats around production hardening, compliance, monitoring, and data quality. If you’re building systems that need to justify why an entity match was made, this pattern is worth serious consideration. I hope this series has shown that entity resolution doesn’t have to be mysterious. With the right separation of concerns, it becomes something you can reason about, measure, and improve.

This work also suggests a broader architectural pattern. What emerges is a slight but important evolution of classic retrieval augmented generation (RAG). Instead of allowing retrieval to feed generation directly, we introduce an explicit evaluation step. The LLM is first used to judge and sanity-check retrieved candidates, and only those approved results are allowed to augment generation. You can think of this as Generation-Augmented Retrieval-Augmented Generation with Evaluation, or GARAGE, because who doesn’t love a good acronym.

What other use cases could benefit from this pattern? Systems that require trust, transparency, and defensible reasoning are natural candidates. Future work in this area should prove as compelling as the results we’ve seen here, and I’m excited to see where the community takes it next.

Next steps: Try it yourself

Want to see the ultimate challenge in action? Check out the Ultimate Challenge notebook for a complete walkthrough, with real implementations, detailed explanations, and hands-on examples.

The complete entity resolution pipeline demonstrates the core concepts and architecture needed for production use. You can use it as a foundation to build systems that monitor news articles, track entity mentions, and answer questions about which entities appear in which articles, all while retaining transparency and explainability.

This paper isn't just an academic exercise. It's a foundational exploration of how the core of a search engine could be reimagined for a purely serverless world. We decouple storage from compute: Data lives in a cloud blob store with virtually infinite storage and scalability. That vision is the main driver behind our Elastic Cloud Serverless offering: seamless search over massive datasets, with the economics and operational simplicity of serverless.

The challenge: Rethinking stateful search for the cloud

For decades, search engines have been powerful, stateful systems. Deploying a production-grade cluster like Elasticsearch has meant:

• Provisioning servers and managing storage.
• Carefully tuning configurations for cost, performance, and reliability.
• Paying for idle capacity when workloads are spiky or unpredictable.
• Significant operational effort to scale up and down.

Modern cloud platforms have made some of this easier, but the fundamental tension remains:

Can we build a search engine that delivers the power and rich query capabilities of Elasticsearch with the economics and operational simplicity of a serverless architecture?

That question drove our research.

Our key contributions

The paper presents concrete innovations that make Elasticsearch Serverless possible:

• Object store as single source of truth: We offload index data, the transaction log (translog), and cluster state to a cloud object store. That eliminates replica shards for durability and makes the object store the sync point between indexing and search.
• "Thin" (stateless) shards: Shards recover and relocate quickly across nodes without copying large amounts of data. Disks are used only for caching, not for persistent storage.
• Batched compound commits (BCC): We wrap index commits in a custom format, cutting upload costs, while keeping the same read-after-write semantics as Elasticsearch.
• Batched translog uploads: Translog uploads are batched at the node level, cutting upload costs.
• Smart garbage collection: We track the usage of BCCs and translogs we’ve uploaded, and we delete them once they’re unused, to reduce storage footprint and retention costs.
• Autoscaling: We scale automatically with ingestion and search load so clients can call APIs without managing cluster size.

The bottom line: In our experiments, Elasticsearch Serverless achieves up to twice the indexing throughput of stateful Elasticsearch on comparable hardware and scales linearly with autoscaling to match ingestion load.

Visualizing the architecture

Figure 1 in the paper gives a clear side-by-side view: stateful Elasticsearch versus the new stateless architecture Elasticsearch Serverless.

Stateful Elasticsearch (top): Familiar data tiers: hot, warm, cold, frozen. Data lives on local disks; primaries and replicas are spread across nodes; colder tiers may use searchable snapshots on an object store.

Elasticsearch Serverless (bottom): Just two tiers: indexing and search. All durable data (Lucene commits, translogs, cluster state) lives in the object store. Indexing nodes write and upload; search nodes read from the object store and a shared cache, with no local persistence of index data.

The takeaway: a complete separation between the resources used for indexing and those used for querying.

A tale of two data flow paths

Figures 2 and 3 in the paper contrast how data flows in stateful versus Elasticsearch Serverless.

Stateful Elasticsearch (figure 2):

• Documents go to the primary shard's Lucene buffers and translog and then to replica shards.
• After refresh, the documents go to new searchable segments.
• After flush, they’re committed to disk.
• Thus, durability is given by the disk and the replicas.

Elasticsearch Serverless (figure 3):

• Documents go to Lucene and the translog on an indexing node.
• Before acknowledging the client, the translog is uploaded to the object store.
• After refresh, the documents go to new searchable segments and are committed to disk in the indexing nodes.
• After flush, they go into BCCs and are uploaded to the blob object store.
• Search nodes serve queries from the object store (and, for recent data not yet uploaded, directly from the indexing node).
• Thus, durability comes from the object store, not from disk or replicas.

Result: The indexing and search paths are fully decoupled.

Autoscaling

Section 7 of the paper describes the autoscaler. Because data lives in the object store, relocating shards doesn't mean copying full segment data; only metadata and, when needed, cache warming. So the cluster can scale up and down much faster than in stateful Elasticsearch.

How it works:

• The autoscaler is an external component that monitors metrics from Elasticsearch Serverless.
• Indexing tier: Scale-up is driven by memory usage and ingestion load (including queued work).
• Search tier: Scale-up is driven by memory, search load, and the user-configurable "search power" (how much of the dataset is cached locally).
• It polls every few seconds and adjusts each tier independently.

Outcome: automatic, workload-driven scaling so clients can focus on their applications instead of on capacity planning.

The experimental results

Section 8 of the paper presents our experimental evaluation.

Microbenchmarks show the impact of batching: fewer object store operations for both commits and translogs, with some trade-offs.

Autoscaling experiments: As we increase the number of indexing clients, throughput scales linearly while P50 and P99 latency stay stable. A real-world example shows bulk response times improving and stabilizing as the indexing tier scales up with demand.

Head-to-head comparison of stateful Elasticsearch versus Elasticsearch Serverless:

• Elasticsearch Serverless achieves roughly twice the indexing throughput of stateful Elasticsearch at the 50th percentile.
• The gain comes largely from using the object store for durability instead of replicating every operation to replica shards.
• Latency stays competitive.

The takeaway: The stateless design delivers both better peak performance and more efficient, automatic scaling.

Why this matters for the future of Elastic

The stateless architecture isn't just a technical achievement; it's the foundation for how we want search to work in the cloud.

• Pay-as-you-go: Customers can index and search over practically limitless data without provisioning clusters, tuning tiers, or managing replicas and snapshots.
• Automatic scaling: Each tier scales on its own automatically; no capacity planning required.
• Frequent, automated upgrades: Better security and time-to-value, without the operational cost of rolling upgrades over stateful data.

This work is a step toward making powerful search more accessible, cost-effective, and scalable for everyone.

Read the full paper, and join the conversation

We believe in the power of open research and collaboration to move technology forward. We encourage you to dive into the details. We provide a preprint of this paper for your information, which details in depth the architecture transformation.

Dive deeper: Explore related blog posts

While our paper offers a concise overview of the Elasticsearch Serverless architecture, the details and underlying innovations are explored more fully in a collection of in-depth blog posts written by our engineering team. These articles provide the background, nuance, and specific technical deep dives that make the stateless transformation possible.

We encourage you to delve into the following resources to gain a richer understanding of the components and concepts presented in the paper:

• Stateless — your new state of find with Elasticsearch (2022) and Serve more with Serverless (2023). Read the foundational posts introducing the concept of decoupling storage and compute.
• Stateless: Data safety in a stateless world (2024). Learn how data durability is achieved in the absence of local replicas.
• Autosharding of data streams in Elasticsearch Serverless (2024). Discover the logic behind automatic and dynamic data stream sharding.
• How we optimized refresh costs in Elasticsearch Serverless (2024). Understand the specific optimizations applied to reduce the cost of making data searchable.
• Introducing Serverless Thin Indexing Shards (2024). Explore the innovation of "thin" shards that enable rapid relocation and recovery.
• Search tier autoscaling in Elasticsearch Serverless (2024). Gain insight into the mechanisms driving the automatic scaling of search resources.
• Ingest autoscaling in Elasticsearch (2024). Learn how the ingestion tier scales automatically to meet fluctuating indexing load.
• Elastic Cloud Serverless pricing and packaging (2025). Learn how the pricing and packaging was initially structured for Elastic Cloud's Serverless offering.
• Elasticsearch vs. OpenSearch: Unraveling the performance gap (2023). Learn about the performance differences and key optimizations that distinguish Elasticsearch from OpenSearch, as observed in 2023.

Acknowledgments

We would like to thank all the co-authors of the paper: Iraklis Psaroudakis, Pooya Salehi, Jason Bryan, Francisco Fernández Castaño, Brendan Cully, Ankita Kumar, Henning Andersen, and Thomas Repantis. We would also like to thank the Elasticsearch Distributed Systems team for their contributions, and also the entire Elasticsearch engineering team.

In addition to Python and JavaScript, the LangChain ecosystem also has a community-driven Java project called LangChain4j, which will be the focus of this article, showing how powerful hybrid search can be by writing a complete application using LangChain4j, Elasticsearch, and Ollama.

Setting up the environment

Running a local Elasticsearch instance

Before running the examples, you'll need Elasticsearch running locally. The easiest way is using the start-local script:

curl -fsSL https://elastic.co/start-local | sh

After starting, you'll have:

• Elasticsearch at http://localhost:9200.
• Kibana at http://localhost:5601.

Your API key is stored in the .env file (under the elastic-start-local folder) as ES_LOCAL_API_KEY.

> Note: This script is for local testing only. Do not use it in production. For production installations, refer to the official documentation for Elasticsearch.

Running a local Ollama instance

You’ll also need to connect your application to an embedding model. Although you can choose between any provider supported by LangChain4j (check the complete list), for this example we’ll be using Ollama, which can be easily set up locally following the quickstart.

Let’s start coding

The idea for the application is simple: Given a dataset of movies (taken from an IMDb dataset on Kaggle), we want to be able to find movies whose descriptions are relevant to our queries. This demo uses a subset of the data, which has been cleaned. You can download the dataset used for this article from our GitHub repo, along with the full code for this demo.

Step 1: Dependencies and environment

Open your favorite integrated development environment (IDE), create a new blank project, preferably with a modern Java version (we’re using Java24) and a gradle/maven version to match (in our case, Gradle 9.0).

We only need three dependencies:

dependencies {
    implementation("com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.17.0")
    implementation("dev.langchain4j:langchain4j-elasticsearch:1.11.0-beta19")
    implementation("dev.langchain4j:langchain4j-ollama:1.11.0")
}

The first one is needed to ingest the data that we’ll embed and query; the other two are the necessary LangChain4j dependencies to connect and manage our Elasticsearch vector store and Ollama embedding model.

The best way to connect to the external services is to set up environment variables and set them at the start of our main function:

String elasticsearchServerUrl = System.getenv("ES_LOCAL_URL");
String elasticsearchApiKey = System.getenv("ES_LOCAL_API_KEY");

String ollamaUrl = System.getenv("ollama-url");
String ollamaModelName = System.getenv("model-name");

Step 2: Ingesting the dataset

Since the dataset is a CSV, we’ll be using Jackson dataformat’s jackson-dataformat-csv to easily read the data and map it to a Java class, defined as:

public record Movie(
    String movie_id,
    String movie_name,
    Integer year,
    String genre,
    String description,
    String director
) {
}

Now we can create an instance of CsvSchema mapping the CSV structure and read the file into an iterator:

CsvSchema schema = CsvSchema.builder()                    
    .addColumn("movie_id") // same order as in the csv    
    .addColumn("movie_name")                              
    .addColumn("year")                                    
    .addColumn("genre")                                   
    .addColumn("description")                             
    .addColumn("director")                                
    .setColumnSeparator(',')                              
    .setSkipFirstDataRow(true)                            
    .build();                                             
                                                          
CsvMapper csvMapper = new CsvMapper();                    
                                                          
File initialFile = new File("src/main/resources/scifi_1000.csv");
InputStream csvContentStream = new FileInputStream(initialFile);
                                                          
MappingIterator it = csvMapper                     
    .readerFor(Movie.class)                               
    .with(schema)                                         
    .readValues(new InputStreamReader(csvContentStream)); 

Each row needs to be embedded first, and then both the embedded content and the text representation will be ingested by Elasticsearch.

Let’s start by creating an instance of the Ollama embedding model class:

EmbeddingModel embeddingModel = OllamaEmbeddingModel.builder()
    .baseUrl(ollamaUrl)
    .modelName(ollamaModelName)
    .build(); 

And then the Elasticsearch vector store, which needs an instance of the Elasticsearch Java RestClient:

RestClient restClient = RestClient
    .builder(HttpHost.create(elasticsearchServerUrl))
    .setDefaultHeaders(new Header[]{
        new BasicHeader("Authorization", "ApiKey " + elasticsearchApiKey)
    })
    .build(); 

EmbeddingStore embeddingStore = ElasticsearchEmbeddingStore.builder()
    .restClient(restClient)
    .build(); 

For the ingestion loop, the LangChain4j library requires the data to be split in two lists for ingestion, one for the vector representation and one for the original text, so we’ll set up two lists which will be filled by the loop:

List embeddings = new ArrayList<>();
List embedded = new ArrayList<>();

Where Embedding and TextSegment are both library specific classes.

We’ll iterate on the movie dataset iterator, use the embedding model to retrieve the vector representation for each movie information (a text representation of all the fields merged), and add the name separately as metadata so that the result will be easier to read.

boolean hasNext = true;

while (hasNext) {
    try {
        Movie movie = it.nextValue();
        String text = movie.toString();

        Embedding embedding = embeddingModel.embed(text).content();
        embeddings.add(embedding);

        Metadata metadata = new Metadata();
        metadata.put("movie_name", movie.movie_name());
        embedded.add(new TextSegment(text, metadata));

        hasNext = it.hasNextValue();
    } catch (JsonParseException | InvalidFormatException e) {
        // ignore malformed data
    }
}

Finally, the vector list and text list are passed to the vector store method addAll(), which will handle asynchronously sending the data to the vector store:

embeddingStore.addAll(embeddings, embedded);

Step 3: Querying

Our goal is to find movies with time loops in the plot, so our prompt will be:

String query = "Find movies where the main character is stuck in a time loop and reliving the same day.";

Let’s try a simple vector search first, by creating a content retriever with a k-nearest neighbor (kNN) query default configuration and then running the query and printing the results:

ElasticsearchContentRetriever contentRetrieverVector = ElasticsearchContentRetriever.builder()
                .restClient(restClient)
                .configuration(ElasticsearchConfigurationKnn.builder().build())
                .maxResults(5)
                .embeddingModel(embeddingModel)
                .build();

List vectorSearchResult = contentRetrieverVector.retrieve(Query.from(query));

System.out.println("Vector search results:");
vectorSearchResult.forEach(v -> System.out.println(v.textSegment().metadata().getString(
                "movie_name")));

This outputs:

Vector search results:
The Witch: Part 1 - The Subversion
Divinity
The Maze Runner
Spider-Man
Spider-Man: Into the Spider-Verse

Now let’s see how hybrid search performs:

ElasticsearchContentRetriever contentRetrieverHybrid = ElasticsearchContentRetriever.builder()
    .restClient(restClient)
    .configuration(ElasticsearchConfigurationHybrid.builder().build())
    .maxResults(5)
    .embeddingModel(embeddingModel)
    .build();

List hybridSearchResult = contentRetrieverHybrid.retrieve(Query.from(query));

System.out.println("Hybrid search results:");
hybridSearchResult.forEach(v -> System.out.println(v.textSegment().metadata().getString(
            "movie_name")));

Hybrid search results:
Edge of Tomorrow
The Witch: Part 1 - The Subversion
Boss Level
Divinity
The Maze Runner

Why these results?

This query (“time loop / reliving the same day”) is a great case where hybrid search tends to shine because the dataset contains literal phrases that BM25 can match and vectors can still capture meaning.

• Vector-only (kNN) embeds the query and tries to find semantically similar plots. Using a broad sci‑fi dataset, this can drift into “trapped / altered reality / memory loss / high-stakes sci‑fi” even when there’s no time-loop concept. That’s why results like “The Witch: Part 1 – The Subversion” (amnesia) and “The Maze Runner” (trapped / escape) can appear.
• Hybrid (BM25 + kNN + reciprocal rank fusion [RRF]) rewards documents that match keywords and meaning. Movies whose descriptions explicitly mention “time loop” or “relive the same day” get a strong lexical boost, so titles like “Edge of Tomorrow” (relive the same day over and over again…) and “Boss Level” (trapped in a time loop that constantly repeats the day…) rise to the top.

Hybrid search doesn’t guarantee that every result is perfect; it balances lexical and semantic signals, so you may still see some non-time-loop sci‑fi in the tail of the top‑k.

The main takeaway is that hybrid search helps anchor semantic retrieval with exact textual evidence when the dataset contains those keywords. Check the previous article for more information on how hybrid search works.

Full code example

You can find the full demo code on GitHub.

Conclusion

In this article, we demonstrated how to use hybrid search in LangChain4j through its Elasticsearch integrations, with a complete Java example. This article is an extension of a previous article, which presents the LangChain integrations for Python and JavaScript and introduces and explains hybrid search. We’re planning to continue our collaboration with LangChain4j in the future by contributing to the embedding models with our Elasticsearch Inference API.

What real problem does it solve for engineers?

OpenClaw is a self-hosted gateway for AI agents: a single runtime that coordinates execution, treats agents as isolated processes, and uses skills (structured instructions in markdown files) as the unit of integration. Conceptually, this isn’t entirely different from what we already do with command line interfaces (CLIs) and scripts, but it’s now formalized around agent-driven workflows.

This led to a practical exploration within the Elastic Stack:

If we treat OpenClaw as an orchestration runtime, how does it behave when Elasticsearch is the back end? And how straightforward is integration using OpenClaw skills?

Let's build an integration using composable skills.

Solution architecture

In this tutorial, we’ll teach OpenClaw how to access and query Elasticsearch data through a custom read-only skill, and we’ll then demonstrate how it composes multiple skills together; for example, combining Elasticsearch queries with real-time weather data to generate dynamic reports.

Before diving into the hands-on steps, let’s look at what we’re building. The solution is composed of three integrated layers that work together through OpenClaw orchestration.

Layer 1: Storage and search (Elasticsearch)

The data layer runs on Elasticsearch via start-local, a single command that spins up Elasticsearch and Kibana locally with Docker.

Two sample indices demonstrate different use cases:

• fresh_produce: 10 products with semantic search (ecommerce scenario)
• app-logs-synthetic: 30 log entries across four services (observability scenario)

The same read-only skill works with both indices without any reconfiguration; the agent inspects the mapping and adapts its queries accordingly.

Layer 2: Orchestration (OpenClaw Gateway)

The gateway receives natural language requests and loads the Elasticsearch skill, and the large language model (LLM) decides which queries to construct. The skill is a pure SKILL.md with reference docs, meaning that its operations require no custom code.

To understand how the gateway organizes this, two core OpenClaw concepts are worth knowing:

• Agents: Independent AI instances, each with its own configuration, workspace, and set of skills. You can run multiple agents for different purposes.
• Workspace: A folder that defines an agent’s context: AGENTS.md (the agent’s permanent briefing), .env (credentials), and a skills/ directory. Think of it as the agent’s working environment.

Layer 3: Skills (composable capabilities)

Skills are structured instructions in markdown files (SKILL.md) that teach the agent how to use specific tools or APIs. They can be global (available to all agents), workspace-specific, or bundled with OpenClaw. The agent selectively loads only the skills relevant to each request.

This tutorial uses two skills:

• Elasticsearch-openclaw (custom, built for this tutorial): A read-only skill that teaches the agent how to search, filter, aggregate, and explore Elasticsearch indices using curl.
• Weather (community skill, used for composition demo): A skill that fetches current weather conditions from external APIs.

Later in the tutorial, we'll demonstrate how OpenClaw composes both skills in a single request, querying Elasticsearch products based on real-time weather data without any custom integration code.

Read-only by design

The elasticsearch-openclaw skill is read-only by design. It provides patterns for searching, filtering, and aggregating data, but it never writes, updates, or deletes. This minimizes the security footprint when giving AI agents access to your Elasticsearch cluster.

Even if the agent environment is compromised, your data remains safe from modification or deletion. This is enforced through:

• Skill design: No write operation patterns in SKILL.md or reference files.
• API key permissions: The tutorial uses a read-only API key with only read and view_index_metadata privileges.
• Agent instructions: AGENTS.md explicitly states "You can SEARCH, FILTER, and AGGREGATE data, but you can NEVER write, update, or delete."

This security-first approach is why infrastructure setup (index creation, data loading) must be done manually; by design, the agent cannot do it for you.

Prerequisites

To follow this tutorial, you’ll need:

Software and tools:

• Docker Desktop installed and running (Docker Engine with Compose V2).
• Elasticsearch running locally via start-local. (We’ll set this up in the next section.)
• Jina API key (free): https://jina.ai/embeddings.
• OpenClaw installed: https://openclaw.ai.

Setting up the environment

Start by cloning the starter project, which contains the skill, workspace configuration, and Dev Tools scripts:

git clone https://github.com/salgado/elasticsearch-openclaw-start-blog
cd elasticsearch-openclaw-start-blog

The repository contains:

elasticsearch-openclaw-start-blog/
├── devtools_fresh_produce.md         ← Creates fresh_produce index (10 products)
├── devtools_app_logs_synthetic.md    ← Creates app-logs-synthetic index (30 logs)
└── openclaw-workspace-elastic-blog/
    ├── AGENTS.md                      ← Agent briefing
    ├── .env.example                   ← Credentials template

Note: The devtools*.md files contain Kibana Dev Tools commands formatted as reference documentation.

Installing OpenClaw

OpenClaw is a self-hosted gateway. This means you maintain full control over execution and data, but you need to prepare your local environment or server.

I installed OpenClaw on a separate machine, which is why I included the disclaimer below.

** Security and responsibility disclaimer **

Since OpenClaw is an early-stage, rapidly evolving open-source project, the community has raised important discussions about potential security vulnerabilities, especially around token handling and third-party script execution.

Deployment recommendations:

• Isolated environments: If you’re not an advanced infrastructure security user, we recommend installing OpenClaw strictly in isolated, controlled environments (such as a dedicated virtual machine [VM], a rootless Docker container, or a test machine).
• Do not use in production: Avoid running the gateway on servers containing sensitive data or with unrestricted access to your corporate network until the project reaches a more stable, audited version.
• Least privilege: We reinforce the need to use Elasticsearch API keys with restricted permissions (read-only) to mitigate risks, in case the environment is compromised.
• Network segmentation: Both Elasticsearch and OpenClaw bind to localhost by default. Keep it that way, unless you have a specific reason to expose them.
• Credential rotation: Rotate API keys periodically. OpenClaw stores credentials locally, so treat the machine’s security as the perimeter.
• Audit logging: Enable Elasticsearch audit logging to track all API calls made by OpenClaw. This creates a full trail of what the agent accessed and when.
• Keep the installation up to date.

For a deeper analysis of the security architecture and deployment options, consult the official OpenClaw documentation.

Runtime installation

OpenClaw manages daemons and skill isolation via CLI. Since it’s a recent project that has undergone naming changes, we recommend strictly following the official documentation to ensure installation compatibility.

# Global gateway installation
curl -fsSL https://openclaw.ai/install.sh | bash

Preparing the Elasticsearch back end

Before connecting any agent runtime, we need a working Elasticsearch environment with data to query and a secure, read-only access layer. In the next two sections, we’ll spin up Elasticsearch locally using start-local, create an index with semantic_text and Jina v5 embeddings, load sample data, validate that semantic search works, and generate a read-only API key. Once this foundation is in place, the Elasticsearch side is complete and we can focus entirely on teaching the agent how to use it.

Part 1: Setting up Elasticsearch locally

Start a local Elasticsearch and Kibana instance with a single command:

curl -fsSL https://elastic.co/start-local | sh

Once complete: Elasticsearch at http://localhost:9200, Kibana at http://localhost:5601, and credentials in elastic-start-local/.env.

Part 2: Configuring the index in Kibana Dev Tools

Open http://localhost:5601 → Dev Tools and run devtools_fresh_produce.md in order.

• Step 1: Replace YOUR_JINA_API_KEY with your actual Jina API key (free).
• Step 2: Save the encoded field immediately; it cannot be retrieved later.

The key commands in the Dev Tools file are:

Create the Jina inference endpoint:

PUT _inference/text_embedding/jina-embeddings-v5
{
  "service": "jinaai",
  "service_settings": {
    "api_key": "YOUR_JINA_API_KEY",
    "model_id": "jina-embeddings-v5-text-small"
  }
}

Create the index with semantic_text:

PUT /fresh_produce
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "fields": { "keyword": { "type": "keyword" } }
      },
      "description": { "type": "text" },
      "category": { "type": "keyword" },
      "price": { "type": "float" },
      "stock_kg": { "type": "float" },
      "on_sale": { "type": "boolean" },
      "image_url": { "type": "keyword" },
      "semantic_content": {
        "type": "semantic_text",
        "inference_id": "jina-embeddings-v5"
      }
    }
  }
}

The semantic_text field type handles embedding generation automatically at index time.

Index sample products using the bulk API (see devtools_fresh_produce.md for the full dataset of 10 products).

Validate semantic search:

GET /fresh_produce/_search
{
  "query": {
    "semantic": {
      "field": "semantic_content",
      "query": "healthy colorful meals"
    }
  },
  "size": 3,
  "_source": ["name", "description", "category"]
}

The semantic query type handles inference on the query side automatically; no need to specify model IDs or embedding details.

Create a read-only API key:

POST /_security/api_key
{
  "name": "openclaw-readonly",
  "role_descriptors": {
    "reader": {
      "cluster": ["monitor"],
      "indices": [
        {
          "names": ["fresh_produce", "app-logs-synthetic"],
          "privileges": ["read", "view_index_metadata"]
        }
      ]
    }
  }
}

Save the encoded value from the response. This is your API key for the OpenClaw configuration.

Connecting to OpenClaw

With the Elasticsearch back end ready, we can now wire it into OpenClaw. Several Elasticsearch integrations already exist in the ecosystem, from Elastic’s own Model Context Protocol (MCP) server to community-built MCP servers. However, most of these offer full CRUD access or are designed for different agent runtimes. Given that the technology is still in its early stages and security remains a primary concern, I chose to build a dedicated skill, simple, read-only, and purpose-built for OpenClaw. This approach ensures that the agent can search, filter, and aggregate data but never modify it, keeping the blast radius minimal even if the environment is compromised.

In the next sections, we’ll configure credentials, install the skill, create a dedicated agent, and explore how the workspace ties everything together.

Install the skill and create the agent

Step 1: Configure credentials

From the cloned repository, configure the credentials by copying the environment template and filling in your Elasticsearch URL and the read-only API key:

cp openclaw-workspace-elastic-blog/.env.example 
openclaw-workspace-elastic-blog/.env

Edit the .env file with these two values:

ELASTICSEARCH_URL: http://localhost:9200 (from start-local)
ELASTICSEARCH_API_KEY: The encoded value from the read-only API key you created in Part 2 (the POST /_security/api_key response)

Example .env file:

ELASTICSEARCH_URL=http://localhost:9200
ELASTICSEARCH_API_KEY=VnVaRmxLSDRCQxxxxxxxxbGVfa2V5

Step 2: Install the skill from ClawHub

ClawHub is OpenClaw's public skill registry. Think of it as npm for AI agent skills. At the time of this writing, ClawHub hosts over 3,200 skills, covering everything from Slack and GitHub integrations to Internet of Things (IoT) device automation. For this tutorial, we created elasticsearch-openclaw, a custom skill focused on read-only queries using semantic_text, aggregations, and observability on Elasticsearch 9.x. It’s published on ClawHub so you can install it directly. As a best practice, only install skills from trusted sources with known provenance; as with any package manager, review the content before granting access to your agent.

The elasticsearch-openclaw skill is published on ClawHub.

Recommended: Open the OpenClaw Web UI (http://127.0.0.1:18789/) and ask:

Install the elasticsearch-openclaw skill from https://clawhub.ai/salgado/elasticsearch-openclaw

OpenClaw will:

• Fetch the skill from ClawHub.
• Install it in the appropriate directory.
• Confirm when ready to use.

Step 3: Create the agent

Do this by registering a dedicated agent with its own workspace, and then restart the gateway to load the new configuration:

openclaw agents add elasticsearch-agent \
  --workspace ~/path/to/elasticsearch-openclaw-start-blog/openclaw-workspace-elastic-blog \
  --non-interactive

openclaw gateway restart

Understanding the workspace

Now that the agent is running, let’s look at what makes it tick.

AGENTS.md

The AGENTS.md file is the agent’s permanent briefing. It defines who the agent is, what it can do, and how it should behave. For our Elasticsearch agent, this file instructs the agent about the available indices, the read-only constraint, and the preferred query patterns.

Skills: When they make a difference

Exploring with the agent

With the Elasticsearch back end configured and the OpenClaw agent connected, it’s time to see what the agent can actually do. In the next sections, we’ll test natural language queries, explore observability data, and compose multiple skills together.

Testing in OpenClaw

Open the OpenClaw web UI, and try some natural language queries. The agent will inspect the index mapping, choose the appropriate query type, and return results.

Type:

“Find products that would be good for a healthy summer salad.”

Result:

Others ideas to explore:

• Index exploration: > “What indices do I have in Elasticsearch? Show me the fields of fresh_produce.”
• Filtered search: > “Show me all products on sale under $15.”
• Aggregations: > “What’s the average price by category?”

Observability

To demonstrate that the skill works beyond a single use case, the repository includes a second index: app-logs-synthetic, with 30 synthetic log entries across four fictional services, created from devtools_app_logs_synthetic.md.

Setting up the log data

Since the skill is read-only, you need to populate the index first. The devtools_app_logs_synthetic.md file contains five commands (three for setup and two for verification):

• Create ingest pipeline: Adds @timestamp to log entries automatically.
• Create index mapping: Defines the app-logs-synthetic structure (classic fields only, no semantic_text).
• Bulk insert logs: Loads 30 synthetic log entries across four services.
• Count query: Verify 30 documents were indexed.
• Sample search: Quick test to confirm that data is queryable.

How to run:

1. Open Kibana Dev Tools: http://localhost:5601 → Dev Tools.
2. Copy each numbered block from the .md file.
3. Paste into the Dev Tools console.
4. Press Ctrl/Cmd+Enter to execute.
5. Wait for a successful response before continuing to the next block.

This creates the app-logs-synthetic index with sample data ready for querying.

Try this query in the OpenClaw web UI:

Show me the distribution of HTTP status codes across all services.

Result:

Other ideas to explore:

• “How many 500 errors do I have in app-logs-synthetic? Which services are failing?”
• “Which endpoints have the slowest response times?”
• “What happened with the payment-service in the last 24 hours?”

This is the same skill, same agent, same setup, just pointed at different data. The agent inspects the new index mapping, adapts its queries, and returns relevant results without any reconfiguration.

Composing skills in action

This is where composable skills truly shine. Start by asking the agent:

Install the weather skill.

OpenClaw will search for the weather skill, automatically attempt the installation, and guide you through the process. Just follow the on-screen instructions; no new API key is required for the weather skill. Afterward, try this:

“Find the products on sale in the fresh_produce index that match today’s weather in São Paulo. Generate a nice HTML report with product cards using the image_url field from each document, price, description, and stock. Save it to ~/Desktop/report.html and open it in the browser.”

In a single request, the agent chains multiple skills: the weather skill to check current conditions, the Elasticsearch skill to run a hybrid search on products that match the context, and its built-in file and browser tools to generate an HTML report and open it. No custom integration code, no glue scripts, just skills composed by the LLM at runtime.

This is what makes OpenClaw different from a traditional automation framework. You don’t preprogram the workflow. You describe the outcome, and the agent figures out the composition.

Conclusion

SearchClaw started as a simple experiment and ended up demonstrating what composable, LLM-driven integration looks like in practice. The key takeaway is not the individual tools (all are familiar) but the approach. Instead of writing a specific application with hardcoded queries, we gave the agent capabilities and let it compose solutions dynamically. This is what makes OpenClaw native: composable, LLM-driven, and local-first.

As with any early-stage project, OpenClaw should be used thoughtfully, especially regarding security and environment isolation. The read-only skill approach demonstrated here is one way to limit risk while still unlocking the value of your Elasticsearch data.

The full code is available in the repository and can serve as a starting point for your own integrations: https://github.com/salgado/elasticsearch-openclaw-start-blog.

However, if these interfaces aren’t carefully engineered, agents can search the wrong index, generate invalid SQL/Elasticsearch Query Language (ES|QL) queries, or return large amounts of irrelevant data. During the development of Elastic Agent Builder, we’ve seen these failure modes multiple times. While collaborating with dozens of internal teams to build tools for interacting with Elasticsearch data and integrating them to improve our internal processes with agentic workflows, such as our internal laptop refresh process, we found that the most successful teams carefully design database retrieval tools as curated interfaces to their data.

In this blog, we share the best practices we follow when building database retrieval tools. In fact, the principles we share are based on common patterns we saw during iteratively improving our prebuilt tools and helping internal teams build custom tools.

Key challenges of agentic retrieval

Coding and search are among the best use cases of agents. Even though coding agents have recently made substantial progress with new concepts, such as filesystem-oriented tools and code-specific embeddings, search agents (specifically for database retrieval) lack reported breakthroughs.

Agent use cases are challenging for multiple reasons: They can ignore the available tools to accomplish a task; they can call the wrong tools; and they can call the right tools with the wrong parameters. In addition to these general challenges, we believe that database retrieval use cases are challenging for the following three key reasons:

• Identifying the right index of data requires the large language model (LLM) to understand what it contains. But sometimes, the number of indices can already be so large that even representing those to select may cause context length problems.
• Generating efficient queries that balance retrieving relevant information with minimizing latency and resource usage can be challenging.
• Avoiding context bloat with tool responses requires the tool response to be optimized for contextual relevance and token efficiency. This isn’t always easy, especially when the agent generates the query from scratch. Once the context is no longer relevant to answer a user’s query, offloading the data for later reference is another challenge.

During the development of Agent Builder and integrating it into our own processes, we faced these challenges multiple times. In fact, the principles we share in the following sections are based on common patterns we saw during iteratively improving our built-in and custom tools and our internal workflows built on top of them.

Principles for building effective database retrieval tools

In this section, we translate our learnings into guiding principles for building effective database retrieval tools: deciding which tools to build, making sure the agent finds the right index to search and calls the right tool with appropriate parameters, optimizing the tool responses, handling errors, and safeguarding the data.

Building the right database retrieval tools (“low floor, high ceiling”)

When deciding on what database retrieval tools to build, we follow the principle of “low floor, high ceiling” for good agent experience:

• High ceiling: Tools that don’t limit the agent's potential to handle ambiguous user queries in the worst case. In the context of database retrieval, these are general-purpose tools that, for example, allow the agent to write full SQL/ES|QL queries from scratch. These come with the trade-off of reasoning overhead of the agent and result in higher latency, higher cost, and lower reliability.
• Low floor: Tools with high accessibility that the agent can use successfully on the first try with minimal reasoning overhead for repeating user queries. In the context of database retrieval, these are specialized tools that, for example, wrap specific queries. These have the benefit of lower latency, lower cost, and higher reliability than general-purpose tools. However, they require engineering effort, and realistically, it may not be possible for engineers to anticipate every possible user query.

For example, in our experience, a generic search tool is mandatory to allow the agent to handle unique and ambiguous user queries in the worst case. However, we found it necessary to reduce reasoning overhead and increase efficiency by creating specialized tools (for example, get_top_performing_products(category)).

Another lesson we learned is to consider the level of abstraction of a tool. During our preview phase, the agent had access to a large number of atomic general-purpose tools (for example, get_index_mappings, generate_esql, execute_query, and others). In practice, this had two downsides: When faced with a complex, open-ended question, the agent would confuse tools and their expected order, despite guiding instructions. Combining multiple tools in an agentic workflow also requires hand-off of information via the context window, which leads to filling up the context window with information that’s only temporarily important. To overcome this inefficiency, we wrapped the functionality of multiple atomic tools into one self-contained search tool.

Finding the right index

While the majority of tools that interact with a database will focus on querying the database, in some use cases, like for index selection, the tool will allow the agent to interact with the database’s metadata to decide which indices to search against based on a user’s query and intent.

Initially, our naive index selection relied on the index names and a sample of their schema definitions. This worked well in our internal testing, but when internal teams experimented with it, we realized real use cases often don’t have index names that are distinct and descriptive but are instead vague (for example, users, logs, flight_travels versus web-logs-2026.01, web-logs-2026.02).

To overcome this, we started exposing each index’s metadata and schema definitions in the tool. This significantly improved the selection by allowing engineers to add descriptions to translate technical names into natural language on two levels:

• Index-level descriptions: What data is stored in the index and how documents relate to one another.
• Field-level hints: Specific guidance on a field's format, expected values, or business meaning (for example, ”Use this field for exact ID matches only”).

In another iteration, we decided to add data sampling. For each index, we sample some of the data present in each field to let the agent have a clearer understanding of the type of data in the index. While it significantly improved the efficiency of index selection, it comes at the cost of increased tool response data.

Calling the right database retrieval tool

Guiding an agent to call the right tool is already challenging enough in general. This section discusses what helped us make sure the agent calls a tool to ground their response, as well as calling the right database retrieval tool.

Naming and namespacing: Standardizing identifiers for reliable selection

A tool’s name acts as a skimmable header that agents use to decide which one to investigate further. In practice, this means using descriptive and distinct tool names. Additionally, tool selection is more reliable when their names are consistent in formatting (for example, standardize on snake_case) and wording. Using action-oriented verbs helps the LLM map the user’s intent to the tool’s purpose, although the exact wording of the tool names is less critical in practice (for example, search versus find).

Namespacing tools to group related tools under common prefixes or suffixes is similarly helpful. In the case of databases, namespacing tools by index or domain helps the LLM understand tool relationships and prevents naming collisions (for example, finance.search_ticker or support.get_ticket_details).

Tool description: Instructing the agent on proper usage

The description is the most crucial component of any tool definition because it instructs the agent on when and how to use it, especially when tools have similar names (for example, search_logs and find_errors).

Consider this template for complex tools:

• Core purpose: A high-level summary of what the tool does.
• Trigger: When the tool should be used (and when it should not).
• Action: Which specific data the tool retrieves or modifies, and what type of questions it can answer.
• Limitations: What important limitations and constraints exist, such as specific query languages or formats.
• Relationships with other tools: Does one tool affect another tool, or are there any preconditions?
• Examples: Specific few-shot examples of user queries and how to use the tool for them, such as how to determine the optimal search strategy or when to use which operator.

A note on model sensitivity: While top-tier models like Claude 4.5 Sonnet are forgiving, smaller models often need clearer and more elaborate descriptions to select the right tool.

Adding reasoning parameters

Inspired by the paper on think-augmented function calling, we added a reasoning parameter. This approach improves the parameter accuracy by providing a scratchpad for the LLM to process its thoughts and facilitates a more transparent user experience.

This works well for complex tool calls or when a large number of tools are exposed to the agent. However, it can lead to regression in simple scenarios, and the benefits are further diminished for thinking-based LLMs. In our Agent Builder implementation, the reasoning parameter is often optional and stripped before execution and is only used for tool selection and parameter filling.

"properties": {
   "reasoning": {
      "type": "string",
      "description": "Brief explanation of why you're calling this tool"
   }
}

Support: Reinforcing instructions in the agent prompt

A common error we observed is that the LLM would sometimes ignore the available tools and instead use its innate knowledge to generate a (hallucinated) response. For example, when asked ”Can you tell me more information about Elasticsearch’s ES|QL language?”, it would assume it was fine to answer by itself instead of calling the tool, which was specifically designed to fetch documentation about Elastic products.

To mitigate this problem, we added repeated, explicit instructions in the system prompt of the agent itself to guide the agent to find the right balance between using its innate knowledge and grounding its answer in a tool response. Our testing indicates that this is especially effective when multiple tools with similar purposes are exposed to the agent.

Forcing tool usage

Beyond reinforcing the instructions in the agent prompt, we found it helpful to force tool usage when mandatory by explicitly binding tools using tool_choice: ‘any’.

Calling database retrieval tools with appropriate values and writing queries

Another challenge is to enable the agent to call a tool with appropriate parameter values. We’ve observed a consistent pattern where a strong definition, the number of parameters, and their complexity play an important role in reducing errors.

Parameter definition

A strong parameter definition significantly improves the parameter accuracy. General best practices for parameter definitions in agent tools are:

• Unambiguous name: Clearly identifies the purpose (for example, user_id versus user).
• Strong typing: Uses integer, string, or enums, among others, for finite sets of valid values.
• Detailed description: Explains what the parameter means and when and how to use it. Specifies default values for missing values, document formats (for example, for dates), hidden rules (for example, ”at least one of agent_id | user_id is required”), and includes small examples.

# Weak parameter description
"properties": {
   "index": {
      "description": "Name of the index",
   },
}

# Strong parameter description
"properties": 
{
   "index": {
      "type": "string","description": "The specific index, alias, or datastream to search. Defaults to 'main-alias' if unknown. ",
   },
}

Number of parameters

Agents struggle to call a tool with a large number of parameters with appropriate values, especially mandatory ones. As a general rule of thumb, we try to keep the mandatory parameters below five and the total parameters below 10.

Parameter complexity

Reducing the complexity of the input parameter when possible leads to fewer mistakes. For example, it requires reasoning overhead to let an LLM generate a search term than generating full SQL or ES|QL queries from scratch. Especially for repeating user queries, “pre-canning” search queries reduces latency, cost, and error rates (although modern LLMs are good at using well-known languages, such as SQL).

To follow the principles of “low floor, high ceiling,” we gravitated toward wrapping a specific query inside a tool and letting the agent only provide the search term. Below is an example of input parameters with varying complexity for the same user query, "Find the 5 most relevant 'resolved' support tickets based on a new problem description to find previous solutions."

# Complex parameter
search_support_tickets("FROM support_tickets | WHERE status = 'resolved' AND customer_email = ?email | MATCH(issue_title, issue_description, ?problem_description) | KEEP ticket_id, issue_title, resolution | LIMIT 5",
)

# Simple parameter
find_similar_customer_resolved_tickets(problem_description)

Model sensitivity

Models have a strong impact on parameter accuracy and query generation. Our internal benchmarking showed that switching from Claude 3.7 Sonnet to Claude 4.5 Sonnet reduced the syntax errors of the generated ES|QL queries from ~28% to ~4%.

Input validation

Although the above techniques increase parameter accuracy, they don’t eliminate the possibility of errors. Instead of trusting the LLM's input, we gravitated toward always validating and sanitizing it to ensure queries adhere to the expected schema.

Optimizing database retrieval tool responses

One common mistake is neglecting the size of the tool’s output. Because the tool’s output is what goes into the agent’s context window, not optimizing it for contextual relevance (quality) can distract the agent, and not optimizing it for token efficiency (quantity) can increase cost and risk exceeding the LLM’s context window limit. Working alongside internal teams, we’ve identified three dimensions for reviewing the return values:

The first dimension is length: The number of search results. A recurring pitfall we’ve observed among teams is the failure to limit search results, which can lead to overflowing the context window. While vector search queries inherently require a limiting parameter, other search methods often do not. We recommend including limit clauses (for example, 10 - 20) in all query types. This prevents returning low-signal results and ensures token efficiency.

The second dimension is width: The number of fields per data object. Instead of returning all properties, including cluttering ones (for example, timestamps and internal IDs, among others), curating a set of relevant fields can help improve both the user experience and the agent behavior.

Consider the following examples:

• Returning identifiers with a concise set of properties of a data object allows the agent to retrieve full information about a data object later when it needs it (“just-in-time context engineering”), rather than cluttering the context window.
• Returning metadata for citations (for example, page numbers in large PDF documents) can help build trust for the user.
• Returning the number of search results and status messages can help the agent reason the status of the search query.

The third dimension is depth: The size of a single field. Consider a case where the documents themselves are large (for example, in the 10s of MB scale). Those can’t just be passed back to the agent in full, as context length will instantly reach its limit. To mitigate this, we recommend truncating long text fields when an excerpt is sufficient. We found features such as Elasticsearch’s highlighting are helpful for this without the need for another LLM API call.

But even when working with smaller documents, letting the tool format the data into an easily digestible format for the LLM, such as sanitizing the content (for example, stripping HTML tags), formatting for readability (for example, tables to Markdown tables or links as “[Title](url)”), improved downstream performance.

While using only one of these techniques is often not sufficient for improving the contextual relevance, combining all of them might reduce the retrieval recall (for example, limiting the number of search results might risk not retrieving all of the relevant documents). In practice, this requires evaluating different combinations of these approaches to find the best balance.

Handling errors and enabling self-correction

We’ve observed that agents can get stuck in infinite loops or hallucinate responses when they encounter an error. Even if an agent follows its instructions perfectly, if a tool doesn’t provide any error message, only returns an error code, or at best provides a short, nondescriptive error message, the agent has no chance to self-correct from an error it doesn’t understand.

An informative error message enables the agent to understand why the error is happening and how to recover from it. For this, engineers need to think about the “not so happy” paths and the anticipated edge cases, such as the following examples:

If an error occurred because of a wrongly formulated search query, the agent should be able to reason over the failure and reformulate the query. In this example, returning the number of search results and the generated query can help the agent self-correct.

In general, engineers need to consider whether “zero results” is an expected behavior or an error for a given tool. In cases where an empty result likely indicates an error, both an error message and agent instructions can be helpful.

# Example error message from tool
"No product data found for product ID [XYZ]. 

Review the search query [insert used search query here].

Ask the customer to confirm the product name, and look up the product ID by name to confirm you have the correct ID."

---
# Example agent instruction
"If the product_search tool returns no results, do not state that the product does not exist. 

First, check that the `:` operator was used for multi-value fields.


Second, ask the user to provide and confirm the product ID or provide an alternative identifier like product name."

When encountering an API failure due to an expired API key, strictly limit retries (for example, a maximum of two or three) to prevent the agent from repeatedly trying a failing deterministic flow.

Safeguarding data

The primary engineering challenge for production-grade agent applications with different tools lies in identity propagation, specifically the distinct requirements of authentication (verifying who the user is) and authorization (verifying what they can access). While an initial layer (for example, Okta) can provide the base authentication, downstream systems (for example, ServiceNow, Elasticsearch, and others) maintain their own nonuniform authorization architectures with varying levels of granularity.

The most successful implementations we’ve seen enforce identity verification at every touchpoint within the tool's logic. This prevents the agent from accidentally accessing data that the end user isn't allowed to see. By verifying identity at every system level, we ensure that the agent respects privacy even when navigating complex, multisystem workflows. Be aware of the downside; this introduces intentional, security-mandated latency.

Beyond verifying the identity at every touchpoint, managing credentials securely is critical. Exposing sensitive API keys in tool definitions or hardcoding database credentials in YAML configuration files is a high-risk vulnerability. Instead, we recommend that engineers use secure credential management systems.

Evaluating database retrieval tools

The development of database retrieval tools for agentic systems is an iterative, evaluation-driven process. To evaluate the effectiveness of our database retrieval tools and uncover issues, our internal teams maintain evaluation datasets with realistic user queries and the expected tool calls (for example, ”Am I eligible for a laptop refresh?” expects the check_eligibility tool). We’ve used the following metrics for evaluation of our database retrieval tools and for benchmarking different LLMs for model selection:

• Tool selection accuracy: How often the correct tool was selected for a specific query type (for example, retrieval, analytical, hybrid, adversarial).
• First-pass success rate: A critical metric for us was distinguishing between eventual success and immediate success. Does the agent pick the right tool on the first try, or does it require a self-correction loop? (High self-correction indicates poor tool descriptions.)
• Average tool calls per answer: We track the efficiency of the agent. If the average number of tool calls to answer a simple question jumps from 1.5 to 4, it usually indicates that the agent is getting lost or that the tools are too granular.
• Tool-specific recall/precision: For dedicated database retrieval tools, we measure standard retrieval metrics to ensure that the documents returned are relevant to the arguments passed by the LLM.
• Failure rate: We strictly monitor the rate of malformed tool calls (for example, missing parameters) to identify which models need more "hand-holding" in the prompt instructions.

Once live, our teams at Elastic continue to monitor agentic health and log telemetry data (for example, every success and failure and the time taken for the agent to complete the task) in Kibana. This allows our ITOps teams to build dashboards to answer meta-questions like, "What is the failure rate this week?," "How many laptop requests came from California?," "How many requests were fulfilled?," without building a custom analytics engine.

Summary

During the iterative, evaluation-driven development process of Elastic Agent Builder, we identified consistent patterns in what makes database retrieval tools effective for context engineering. When implementing tools to search, retrieve, and manipulate data, we try to follow the following core principles:

1. Building the right database retrieval tools by following the “low floor, high ceiling” principle.
2. Helping the agent call the right database retrieval tool with appropriate parameter values through careful and reinforced prompting and interface design.
3. Avoid context flooding by optimizing the tool response for context relevancy (quality) and token efficiency (quantity).

However, there remain open challenges, and we’re actively working on improving these techniques:

• Context bloat is a primary hurdle for agent performance because retrieved data remains in the context window as the conversation progresses. A future direction is to dynamically off-load large chunks of data, such as tool responses or attachments, in a file store and allow the agent to retrieve them on demand.
• Efficient discovery of large volumes of tools and data attachments will be central for building production-grade agents. We plan to introduce agent skills with the functionality of progressive disclosure (loading information only as needed) and implementing a semantic metadata layer.

Acknowledgements

Written by Leonie Monigatti with valuable contributions from colleagues across Search Solutions Engineering (Sean Story, Pierre Gayvallet, Abhimanyu Anand) and Enterprise Applications (Sri Kolagani).

EIS already provides access to state-of-the-art large language models (LLMs) that power out-of-the-box AI capabilities across Elastic Agent Builder and Elastic AI Assistants, including automatic ingest, threat detection, problem investigation, and root cause analysis. We’re now extending this foundation with a broader catalog of managed models, giving developers more control over how agents reason, retrieve, and act.

In practice, this reflects a broader shift in how enterprises build AI systems. The idea of a single, all-purpose AI model no longer holds up. Real-world agent workflows require multiple models with different strengths, costs, and performance characteristics. With EIS, teams can either choose and switch models directly in Agent Builder, with zero setup, cost, or hosting overhead, or they can mix and match models in an agent workflow so each step uses the model best suited to the task.

Developers can use models from OpenAI, Anthropic, and Google directly in Elasticsearch, selecting different models for different agent steps while Elastic fully manages inference, scaling, and GPU execution for production agents.

An expanded catalog of managed models on EIS

The expanded EIS catalog now includes models optimized for different classes of tasks, from lightweight generation to large-context reasoning and embeddings for retrieval.

For generation, the catalog includes:

• Anthropic Claude Opus 4.5 and 4.6.
• Gemini 2.5 Flash.
• Gemini 2.5 Pro.
• OpenAI GPT-4.1 and GPT-4.1 Mini.
• OpenAI GPT-5.2.
• OpenAI GPT-OSS-120B.

For retrieval, EIS includes native Jina AI models, jina-embeddings-v3 and jina-embeddings-v5, which provide fast, high-quality embeddings for multilingual retrieval. The service also includes embedding models from Microsoft, OpenAI, Google, and Alibaba.

Choosing the right models for agent tasks

With EIS, model choice becomes a design decision inside the agent, rather than an operational concern. Agents can select models based on the role they play, without changing how inference is deployed or scaled.

To see how this plays out in practice, consider a few common agent scenarios.

Simple informational query

Simple interactions, such as answering “What is our holiday policy?,” do not require an expensive frontier model and can be handled by a fast, low-cost option.

• Task: “What is our holiday policy?”
• Pattern: Retrieve and summarize.
• Model choice: Fast, low-cost generation model.

This can also be configured through the API by selecting the model you want to use:

POST kbn://api/agent_builder/converse
{
 "input": "What is our holiday policy?",
 "agent_id": "internal-kb-bot",
 "connector_id": "Anthropic-Claude-Sonnet-4-5"
}

This step relies primarily on retrieval quality. A lightweight model is sufficient to summarize a small set of documents quickly.

Moderate capability

More complex tasks may benefit from a more capable generation model, without necessarily requiring the most expensive reasoning model available.

• Task: “Compare our holiday policy with new labor laws in France and draft an email.”
• Pattern: Retrieve relevant documents, compare policy details across sources, and generate output such as a draft email.
• Model choice: More capable generation model.

Here’s the API example:

POST kbn://api/agent_builder/converse
{
 "input": "Compare our holiday policy with new labor laws in France and draft an email.",
 "agent_id": "internal-kb-bot",
 "connector_id": "Google-Gemini-2-5-Pro"
}

This task requires synthesis across multiple sources and structured output but doesn’t need the heaviest frontier reasoning model.

Investigation or audit task (high capability)

• Task: Review a large document set to identify compliance risks.
• Pattern: Multistep reasoning over large context, where the model evaluates information across many documents and synthesizes findings before producing a final judgment.
• Model choice: Frontier or large-context model.

Try it out using the API:

POST kbn://api/agent_builder/converse
{
 "input": "What are the compliance risks associated with Example AI products?",
 "connector_id": "Anthropic-Claude-Opus-4-6"
}

Because the task requires deeper reasoning and consistent evaluation across many inputs, output quality matters more. A high-capability model is therefore appropriate for this step.

EIS also enables more advanced orchestration patterns. Enterprises increasingly recognize that using a frontier model for every agent step is inefficient.

With Agent Builder and Elastic Workflows, teams can design agents where each subtask is executed by the most efficient model for the job, based on cost, complexity, and accuracy requirements.

Models-as-judge pattern (quality control)

• Task: Validate an agent’s output using a second model
• Pattern: Generate and evaluate.

In this Elastic Workflow example, the agent uses one model to generate a response and a second model to evaluate its quality, adding a validation layer for the result. Elastic Workflows, the automation engine built into Elasticsearch, let developers combine reliable scripted automation with AI-driven steps for tasks that require reasoning.

The multimodel approach enables new reliability patterns by separating generation from evaluation, allowing one model to produce a response and another to validate it. Today, teams can implement this by pairing a general-purpose generation model with a lighter-weight evaluation model.

Over time, this pattern naturally lends itself to specialized judging and safeguard models designed specifically for validation, policy checks, and quality control. As these models become available, EIS makes it straightforward to introduce them into agent workflows without changing how inference is deployed or managed.

What’s next

EIS is actively evolving, with more models on the way. You can track what’s coming next and what we’re currently building on the Elastic public roadmap.

Get started

Elastic Inference Service makes it easy to start with default models and evolve toward sophisticated, multimodel agent workflows over time, all within Elasticsearch. Whether you’re building global retrieval augmented generation (RAG) systems, search, or agentic workflows that need reliable context, Elastic now gives you high-performance models out of the box, along with the operational simplicity to move from prototype to production with confidence.

All Elastic Cloud trials have access to Elastic Inference Service. Try it now on Elastic Cloud Serverless or Elastic Cloud Hosted, or use EIS via Cloud Connect with your self-managed cluster.

Do we still need a search engine at all?

If agents can call tools on demand and models can reason over massive context windows, why not just fetch data live from every system and let the LLM figure it out?

It’s a reasonable question. It’s also the wrong conclusion.

The reality is that MCP and agent tooling don’t eliminate the need for search. They make the quality of search more critical than ever. In this blog, we’ll explore why MCP, federated search, and large context windows don’t replace search engines and why indexes remain the foundational layer for scalable, accurate, enterprise-grade AI.

What MCP actually is (and what it is not)

MCP is a coordination protocol. It standardizes how an agent requests information or actions from external systems.

What MCP doesn’t do:

• Rank results across systems.
• Understand relevance across heterogeneous data.
• Normalize schemas or metadata.
• Data transformations or enrichments at scale.
• Apply consistent security and permissions.
• Optimize for latency, cost, or scale.

In other words, MCP tells agents how to ask for data, not which data matters most.

Modern retrieval requires query intelligence, not just data access

In modern enterprise search architectures, retrieval quality is determined long before a query reaches an index. Raw queries — especially those generated by agents — may be incomplete, overly literal, schema-driven rather than intent-driven, and at times syntactically invalid.

This is why mature search platforms introduce a query intelligence layer that performs query rewriting, entity normalization, synonym expansion, and intent disambiguation before retrieval even begins.

For example, an agent-generated request such as: “Show severity 2 authentication failures from last sprint” may be rewritten to include authentication synonyms (login, SSO, OAuth), normalized severity mappings, and sprint-to-date-range translation. The result is not just more matches — it is more relevant matches.

In enterprise AI, retrieval is not a single step. It is a controlled pipeline.

This distinction is crucial because once MCP-based agents start pulling information live from multiple tools, they recreate a familiar pattern under a new name: federated search.

MCP-based retrieval is federated search in disguise

Federated search isn’t new. Enterprises have tried it for decades.

The model is simple:

• Send the user’s query to multiple systems in parallel (SharePoint, GitHub, Jira, customer relationship management [CRM]).
• Collect the responses.
• Merge and present the results.

MCP-driven tool calls follow the same pattern, except that the caller is now an agent instead of a user interface.

And the same problems resurface.

Why federated search breaks down at enterprise scale

• Latency becomes unpredictable: A federated query is only as fast as its slowest system. Enterprise systems can have wildly different response times and rate limits, so federated queries tend to be slow and jittery. Agents must wait for multiple round trips before reasoning can even begin. The result is a laggy experience and unpredictable wait times.
• Relevance is fragmented: Because each system ranks results on its own, there’s no unified relevance model. Federated search cannot apply a single ranking or semantic understanding across all content, so results often seem disjointed or incomplete. Agents may retrieve correct information but not the most useful information.
• Context is shallow and incomplete: Federated systems typically expose only what’s directly accessible through an API call.They rarely surface:
  • Usage signals, like clicks, dwell time, recency of access, popularity, or authority.
  • Relationships between documents across different systems to correlate the insights.
  • Organizational knowledge beyond a single silo.
    
    This strips agents of the broader context required for high-quality reasoning.
    
• Limited filtering and features: In a federated setup, you can only filter on fields that every system supports (the “lowest common denominator”). If one system doesn’t support a particular filter or facet, you lose that functionality entirely. This severely limits rich search features, like date ranges, categories, or tags.

The power of an indexed search

Search engines achieve millisecond-level retrieval at massive scale by using specialized data structures, including inverted indexes for lexical search and k‑dimensional trees (k-d trees) for vector-based retrieval. The approach is to crawl or ingest every source into search engines, creating a central place of company knowledge. This brings big advantages:

• Speed by design: Searching an index is lightning fast. Queries hit inverted indexes and specialized data structures, avoiding the need to poll each backend system.
• Relevance that compounds over time: Search engines that support semantic search are capable of comprehending the intent, and machine learning models can rerank results for enterprise contexts. In one Elastic experiment, Elastic users see more accurate results when combining vector search with a question-answering (QA) model to extract answers. It gives better precision than keyword matching.
• Advanced features: Elastic’s Graph retrieval augmented generation (RAG) solution shows how structuring an index as a knowledge graph can power more contextual retrieval. In other words, indexes aren’t just backward-looking dumps of text; they can also encode relationships and ontologies that let AI connect the dots across documents.
• Permission-aware search: Enterprise AI cannot compromise on security. Indexed search allows:
  • Document-level security.
  • Role-based access control.
  • Permission-aware retrieval for RAG and agents.

Agents see only what users are allowed to see, without leaking data into model prompts or training. Elasticsearch is suitable for the indexed search layer in the diagram below, as it provides the essential components for context engineering.

Retrieval consistency through search templates and governed execution

At scale, retrieval must be predictable, secure, and repeatable. This is where search templates become critical.

Search templates act as retrieval contracts between applications, agents, and the search platform. Instead of dynamically constructing queries at runtime, agents invoke pre-defined retrieval patterns that enforce:

• Consistent relevance logic
• Mandatory security filters
• Cost and latency guardrails
• Business-specific ranking rules
• Explicit index and field scope boundaries

In MCP-driven architectures, this becomes even more important. Agents should not dynamically invent retrieval strategies. Instead, MCP tool calls can map directly to approved search templates, ensuring that every retrieval request adheres to enterprise relevance and governance standards.

This approach shifts retrieval from ad-hoc query execution to controlled retrieval orchestration.

Retrieval is now a multi-layer engineering discipline

Modern enterprise retrieval is no longer a simple query-to-index operation. It typically includes multiple coordinated layers:

• Query understanding — rewriting, expansion, entity resolution
• Retrieval strategy selection — hybrid search, vector search, graph retrieval, or synthetic query techniques such as Hypothetical Document Embeddings (HyDE), where the system generates a representative answer or expanded context first and retrieves documents using that richer semantic signal.
• Execution governance — templates, security enforcement, and performance guardrails
• Ranking and re-ranking — blending lexical precision, semantic similarity, and interaction-derived relevance signals such as click-through patterns, dwell time, and document usage frequency.

When these layers are implemented upstream, agents receive clean, high-confidence context rather than raw, fragmented data.

This is what makes large-scale agent systems reliable in production environments.

Advanced retrieval techniques improve context quality before reasoning begins

Modern retrieval systems increasingly use AI-assisted techniques to improve recall and semantic coverage before ranking is applied.

One example is Hypothetical Document Embeddings (HyDE). Instead of embedding only the original query, the system first generates a hypothetical answer or expanded context, embeds that representation, and retrieves documents based on that richer semantic signal.

This is particularly useful in enterprise environments where:

• Users or agents may not know the exact terminology
• Knowledge is distributed across silos
• Important context is implied rather than explicitly stated

Techniques like HyDE improve the probability that relevant documents are retrieved even when the original query is underspecified.

This reinforces a key principle of enterprise AI: better context retrieval produces better reasoning outcomes.

Agents aren’t data engineers; they’re reasoning systems

They shouldn’t be responsible for stitching together raw data, reconciling schemas, or compensating for poor retrieval.

This is where a search platform such as Elasticsearch becomes foundational.

By ingesting data once and normalizing it upstream (through pipelines, mappings, enrichment processors, and prebuilt indexes), Elasticsearch resolves schema mismatches, joins signals across sources, and materializes retrieval-ready views of the data. At query time, the agent receives clean, ranked, semantically enriched results rather than fragmented raw records.

For example, instead of an agent pulling independently from CRM, ticketing, and documentation systems and attempting to reconcile customer IDs, timestamps, and formats in real time, Elasticsearch can pre-index these sources into a unified customer interaction index with hybrid (keyword + vector) search and relevance ranking. The agent then queries a single, coherent interface and immediately reasons over the most relevant context.

This separation of concerns, that is, Elasticsearch handling data integration and retrieval, and agents focusing on reasoning, planning, and decision-making, is what makes agent systems scalable, reliable, and production ready.

Elastic’s role in the AI stack

Elastic sits at the intersection of search and AI by design.

• Connectors and crawlers ingest data continuously from enterprise systems.
• Semantic and vector search enable intent-based retrieval.
• Hybrid search blends lexical precision with semantic understanding.
• RAG workflows ground LLMs in authoritative, permission-aware data.

Elastic does not compete with agents or MCP. It makes them effective.

Bigger models don’t eliminate retrieval

Some have wondered whether huge new LLMs can bypass traditional search, perhaps by letting the model read everything in one go. Large context windows feel powerful, but they introduce:

• Higher latency.
• Higher cost.
• Lower precision due to noise.
• A higher propensity for confusion, context clash, and context poisoning.

RAG wins because it filters first and then reasons.In another Elastic Search Labs experiment, RAG achieved answers in about 1 second, versus 45 seconds for the raw-LM approach, at 1/1250th the cost, and with far higher accuracy. In other words, giving an LLM a million tokens of documents is slower, more expensive, and actually less precise than filtering through an index first.

Conclusion: MCP changes the interface, not the fundamentals

MCP is a meaningful step forward in how agents interact with tools. But it doesn’t replace the need for fast, relevant, governed retrieval.

In enterprise AI:

• Context quality determines answer quality.
• Indexes create that context.
• Search is the foundation, not the legacy.

Indexes aren’t obsolete in the era of MCP. They’re the reason that MCP-based agents can work at all.

The root of this problem was our somewhat naive choice to use prompt-based JSON generation in which the LLM generates JSON responses in text format. If we asked the LLM to judge more than a couple of matches at a time, the generated JSON was often ill-formed. To mitigate this, we were forced to reduce the processing batch size, which simply won't scale in a production system.

So the prompt-based JSON generation helped validate our approach to entity resolution, but we need a more systematic and reliable method. OpenAI function calling provides a better path by guaranteeing structure and type safety while reducing errors and costs. We chose OpenAI's functions for the educational prototype, but other LLM providers typically provide similar functionality (for example, Claude tools).

Note: While we discuss production challenges here, this is still an educational prototype demonstrating optimization techniques. Real production systems would need additional considerations, like monitoring, alerting, fallback strategies, and comprehensive error handling.

Key concepts: Function calling, schema design, and cost benefits

What is function calling? Function calling is OpenAI's structured output API. With it, we can define schemas for LLM responses, so we always know exactly what we're going to get. By enforcing the JSON format rather than trying to define it in the LLM prompt, we should be able to eliminate parsing errors.

Why is it better than prompt-based JSON? LLMs generate nondeterministic output. One hopes that they'll at least generate content that contains the correct response, but the presentation of that response is unpredictable. With a chatbot, this is often not a problem, but our prototype is trying to programmatically process the output. Computer programs demand consistency, so when the LLM generates what we expect, everything is fine, but as soon as it goes off script, so to speak, the code errors out. We could try to account for the different possibilities, but it would be very difficult to catch everything. We could try to enforce more consistent behavior by adding something like "Always return parsable JSON". We tried this exact technique in the prototype's prompt, but we've seen that prompt-based JSON still goes off the rails pretty quickly, particularly if we try to process a batch of matches.

Function calling makes the LLM generation controllable and predictable, exactly what we need for entity resolution. To aid in the definition of the functions, we’ll also follow minimal schema design principles.

What are minimal schema design principles? Minimal schema design means defining only the fields you need, using simple types, and avoiding nested structures when possible. This reduces token usage (smaller schemas mean fewer tokens), improves reliability (simpler schemas are easier for the LLM to follow), and lowers costs (fewer tokens mean lower API costs).

What are the cost and reliability benefits? Since fewer errors means match processing is much more likely to succeed, even with large batch sizes, we don't have to retry judging matches. The elimination of retries reduces costs by reducing token usage, but using minimal schemas also keeps our token count down. This all leads to a less expensive and more reliable approach that’s much more suitable to use in production.

We need to check one more thing, though. While matches may be getting processed without error, are the errorless results actually correct? How does this new approach compare to the promising results we saw with the prompt-based approach?

Real-world results: Side-by-side comparison

As we did in the previous blog, we ran the function calling approach against the tier 4 dataset, which consists of 206 expected matches across 69 articles. The results demonstrate a dramatic improvement:

Error elimination: The key differentiator

The most striking difference is the complete elimination of JSON parsing errors. This resulted in a modest precision improvement and a far more dramatic recall improvement. The precision metric captures how often the matches the system accepts were expected in the golden document. So the prototype was decent at judging matches correctly in the prompt-based approach, but function calling does that even better.

Conversely, recall tells us how many of the expected matches were found. When a batch of matches comes back with malformed JSON, the system loses all of those matches. It's likely that Elasticsearch sends many of these matches for judgment, but we lose those matches if judgment fails. The significant recall improvement shows that this hypothesis is correct. Elasticsearch identifies the potential matches and function calling verifies which of those matches are correct.

Note: It’s expected that Elasticsearch will find some incorrect matches because we look at the top two or three results from hybrid search. Most of the time, hybrid search returns the correct match as the top result, but having the LLM judge the top few hits ensures that we see how the LLM handles incorrect matches. If we move from the educational prototype to a production system, we’ll likely tune the Elasticsearch queries more carefully so that we only send promising matches to the LLM, further optimizing our LLM costs.

What's next: The ultimate challenge

Now that we've optimized our LLM integration with function calling, we have a complete entity resolution pipeline with improved reliability and cost efficiency. However, can it handle the ultimate challenge? In the next post, we'll explore how the system handles diverse entity resolution scenarios across 50 different challenge types, including cultural naming conventions, business relationships, titles, and multilingual variations.

Try it yourself

Want to see function calling optimization in action? Check out the Function Calling Optimization notebook for a complete walkthrough with real implementations, detailed explanations, and hands-on examples. The notebook shows you exactly how to use function calling for structured output, compare it with prompt-based JSON, and analyze cost and reliability benefits.

Remember: This is an educational prototype designed to teach optimization concepts. When building production systems, consider additional factors, like multi-provider support, advanced caching strategies, monitoring and alerting, comprehensive error handling, and compliance requirements that aren't covered in this learning-focused prototype.

What are subagents?

Subagents are specialized assistants that can be called to execute a specific task, using their own context window. They complete a task and give the results to the main agent, preventing it from saving information that isn’t relevant for the rest of the conversation in the context window.

Their four core principles are:

• Context preservation: Each subagent uses its own context window.
• Specialized expertise: Each subagent is designed for a specific task.
• Reusability: You can reuse a subagent in different sessions and projects.
• Flexible access: You can limit the subagent access to specific tools.

Each subagent can have access to Claude Code tools to work with the terminal, such as glob, read, write, grep, or bash, or to access the internet, like search, fetch, or call external tools with Model Context Protocol (MCP) servers.

A subagent uses the following schema:

---
name: your-sub-agent-name
description: Description of when this subagent should be invoked
tools: tool1, tool2, tool3  # Optional - inherits all tools if omitted
model: sonnet  # Optional - specify model alias or 'inherit'
permissionMode: default  # Optional - permission mode for the subagent
skills: skill1, skill2  # Optional - skills to auto-load
---

Your subagent's system prompt goes here. This can be multiple paragraphs
and should clearly define the subagent's role, capabilities, and approach
to solve problems.

Include specific instructions, best practices, and any constraints
the subagent should follow.

You can call subagents implicitly by talking about the task they run, and Claude will call them automatically. For example, you can say, "I want to plan my new functionality."

You can also call them explicitly by directly asking Claude Code to use a subagent and telling it, "Use the planning subagent to plan my new functionality."

Another important feature is that subagents are stateful, so once you give one a task, it will generate an ID. This way, when you use it again, you can start from scratch or provide the ID to give it context from its previous tasks.

You can read the full documentation here.

When are subagents used?

Subagents are useful when you need to delegate tasks that require specialized context but you don't want to clutter the main chat window. Considering our example of coding, the most common subtasks include:

Claude Code includes three built-in agents that showcase these use cases:

• Explore: Quick agents for read-only search in the codebase. It's great for answering questions like, "Where are the client's errors handled?"
• Plan: Research agent that activates in plan mode to analyze the codebase before proposing changes.
• General-purpose: The most capable agent for complex tasks that require multiple steps and can include modifications.

Context management: Ensuring subagents have the right information

One of the most important decisions when designing subagents is how to handle context. There are three key considerations:

1. Which context the subagent should get

The prompt you give to the subagent must contain all of the necessary information to complete the task since the subagent doesn’t have access to the main chat. You need to be specific:

• Do NOT say, "Review the code."
• SAY, "Review the changes to src/auth/index.ts, focusing on JWT token validation."

Providing the exact file name makes a difference between using the read tool against the file directly and making a wide search using grep and thus wasting time and tokens.

Also consider what not to include. Irrelevant context can distract the subagent or bias results. It’s tempting to ask for multiple things in one pass, but focused tasks yield better results:

• Do NOT say, “Review src/auth/index.ts. Here is also the database schema and our API docs for reference, fix bugs and suggest improvements about the architecture decisions.”
• SAY, “Fix the token refresh bug in src/auth/index.ts that's throwing AUTH_TOKEN_EXPIRED unexpectedly.”

2. What tools to provide

Limit the tools to what’s strictly necessary. This improves security, keeps the subagent focused, and reduces unnecessary tool calls and execution costs.

# For just an analysis agent
tools: Read, Grep, Glob

# For an agent that needs to modify the code
tools: Read, Edit, Write, Grep, Glob

If you don't specify a tools field, the subagent inherits all tools from the main agent, including MCP tools.

You can learn all Claude Code tools here.

3. How to keep context between calls

Subagents can be resumed using their agentId:

# First call
> Use the code-analyzer agent to review the authentication module
[Agent completes the analysis and returns agentId: "abc123"]

# Continue with previous context
> Resume agent abc123 and now analyze the authorization module
[Agent continues with the context from the previous chat]

You can ask Claude for the agent ID or find it in ~/.claude/projects/{project}/{sessionId}/subagents/

This is especially useful for long research tasks or multistep workflows.

Another way to keep context consistent is to ask the agent to write a Markdown checklist with what it's doing and its current progress. Then you can execute /clear without losing the initial instruction. In that request, you can define the task granularity or details to retain that make sense for your use case.

# Task: Review authentication module

## Progress
- [x] Analyzed src/auth/index.ts
- [x] Found JWT validation issue
- [ ] Review authorization module
- [ ] Check rate limiting

## Findings
- Token refresh has race condition in line 42

After you clear the conversation, the next agent can pick it up from here. This is very useful when you want an agent to run a script over a list and watch the output record by record.

Orchestration patterns

It’s important to see subagents as a context optimization mechanism. The way in which you coordinate them determines the efficiency of the whole system. There are different orchestration patterns.

Sequential (chaining)

Here, a subagent completes a task, and its results feed the next one in a sequence of tasks, similar to traditional Linux piping.

Call example:

> First use the planning agent to design the feature,
> then use the coding agent to implement it,
> finally use the reviewer agent to check the code

Parallel

In this pattern, multiple subagents run independent tasks simultaneously. The main Claude Code agent invokes them since subagents cannot spawn other subagents.

This approach reduces the execution time for tasks like code review since it allows you to work with the same code from different angles without impacting the running time.

Hub-and-spoke (delegation)

In this approach, the main agent acts as an orchestrator, delegates tasks to specialized agents, and then consolidates the results.

This is the pattern we’ll implement in our example: The main Claude Code agent will delegate the gathering of business information to a retrieval agent built with Elastic Agent Builder, while the explore agent will look into local files and the planning agent builds a plan.

Why use an agent instead of a single query?

Before building our retrieval subagent, it's worth understanding when an agent adds value versus when a simple Elasticsearch Query Language (ES|QL) query suffices.

If you need a single aggregation, like "What's our most visited page?" just run the query directly. The agent adds value when your question requires:

• Multiple queries that build on each other: The answer from query 1 informs query 2.
• Cross-index reasoning: Correlating data from different sources.
• Ambiguity resolution: The agent interprets and follows leads.
• Synthesis: Combining quantitative data with qualitative knowledge.

Our example will demonstrate all of these capabilities.

Agent Builder as subagent

Generating code using AI is very quick, but the problem is having a good planning phase to set the boundaries for our coding agent. To help with that, Claude created a subagent that specializes in planning to perform deep analysis and create a to-do list for the main agent to execute.

With this flow, you can plan based on what Claude Code can see both in local files and on the internet. However, there's still knowledge available in Elasticsearch that you cannot access via standard tools.

To access our internal knowledge during the planning phase, we'll create a Claude Code subagent by making a retrieval agent using Agent Builder.

You can configure the agent using the UI or an API. In this example, we'll use the latter.

Prerequisites

• Claude Code 2.0.76+
• Elasticsearch 9.2
• Elasticsearch API key

The scenario: Technical debt sprint planning

You're a tech lead. You have two weeks and two developers. Your TECH_DEBT.md lists 12 items. You can tackle maybe three or four. Which ones should you prioritize?

The complexity is that you need to optimize across multiple dimensions simultaneously:

• User impact: How many users hit this issue?
• Business impact: Does it affect paying customers? Enterprise tier?
• Severity: Errors? Performance? Just ugly code?
• Effort: Quick win or rabbit hole?
• Dependencies: Does fixing A unlock fixing B?
• Strategic alignment: Does it align with Q1 priorities?

A single query like, "What's the most important tech debt item?" fails because this requires:

1. Reading TECH_DEBT.md to understand what the 12 items even are.
2. For EACH item, querying error_logsto get error frequency.
3. Cross-referencing with customer_data to see tier breakdown.
4. Checking support_ticketsto see complaint volume.
5. Reading engineering_standards in the knowledge base to see whether any items violate core principles.
6. Reading Q1_roadmap to check strategic alignment.
7. Synthesizing all of this into a prioritized recommendation.

This is where a retrieval agent can be helpful in orchestrating multiple queries across different indices and synthesizing the results.

Steps

Preparing the test dataset

We'll create four indices: a knowledge base with internal documentation, error logs, support tickets, and customer data.

You can create the indices, index the data, and create the agent using one of the following:

• Kibana Dev Tools: Using the Elasticsearch requests provided below.
• Jupyter Notebook: Using the complete notebook written for this article.

Create the indices

Open Kibana Dev Tools, and run the following requests to create each index with its mapping and bulk data. Here's an example of the knowledge index structure and data to be indexed:

PUT customer_data
{
  "mappings": {
    "properties": {
      "user_id": { "type": "keyword" },
      "customer_tier": { "type": "keyword" },
      "company_name": { "type": "text" },
      "mrr": { "type": "float" },
      "joined_at": { "type": "date" }
    }
  }
}

POST customer_data/_bulk
{"index":{}}
{"user_id":"enterprise_user_01","customer_tier":"enterprise","company_name":"Acme Corp","mrr":2500.00,"joined_at":"2023-01-15"}
{"index":{}}
{"user_id":"enterprise_user_02","customer_tier":"enterprise","company_name":"GlobalTech Inc","mrr":4200.00,"joined_at":"2022-08-20"}
{"index":{}}
{"user_id":"enterprise_user_05","customer_tier":"enterprise","company_name":"DataFlow Systems","mrr":3100.00,"joined_at":"2023-06-01"}
{"index":{}}
{"user_id":"user_001","customer_tier":"free","company_name":"","mrr":0,"joined_at":"2024-03-15"}
{"index":{}}
{"user_id":"user_002","customer_tier":"free","company_name":"","mrr":0,"joined_at":"2024-05-20"}
{"index":{}}
{"user_id":"user_045","customer_tier":"pro","company_name":"SmallBiz LLC","mrr":49.00,"joined_at":"2024-01-10"}
{"index":{}}
{"user_id":"user_089","customer_tier":"pro","company_name":"StartupXYZ","mrr":49.00,"joined_at":"2024-02-28"}

Full requests for all indices:

• Knowledge index: knowledge.txt
• Error logs index: error_logs.txt
• Support tickets index: support_tickets.txt
• Customer data index: customer_data.txt

The raw JSON files with the dataset are also available:

• knowledge.json
• error_logs.json
• support_tickets.json
• customer_data.json

Local project files

Create the following Markdown (MD) files in your project. These files look like this:

# Tech Debt Items

## AUTH-001: Token refresh race condition
- **Module**: src/auth/refresh.ts
- **Symptom**: Users randomly logged out
- **Estimate**: 3 days

## EXPORT-002: CSV export timeout on large datasets
- **Module**: src/export/csv.ts
- **Symptom**: Timeout after 30s for >10k rows
- **Estimate**: 2 days

...

Full files:

• TECH_DEBT.md: Tech debt items list.
• REQUIREMENTS.md: FlowDesk Q1 2025 requirements.

This ties directly to the tech debt items and gives the agent clear priorities to work with when cross-referencing with the Elasticsearch data.

Create an agent with Agent Builder

We'll now create an agent capable of running analytics queries with ES|QL to provide us with app usage information while also capable of searching to provide us info from Knowledge Base (KB) in unstructured text format.

We're using the built-in tools since they cover search and analytics on any index. Agent Builder also supports custom tools for more specialized operations, like scoping an index or adding ES|QL dynamic parameters, but that's beyond our scope here.

You can create the agent using the curl request in create_agent.txt.

curl -X POST "https://${KIBANA_URL}/api/agent_builder/agents" \
  -H "Authorization: ApiKey ${API_KEY}" \
  -H "kbn-xsrf: true" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "tech-debt-advisor",
    "name": "Tech Debt Prioritization Agent",
    "description": "I help prioritize technical debt by analyzing error logs, support tickets, customer impact, and aligning with engineering standards and roadmap priorities.",
    "avatar_color": "#BFDBFF",
    "avatar_symbol": "TD",
    "configuration": {
      "instructions": "This agent helps prioritize technical debt items. Use the following indices:\n\n- knowledge: Engineering standards, policies, and roadmap priorities\n- error_logs: Production error frequency by module\n- support_tickets: Customer complaints and their urgency\n- customer_data: Customer tier information (enterprise, pro, free)\n\nWhen analyzing tech debt:\n1. Check error frequency in error_logs\n2. Cross-reference affected users with customer_data to understand tier impact\n3. Count support tickets and note urgency markers\n4. Check knowledge base for relevant policies and Q1 priorities\n5. Synthesize findings into prioritized recommendations",
      "tools": [
        {
          "tool_ids": [
            "platform.core.search",
            "platform.core.list_indices",
            "platform.core.get_index_mapping",
            "platform.core.get_document_by_id",
            "platform.core.execute_esql",
            "platform.core.generate_esql"
          ]
        }
      ]
    }
  }'

You’ll get this response if everything went OK:

{
  "id": "tech-debt-advisor",
  "type": "chat",
  "name": "Tech Debt Prioritization Agent",
  "description": "I help prioritize technical debt by analyzing error logs, support tickets, customer impact, and aligning with engineering standards and roadmap priorities.",
  ...
}

The agent will be available in Kibana, so you can now chat with it if you want:

Configure the agent as Claude Code tool

The agent we just created will expose an MCP server. Let's add the MCP server to Claude Code using the already-generated API key:

claude mcp add --transport http agentbuilder https://${KIBANA_URL}/api/agent_builder/mcp --header "Authorization: ApiKey ${API_KEY}"

We can check the connection status using claude mcp get agentbuilder.

Create a subagent that uses the tool

Now that we have the Agent Builder available as a set of MCP tools, we can create a subagent in Claude Code that will use all or some of those tools, in combination with Claude Code ones.

Claude Code recommends using its agent creator tool for this step:

1. Type /agents in Claude Code.

2. Choose Create new agent.

3. Select Project scope so that it's only available for this project. (This is the recommended setting to avoid agent overflow.)

4. Select Generate with Claude (recommended).

5. Type in the description: "Agent that analyzes technical debt by querying Elasticsearch for error logs, support tickets, customer data, and engineering knowledge base. Use this agent when you need to prioritize tech debt items based on business impact."

6. In “Select tools,” choose Advanced options and select the tools we defined on the agent creation.

Individual Tools:
☒ platform.core.search (agentbuilder)
☒ platform.core.list_indices (agentbuilder)
☒ platform.core.get_index_mapping (agentbuilder)
☒ platform.core.get_document_by_id (agentbuilder)
☒ platform.core.execute_esql (agentbuilder)
☒ platform.core.generate_esq (agentbuilder)

7. Select [ Continue ].

Now choose the model. For planning tasks, the recommendation is to use Opus due to its significant reasoning capacity. So let's select that and continue.

Finally, choose the background color for our subagent text and confirm.

Claude automatically names our subagent based on the description (for example, tech-debt-analyzer).

Testing the agent

Once the agent has been created, we can test it with a complex prioritization question that requires multistep reasoning:

> Based on TECH_DEBT.md, which items should we prioritize for our 2-week sprint?
> Use the tech-debt-analyzer agent to check error frequency, customer impact,
> support ticket volume, and alignment with engineering standards.

Watch how the agent orchestrates multiple queries:

And will give you a comprehensive analysis of the local files combined with Elasticsearch data:

This demonstrates why a single query fails and an agent succeeds: It orchestrates five or more queries across different indices, correlates the data, and synthesizes a recommendation that contradicts the naive "fix highest error count" approach.

By typing /context, we can see how much context each of the MCP tool's definitions uses and our subagent's prompt. Keep an eye on this overhead when creating subagents.

Start planning

We can now start planning using local files, the internet, and our Elasticsearch knowledge as information sources.

Ask something like:

"Based on our requirements defined in REQUIREMENTS.md, use the planning agent
to create a detailed implementation plan, prioritizing tasks according to
business impact. Use the tech-debt-analyzer agent to query about internal
company knowledge and make analytical queries about error patterns and
customer impact."

Note that Claude decides to run the Elasticsearch data analysis and the local documentation reading in parallel, following the hub-and-spoke orchestration pattern.

After the analysis, you should get a plan that prioritizes based on actual business data rather than on assumptions. This context will make your AI coding experience much more reliable, as you can feed this plan directly to the agent and execute step by step:

The more details you provide and the more focused the instructions are, the better the quality of the plan will be. If you have an existing codebase, it will suggest the code changes.

Conclusion

Subagents are a great tool to offload specific tasks where we only need the final result for the main chat (without going through how we got there), keeping the chat flow focused.

By choosing the right orchestration pattern (sequential, parallel, or hub-and-spoke) and handling the context properly, we can build efficient and maintainable agent systems.

Elastic Agent Builder and its MCP feature allow us to access our data using a retrieval subagent to facilitate planning and coding by combining local (files, source code), external (internet), and internal (Elasticsearch) sources. The key insight is that agents add value not for simple queries but when you need multistep reasoning that builds on previous results and synthesizes information from multiple sources.

Resources

• Claude Code Subagents
• Elastic Agent Builder
• Agent Builder MCP

In HNSW, search proceeds by iteratively expanding candidate nodes in the graph, maintaining a bounded set of nearest neighbors discovered so far. Each expansion has a cost (vector operations, random seeks to disk, and more), and the marginal benefit of that cost tends to decrease as the search progresses.

One way to optimize HNSW graph traversal is to stop searching when the marginal likelihood of finding new true neighbors doesn’t increase. For this reason, in Elasticsearch 9.2 we introduced a new early termination mechanism. This stops the search process when visiting graph nodes doesn’t provide enough new nearest neighbors, consecutively, for a fixed number of times.

This article guides you through how we improved over the mentioned early termination mechanism in HNSW to make it better suited for different datasets and data distributions.

Early termination in HNSW

In HNSW, search proceeds by iteratively expanding candidate nodes in the proximity graph, maintaining a bounded set of nearest neighbors discovered so far, until it either has visited the whole graph or meets some early stop criteria.

Early termination is therefore not necessarily always an optimization, it’s part of the search algorithm itself. The moment we decide to stop determines the balance between efficiency and recall. In Elasticsearch, there are already a number of ways a query on HNSW can early terminate:

• A fixed maximum number of nodes is visited.
• A fixed timeout is reached.

While simple and predictable, these rules are largely agnostic to what the search is actually doing. Also they’re used mostly to make sure that the query finishes in reasonable time for the end user.

In a previous blogpost, we introduced the concept of redundancy in HNSW. In short, redundant computations occur when HNSW continues to evaluate new candidate nodes that don’t result in finding more nearest neighbors.

Patience: Measuring progress instead of effort

The notion of patience reframes early termination around progress rather than effort.

Instead of asking:

“How many steps have we taken?”

The new question becomes:

“What is the amount of computation we accept to waste, until we lose hope?”

During HNSW search, early exploration typically produces peak improvements to the top-k candidate set. During first steps of the HNSW graph exploration, the set of neighbors is continuously updated as the algorithm keeps discovering nearer and nearer neighbors to the query vector. Over time, these improvements become rarer as the search converges. Patience-based termination monitors this pattern and terminates the search once improvements have ceased for a sustained period.

In practice, while visiting the HNSW graph we also compute the queue saturation ratio as we hop through candidate nodes. This measures the percentage of nearest neighbors that were left unchanged while visiting the most recent graph node (or the inverse of the number of new neighbors introduced during the last iteration). When such a ratio becomes too big for too many consecutive iterations, we stop visiting the graph.

Conceptually, patience treats HNSW search as a diminishing returns process. When returns flatten out, continuing to explore the graph yields little benefit.

This framing is powerful because it ties termination directly to observable outcomes rather than to arbitrary fixed limits.

The benefit of using this smart early termination technique is that HNSW graph explorations tend to visit a smaller number of graph nodes while retaining an almost perfect relative recall.

To visualize this, we can plot the amount of recall per visited node that we got with the patience based early termination (labeled as et=static), when compared to the default HNSW behavior (labeled as et=no) on a couple of datasets, FinancialQA and Quora, and models, JinaV3 and E5-small.

Static thresholds and HNSW dynamics

In practice, in Elasticsearch this is implemented using static thresholds. One threshold refers to the saturation threshold: that is, the ratio of saturation that we consider suboptimal. The other threshold refers to the number of consecutive graph nodes that we allow to be visited while still having a suboptimal queue saturation: that is, the patience threshold.

When we introduced this early termination strategy in Elasticsearch 9.2, we decided to opt for conservative defaults, so as to let the recall as much as possible, while still gaining in terms of latency and memory consumption. For this reason, we set the saturation threshold to be 100% and the patience threshold to be set as a (bounded) 30% of the num_candidates in the KNN query.

In many scenarios, these settings resulted to work nicely; however, two queries requesting the same number of neighbors might have radically different convergence behaviors. Some queries encounter dense local neighborhoods and saturate quickly; others must traverse long, sparse paths before finding competitive candidates. The latter resulted to be the most difficult to handle effectively.

As a result, we sometimes noticed:

• Over-exploration for easy queries.
• Premature termination for hard queries.

Therefore, we figured that fixed threshold values encode global assumptions about convergence, whereas we could make HNSW better adapt to different dynamics.

Making HNSW early termination adaptive

Adaptive early termination approaches this problem from a different angle. Instead of enforcing predefined stopping thresholds, the algorithm infers when to stop from the search dynamics themselves.

So instead of comparing the queue saturation ratio between two consecutive candidates, we decided to introduce both an instant smoothed discovery rate  $d_{q,i} $ (how many new neighbors were introduced for a query q, in the last visit i) together with rolling mean $\mu_{q,i}$ and standard deviation $\sigma_{q,i}$ of such a discovery rate during the graph visit (using Welford’s algorithm). These statistics about the discovery rate are calculated per query, so that this information can be used to decide different degrees of patience for each query.

The previously static thresholds become adaptive to the discovery rate statistics: The saturation threshold becomes the rolling mean plus the standard deviation; whereas we make the patience adapt and scale inversely with the standard deviation.

The early exit rules remain the same; the saturation happens when the instant discovery rate is lower than the adaptive saturation threshold. The graph visit stops if the saturation persists for a number of consecutive candidate visits that’s larger than the adaptive patience.

This way, we obtain a behavior that doesn’t depend on the num_candidates parameter in the KNN query (which might be always set or left as the default, regardless of early exit) and that better adapts to each query and vector distribution dynamically.

The recall per visited node on FinancialQA and Quora with the adaptive strategy (labeled as et=adaptive) reports a higher recall per visited node, when compared to the static strategy (et=static) and the default HNSW behavior (et=no).

Adaptive early termination is turned on by default in Elasticsearch 9.3 for HNSW dense vector fields (and it can eventually be turned off via the same index level setting).

Integrations configure Filebeat inputs to do the data collection. To collect data from HTTP APIs, we’ve often used the HTTP JSON input. However, even basic listing APIs can differ greatly in the details, and the HTTP JSON input's model of YAML-configured transformations can make it awkward and sometimes impossible to express the required collection logic.

The Common Expression Language (CEL) input was introduced to allow more flexible interaction with HTTP APIs. CEL is a language designed to be embedded in applications that require a fast, safe, and extensible way to express conditions and data transformations. The CEL input lets an integration builder write one expression that can read settings, keep track of its own state, make requests, process responses, and ultimately return events ready to ingest.

In this article, we’ll look at how CEL differs from other programming languages, how we’ve extended it for the CEL input, and the flexibility and power that gives you to express your data collection logic.

CEL and how it works in the input

CEL is an expression language. It has no statements. When you write CEL, you don’t tell it what to do by writing statements, you tell it what value to produce by writing an expression. Every CEL expression produces a value, and smaller expressions can be combined into a larger expression to produce a result according to more complex rules. Later, we’ll see how to use expressions for things that may be written with statements in other languages.

CEL is intentionally a non-Turing complete language. It doesn’t allow unbounded loops. Later, we’ll see how you can process lists and maps using macros, but by avoiding unbounded loops, the language guarantees predictable and limited execution time for individual expressions.

The CEL input is configured with a CEL program (an expression) and some initial state. The state will be provided as input to the program. The program is evaluated to produce an output state. If the output state includes a list of events, those will be removed and published. The rest of the output state will be used as the input for the next evaluation. If the output state includes one or more events and the flag want_more: true, the next evaluation will be performed immediately; otherwise, it will sleep for the rest of the configured interval time before continuing. Here’s a simplified diagram of the input’s control flow:

The output of each evaluation will be passed forward as the input to the next evaluation, for as long as the input runs. Output data under the key "cursor" will be persisted to disk and reloaded after the input is restarted, but the rest of the state will not be preserved across restarts.

The CEL language itself has limited functionality and avoids side effects, but it is extensible. The cel-go implementation adds some functionality, such as optional syntax and types. The Mito library builds on cel-go and adds more functionality, including the ability to make HTTP requests. The CEL input uses Mito’s version of CEL.

Working with Mito

To build or debug an integration using the CEL input, the most important thing to understand is what output state your CEL program will produce for a given input state. During development, it can be cumbersome to have your CEL program run by the input, surrounded by the full Elastic stack. One way to achieve a faster feedback loop is to use Mito’s command-line tool, which will let you run a CEL program directly and see the output it produces for a given input.

Mito is written in Go and can be installed as follows:

go install github.com/elastic/mito/cmd/mito@latest

When you run a CEL program with Mito, you typically give it two files: a JSON file with the initial input state, and another file with the source code of your CEL program:

mito -data state.json src.cel

For easier copy and pasting, the examples in this article are written as single commands that have the shell create temporary files on the fly, by wrapping the content of each file in <(echo '...content...'). In your own development, working with actual files will be easier.

Fetching issues data from GitHub

The following example includes a full CEL program that will fetch data about issues from the GitHub API. Its initial input state has a URL for the API endpoint, and some information about how it should handle pagination. The CEL program uses the data in the input state to generate a request. It will decode the response, produce events from it, and return them as part of its output state.

mito -data <(echo '
  {
    "url": "https://api.github.com/repos/elastic/integrations/issues",
    "per_page": 3,
    "max_pages": 3
  }
') <(echo '
  int(state.?cursor.page.orValue(1)).as(page,
    (
      state.url + "?" + {
        "state": ["all"],
        "sort": ["created"],
        "direction": ["asc"],
        "per_page": [string(state.per_page)],
        "page": [string(page)],
      }.format_query()
    ).as(full_url,
      request("GET", full_url).with({
        "Header": {
          "Accept": ["application/vnd.github+json"],
          "X-GitHub-Api-Version": ["2022-11-28"],
        }
      }).do_request().as(resp,
        resp.Body.decode_json().as(data,
          state.with({
            "events": data.map(i, {
              "html_url": i.html_url,
              "title": i.title,
              "created_at": i.created_at,
            }),
            "cursor": { "page": page + 1 },
            "want_more": size(data) == state.per_page && page < state.max_pages,
          })
        )
      )
    )
  )
')

Its first evaluation produces the following output:

{
  "cursor": {
    "page": 2
  },
  "events": [
    {
      "created_at": "2018-09-14T09:47:35Z",
      "html_url": "https://github.com/elastic/integrations/issues/3250",
      "title": "Increase support of log formats in haproxy filebeat module"
    },
    {
      "created_at": "2019-02-06T12:37:37Z",
      "html_url": "https://github.com/elastic/integrations/issues/487",
      "title": "ETCD Metricbeat module needs polishing and grooming"
    },
    {
      "created_at": "2019-08-13T11:33:11Z",
      "html_url": "https://github.com/elastic/integrations/pull/1",
      "title": "Initial structure"
    }
  ],
  "max_pages": 3,
  "per_page": 3,
  "url": "https://api.github.com/repos/elastic/integrations/issues",
  "want_more": true
}

The events will be removed, and when run in the CEL input, they’ll be published for ingestion. The rest of the output will be provided to the next CEL program evaluation as its input state.

To understand how that CEL program works, we’ll look at some smaller CEL examples and discuss more details of how the CEL input operates.

CEL basics

In the CEL language, there are no statements; there are only expressions. Every successful CEL expression evaluates to a final value. Here’s one of the smallest CEL expressions you can write, along with its output:

mito <(echo '
  "hello" + " " + "world"
')

"hello world"

Many simple expressions are intuitive. Mathematical operations are only supported on values of the same type (for example, int with int), so convert types as you need (here from int to double):

mito <(echo '
  double((1 + 2) * (3 + 4)) / 2.0
')

10.5

There are no variables in the CEL language, but an expression can be given a name and used in a larger expression with the help of Mito’s as macro. In this example, the expression (1 + 1) evaluates to the value 2, and .as(n, ...) gives that value the name n for use in the expression "one plus one is "+string(n):

mito <(echo '
  (1 + 1).as(n, "one plus one is "+string(n))
')

"one plus one is 2"

It's also possible to accumulate information in a map and use it later in the expression, as demonstrated here using with:

mito <(echo '
  { "key": "value" }.with({ "key2": "value2" }).as(data,
    {
      "data": data,
      "size": size(data),
    }
  )
')

{
  "data": {
    "key": "value",
    "key2": "value2"
  },
  "size": 2
}

Look at that example again. Notice that the nested part, ({ "data": data, "size": size(data), }), gives us the shape of the final value. It’s a map with the keys "data" and "size". The values for those keys depend on data, which is defined by the outer part of the expression. Reading CEL expressions from the inside out can help to quickly see what they’ll return.

CEL has no control flow statements, like if, but conditional branching can be done with the ternary operator:

mito <(echo '
  1 + 1 < 12 ? "few" : "many"
')

"few"

Unbounded loops and recursion are not supported, as CEL is not a Turing complete language. That makes execution time predictable and proportional to the size of the input data and the expression complexity.

Although unbounded loops are not possible in individual CEL expressions, you can process lists and maps using macros like map:

mito <(echo '
  [1, 2, 3].map(x, x * 2)
')

[2, 4, 6]

In this section, we’ve covered:

• Strings, numbers, lists, and maps.
• String concatenation.
• Mathematical operations.
• Type casting.
• Conditionals.
• Naming sub-expressions.
• Processing collections.

Next, we’ll look at how to make HTTP requests.

Requests

Mito extends CEL with the ability to make HTTP requests:

mito <(echo '
  get("https://example.com").as(resp, string(resp.Body))
')

"..."

Requests can be explicitly constructed before they’re executed. That makes it possible to use different HTTP methods and to add headers and a body.

In this example, we build a URL with the help of format_query, add a header to the request, and parse the response body with decode_json. When given the -log_requests option, Mito will log detailed information in JSON format about each request and response.

mito -log_requests <(echo '
  request("GET",
    "https://postman-echo.com/get?" + {
        "q": ["query value"]
     }.format_query()
  ).with({
    "Header": { "Accept": ["application/json"] }
  }).do_request().as(resp, {
    "status": resp.StatusCode,
    "data": resp.Body.decode_json(),
  })
')

{"time":"...","level":"INFO","msg":"HTTP request",...}
{"time":"...","level":"INFO","msg":"HTTP response",...}
{
  "data": {
    "args": {
      "q": "query value"
    },
    "headers": {
      "accept": "application/json",
      "accept-encoding": "gzip, br",
      "host": "postman-echo.com",
      "user-agent": "Go-http-client/2.0",
      "x-forwarded-proto": "https"
    },
    "url": "https://postman-echo.com/get?q=query+value"
  },
  "status": 200
}

Managing state and evaluations

Now that we’ve covered how to make requests and the CEL basics required to produce our desired output state, let’s take a closer look at what we should put into the output state and how that lets us direct later processing.

An integration’s CEL program needs to make sure its output state is suitable for use as the input of the next evaluation. Configuration sets the initial state, and that should be repeated in the output with any appropriate changes. An easy way to do that is to use state.with({ ... }), to repeat the state map with some overrides. A common pattern for small programs is to wrap the whole program in state.with(), so that state propagation doesn’t have to be repeated in each branch that generates output data (for example, success, errors).

When there are state values that are initialized by an evaluation rather than hard-coded in the initial input state, the program will need to check for an existing value before setting the initial one. That’s something that the support for optional syntax and types can help with. By using a question mark before the field name in a map key, the access becomes optional: It may or may not resolve to a value, but further optional accesses are possible and it’s easy to supply a default if no value is present:

mito -data <(echo '{}') <(echo '
  int(state.?counter.orValue(0)).as(counter,
    state.with({
      "counter": counter + 1,
      "want_more": counter + 1 < 3,
    })
  )
')

{ "counter": 1, "want_more": true }
{ "counter": 2, "want_more": true }
{ "counter": 3, "want_more": false }

In that example, the counter value read from state is cast to int because all numbers are serialized in the state as floating point numbers, in keeping with conventions established by JSON and JavaScript’s Number type. It should also be noted that "want_more": true is honored here by Mito, but when run in the CEL input, the evaluation would only be repeated if the output also contains events.

It’s a requirement of CEL programs run by the CEL input that they return an "events" key in their output map. Its value can be a list of event maps, an empty list, or a single event map. The single event case is usually used for errors. The event will be published by the input, but its value will also be logged, and if it sets an error.message value, that will be used to update the integration’s Fleet health status. If your program produces a single non-error event, it’s best to wrap it in a list.

Take another look at the output of our GitHub issues program from earlier:

{
  "url": "https://api.github.com/repos/elastic/integrations/issues",
  "per_page": 3,
  "max_pages": 3,
  "cursor": {
    "page": 2
  },
  "events": [
    { ... },
    { ... },
    { ... }
  ],
  "want_more": true
}

The program effectively managed its state, by:

• Repeating initial state values in url, per_page, and max_pages.
• Adding state that should be persisted across restarts in cursor.page.
• Returning events ready to publish in the events list.
• Requesting immediate re-evaluation with want_more: true.

Now that you understand optional access and state management, as well as CEL basics and HTTP requests, the full GitHub issues program should be readable. Try running it with Mito and experimenting with some changes.

Review and resources

In this article, we looked at what the CEL language is and how it has been extended in the Mito library for use in the CEL input. We saw the flexibility of CEL in an example program that fetches issues information from the GitHub API, and went through all the details necessary to understand that program, covering access to settings in the initial state, interaction with HTTP APIs, returning events to be ingested, and managing the state for later program executions.

To learn more and build integrations using the CEL input, there are a number of resources worth exploring:

• CEL input - Filebeat documentation
• Mito documentation
• Common Expression Language - cel.dev website
• Create an Integration - Elastic documentation

And perhaps the most valuable resource for building integrations with the CEL input is the CEL code of existing Elastic integrations, which can be found on GitHub:

cel.yml.hbs files in the Elastic integrations repository - GitHub

1. The new Swift update is here! Developers are eager to try out the new features.
2. The new Swift update is here! The new album will drop next month.

With this added context, we should be able to resolve the name "Swift" to the correct entity.

In the previous post, we set up our watch list and enriched the entities with additional context. Looking at our examples above, we need to have at least the following two entities in the list: Taylor Swift and Swift Programming Language. We also covered how we extract entity mentions from text. Both of these examples would extract "Swift". With these ingredients in place, the enriched watch list, and the extracted entities, we’re finally ready to introduce the star of the show: entity matching.

Remember: This is an educational prototype designed to teach entity-matching concepts. Production systems might use different large language models (LLMs), custom matching rules, specialized judgment pipelines, or ensemble approaches combining multiple matching strategies.

The problem: Why matching is hard

Human language is a remarkable thing. One of the most interesting properties of it is its endless creativity. We can generate and understand an infinite number of new sentences. Is it any wonder, then, that exact matches in entity resolution are rare? Authors strive to be creative when they can. It would get quite tedious if we had to write and read full names whenever an entity is mentioned. So, while exact matches are easy, the reality is that we need a more sophisticated approach to entity resolution: one that’s robust enough to handle at least some of the boundless creativity of human authors. That’s why we separate the problem into two steps: Use Elasticsearch to retrieve plausible candidates at scale, and then use an LLM to judge whether those candidates truly refer to the same real-world entity.

The solution: Three-step matching with transparent LLM judgment

We’re in the midst of a paradigm shift in how we use computers. Just as the rise of the internet took us from localized computing to a globally connected network, generative AI (GenAI) is fundamentally changing how content, code, and information are created. In fact, the educational prototype that accompanies this series was almost exclusively "vibe coded" using an LLM with careful prompting by the author. This is not to say that LLMs have or even will reach the kind of productivity inherent with human language, but it does mean that we now have a powerful resource to help with entity resolution.

A common pattern we use with GenAI is retrieval augmented generation (RAG). Here, retrieval means retrieving entity candidates (not generating answers), and the LLM is used strictly for match evaluation and explanation. While we could ask an LLM to help us with end-to-end entity resolution, that’s a costly approach, both in terms of time and money. RAG helps LLMs do their work by using more efficient ways to provide context to the LLM, thereby empowering the LLM to efficiently help with entity resolution.

For the retrieval part of RAG, we again turn to Elasticsearch. We first find potential matches using a combination of exact matching, matching against aliases, and hybrid search, which combines keyword and semantic search. Once we find these potential matches, we send them to an LLM for judgment. The LLM acts as the final match evaluator. We also make the LLM explain its reasoning, an important differentiator with other entity resolution systems. Without these explanations, entity resolution is a black box; with them, we can see for ourselves why a match makes sense.

Key concepts: Three-step matching, hybrid search, and transparent LLM judgment

What is three-step matching? At the onset of this project, we hypothesized that semantic search will be a crucial part of the system, but not every match requires such sophisticated search. In order to find matches efficiently, we take a progressive approach to the problem. First, we check for exact matches using keyword search. If we find such a match, our work is done and we can move on. If exact matching fails, we turn to alias matching. In the prototype, alias matching is also done using exact matching with keywords, for simplicity. In production, you might expand this step with normalization, transliteration rules, fuzzy matching, or curated alias tables. If we still haven't found a potential match in the first two steps, then it's time to bring in semantic search via Elasticsearch's hybrid search with reciprocal rank fusion (RRF).

What is hybrid search? In Elasticsearch, we can use semantic search to find meaningful matches that take context into account. Elasticsearch is widely used for vector search and hybrid retrieval. Semantic similarity is powerful for meaning, but it’s not a substitute for structured filtering (for example, by time ranges, locations, or identifiers), and it’s often unnecessary when an exact match is available. Elasticsearch made its mark with lexical search, which is great at tasks where semantic search doesn't fit. To take full advantage of both approaches, we use lexical search alongside semantic search in a single hybrid query. We then merge the results to find the most likely matches using RRF. In the prototype, the top two results become potential matches that can be sent for LLM judgment.

Why LLM judgment? LLM judgments and explanations allow our system to handle ambiguity and context transparently. This is vital for cases like "the president", which could refer to multiple entities, depending on the context, but it also makes things like nicknames and cultural variations work well in the system. Finally, when we consider mission-critical tasks, like identifying entities from sanctions lists, we need to know why a match was accepted in order to trust the system. Crucially, the LLM does not search the full corpus; it evaluates only the small set of candidates returned by Elasticsearch.

Real-world results: Matching with LLM reasoning

A major challenge for any natural language processing task is the creation of a golden document, an "answer key" that tells us what the expected results are. Without this, it's next to impossible to judge how well a system performs on a task, but creating such a document can be a laborious process. For the entity resolution prototype, we turned again to GenAI to help set up data we could test against.

We first defined several challenge types, such as nicknames and transliteration, and then asked the LLM to create a tiered collection of datasets that would get progressively larger and more challenging for the system. The creation of the datasets was less straightforward than one might hope. The LLM had a strong propensity for "cheating" by making it too easy to get the right answer. For example, one of the challenge types focused on semantic context. This type included things like resolving "Russian author" to "Leo Tolstoy". The LLM incorrectly put "Russian author" as an alias for "Leo Tolstoy", which negated the need for hybrid search to find the match.

After several refactorings to fix issues like this, we had five dataset tiers to work with. Tiers 1–4 were progressively larger with more challenge types. Tier 5 was the "ultimate challenge" dataset, made up of the trickiest examples from all challenge types. All of the test data is available in the comprehensive evaluation directory.

To evaluate our prompt-based entity resolution approach, we focused our attention on the tier 4 dataset. An important note is that the evaluation was conducted as a controlled experiment so that we could focus on entity match quality. The watch list data was pre-enriched with context, and entities were extracted from the article ahead of time. This ensured that evaluation was focused on matching rather than on extraction accuracy. This isolates match quality; end-to-end performance would additionally depend on extraction recall and enrichment quality.

Evaluation dataset

The tier 4 evaluation dataset provides a comprehensive test of the system's capabilities:[1]

• Watch list entities: 66 entities across diverse types (people, organizations, locations).
• Test articles: 69 articles covering real-world entity resolution scenarios.
• Expected matches: 206 expected entity matches across all articles.
• Challenge types: 15 different challenge types testing various aspects of entity resolution.

The challenge types included in the dataset are:

• Nicknames: "Bob Smith" → "Robert Smith" (seven articles).
• Titles and honorifics: "Dr. Sarah Williams" → "Sarah Williams" (five articles).
• Semantic context: "Russian author" → "Leo Tolstoy" (eight articles).
• Multilingual names: Handling names in different scripts (six articles).
• Business entities: Corporate name variations (seven articles).
• Executive references: "Microsoft CEO" → "Satya Nadella" (five articles).
• Political leaders: Title-based references (five articles).
• Initials: "J. Smith" → "John Smith" (three articles).
• Name order variations: Different name ordering conventions (three articles).
• Truncated names: Partial name matches (three articles).
• Name splitting: Names split across text (three articles).
• Missing spaces/hyphens: Formatting variations (two articles).
• Transliteration: Cross-script name matching (two articles).
• Combined challenges: Multiple challenges in one article (six articles).
• Complex business: Hierarchical business relationships (five articles).

Let's see how prompt-based entity resolution performed.

Overall performance

The results show that there's a lot of promise with LLM-powered match evaluation, but they also reveal a significant reliability issue. Because each candidate pair must be evaluated by the LLM, failures in structured output can suppress acceptance and recall even when retrieval is working well.

The error rate problem

Recall that the first step we take in the prototype is to create potential match pairs using Elasticsearch. Each of these potential matches needs to be evaluated by the LLM. To efficiently process all of those matches, we batch the LLM calls together. This reduces API costs and latency, but there’s also an increased risk of getting malformed JSON in the output. As batch size increases, the JSON becomes longer and more complex, making it more likely that the LLM will generate invalid JSON. This is where the 30% error rate stems from. In the evaluation, we used a batch size of five matches per request. Even with this conservative batch size, we still see JSON parsing failures, which skews the evaluation results significantly.

What's next: Optimizing LLM integration

Now that we've matched entities using semantic search and LLM judgment, we have a complete entity resolution pipeline. This approach introduces a new failure mode, however, when the model’s judgment is correct, but its output isn’t usable. We can optimize the LLM integration for better reliability and cost efficiency. In the next post, we'll explore how to use function calling for structured output, which provides guaranteed structure and type safety while reducing errors and costs.

Try it yourself

Want to see entity matching in action? Check out the Entity Matching notebook for a complete walk-through with real implementations, detailed explanations, and hands-on examples. The notebook shows you exactly how to match entities using three-step search, hybrid search with RRF, and LLM-powered judgment with reasoning.

Remember: This is an educational prototype designed to teach the concepts. When building production systems, consider additional factors, like model selection, cost optimization, latency requirements, quality validation, error handling, and monitoring, which aren't covered in this learning-focused prototype.

Notes

1. These datasets are synthetic and designed for education; they approximate real challenges but are not representative of any single production domain.

Our benchmarks on a 20M document corpus show that Elasticsearch delivers up to 8x higher throughput than OpenSearch for filtered vector search, while also achieving higher Recall@100 across the configurations we tested. Context engineering depends on more than fast vector retrieval. Teams also need strong relevance controls, like hybrid search and filtering, operational simplicity, and predictable performance, as workflows iterate. But because agents often run retrieve, reason, retrieve loops many times per request, retrieval latency becomes a multiplier, so improvements here translate directly into better end-to-end responsiveness and lower cost.

For context engineering, retrieval isn’t a one-time step. Agents and applications repeatedly run loops, such as retrieve → reason → retrieve, to refine queries, verify facts, assemble grounded context, and complete tasks. This pattern is common in agentic workflows and iterative retrieval augmented generation (RAG). Because retrieval may be invoked many times per user request, it adds delay to the response and/or increases infrastructure costs.

Why is vector search performance critical?

Imagine a shopping assistant answering the question, “I need a carry-on backpack under $60 that fits a 15-inch laptop, is water resistant, and can arrive by Friday.”

In production, the assistant rarely issues one vector query and stops. It runs a retrieval loop to build the right context, and each step is typically constrained by filters, like availability, region, shipping promise, brand rules, and policy eligibility.

Step 1: Interpret intent and translate to constraints.

The agent turns the request into structured filters and a semantic query, such as:

• Filters: In stock, deliverable to the user’s postcode, delivery by Friday, price under $60, valid listing
• Vector query: “Carry-on backpack 15-inch laptop water resistant”

Step 2: Retrieve candidates, and then refine.

It often repeats retrieval with variations to avoid missing good matches:

• “travel backpack carry on laptop sleeve”
• “water resistant commuter backpack 15 inch”
• “lightweight cabin backpack”

Each query uses the same eligibility filters, because retrieving irrelevant or unavailable items is wasted context.

Step 3: Expand to confirm details and reduce risk.

The agent then retrieves again to verify key attributes that affect the final answer:

• Material and water resistance wording
• Dimensions and laptop compartment fit
• Return policy or warranty constraints
• Alternate options if inventory is low

This is multistep context engineering: Retrieve, reason, retrieve, assemble.

Why latency and recall matter for context engineering

These interactions can involve dozens of filtered retrieval calls per user session. That makes per-call latency a direct multiplier on end-to-end response time, and low recall forces extra retries or causes the agent to miss eligible items, degrading answer quality.

Takeaway: In context-engineered systems, filtered approximate nearest neighbors (ANN) isn’t a single lookup. It’s a repeated operation under constraints, so vector search performance shows up immediately in latency, throughput, and cost, even when the large language model (LLM) is the most visible component.

Benchmarking

Results

In Graph 2, each dot represents one test configuration. The best results appear toward the top left, meaning higher recall with lower latency. Elasticsearch’s results are consistently closer to the top left than OpenSearch’s, indicating better speed and accuracy under the same workload settings.

Some key insights

• s_n_r_value: Shorthand for size_numCandidates_rescoreOversample (k and numCandidates set equal to numCandidates in these tests), for example, 100_500_1 means size=100, numCandidates=500 and k=500, rescore oversample=1
• Recall: Measured Recall@100 for that configuration
• Avg latency (ms): Average end-to-end latency per query
• Throughput: Queries per second
• Recall %: Relative recall lift of Elasticsearch versus OpenSearch (Elasticsearch minus OpenSearch) / OpenSearch
• Latency Xs: OpenSearch average latency divided by Elasticsearch average latency
• Throughput Xs: Elasticsearch throughput divided by OpenSearch throughput

For example, at 100_9000_1, OpenSearch averages 687 milliseconds per retrieval versus 90 milliseconds on Elasticsearch, and in a 10-step retrieval loop that’s about 10 x (687 - 90) = six seconds of additional waiting time.

See the full results.

Methodology

Using Python to send the queries and track the response timing and other statistics, we sent the following queries to the engines. Bear in mind that the performance of any vector search engine depends on how you tune its core parameters: how many candidates to consider, how aggressively to rescore, and how much context to return. These settings directly affect both recall (the likelihood of finding the right answer) and latency (how fast you get results).

In our benchmarks, we used the same candidate, rescore, and result-size settings you’d typically tune in an agentic retrieval loop, and we measured how Elasticsearch performs under that workload. We then ran OpenSearch with the same settings as a reference.

OpenSearch

GET /_search
{
  "query": {
    "knn": {
      "": {
        "vector": [...],
        "k": ,
        "method_parameters": {
          "ef_search": 
        },
        "rescore": {
          "oversample_factor": 
        },
        "filter": {
          
        }
      }
    }
  },
  "size": ,
  "_source": {
    "excludes": [
      ""
    ]
  }
}

• "size": <RESULT_SIZE>: Number of hits returned to the client. In this benchmark, result size is 100 to compute Recall@100.
• "k": <NUMBER_OF_CANDIDATES>: The number of nearest neighbor candidates.
• "ef_search": <NUMBER_OF_CANDIDATES>: The number of vectors to examine.
• "oversample_factor": <OVERSAMPLE>: How many candidate vectors are retrieved before rescoring.

Elasticsearch

GET /_search
{
  "query": {
    "knn": {
      "field": "",
      "query_vector": [...],
      "k": ,
      "num_candidates": ,
      "rescore_vector": {
        "oversample": 
      },
      "filter": {
        
      }
    }
  },
  "size": ,
  "_source": {
    "excludes": [
      ""
    ]
  }
}

• "size": <RESULT_SIZE>: Number of hits returned to the client. In this benchmark, result size is 100 to compute Recall@100.
• "k": <NUMBER_OF_CANDIDATES>: Number of nearest neighbors to return from each shard.
• "num_candidates": <NUMBER_OF_CANDIDATES>: Number of nearest neighbor candidates to consider per shard while doing knn search.
• "oversample": <OVERSAMPLE>: How many candidate vectors are retrieved before rescoring.

Example

Knn query, (100_500_1), would be as follows:

OpenSearch

GET search_catalog_128/_search
{
  "query": {
    "knn": {
      "search_catalog_embedding": {
        "vector": [...],
        "k": 500,
        "method_parameters": {
          "ef_search": 500
        },
        "rescore": {
          "oversample_factor": 1
        },
        "filter": {
          "term": {
            "valid": true
          }
        }
      }
    }
  },
  "size": 100,
  "_source": {
    "excludes": [
      "search_catalog_embedding"
    ]
  }
}

Elasticsearch

GET search_catalog_128/_search
{
  "query": {
    "knn": {
      "field": "search_catalog_embedding",
      "query_vector": [...],
      "k": 500,
      "num_candidates": 500,
      "rescore_vector": {
        "oversample": 1
      },
      "filter": {
        "term": {
          "valid": true
        }
      }
    }
  },
  "size": 100,
  "_source": {
    "excludes": [
      "search_catalog_embedding"
    ]
  }
}

The full configuration, alongside Terraform scripts, Kubernetes manifests and the benchmarking code is available in this repository in the folder es-9.3-vs-os-3.5-vector-search.

Cluster setup

We ran our tests on six e2-standard-16 cloud servers, each with 16 vCPUs and 64 GB RAM. On each server, we allocated 15 vCPUs and 56 GB RAM to each Kubernetes pod running the search engine node, with 28 GB reserved for the JVM heap.

The clusters ran Elasticsearch 9.3.0 and OpenSearch 3.5.0 (Lucene 10.3.2). Because both systems use the same Lucene version in this benchmark, the throughput and latency differences we observe cannot be attributed to Lucene alone and instead reflect differences in how each engine integrates and executes filtered k-nearest neighbor (kNN) retrieval and rescoring. We used a single index with three primary shards and one replica (so 6 shards total, 1 per node).

We also used a separate server in the same region to run the benchmark client and collect timing statistics.

The dataset

For this benchmark, we used a large-scale ecommerce-style catalog embedding dataset with 20 million documents, designed to reflect real-world filtered vector retrieval at scale.

Each document represents a catalog item and includes:

• A 128-dimensional dense vector embedding used for approximate kNN retrieval.
• Structured metadata fields used for filtering (for example, item validity and availability plus other catalog constraints) enabling the common production pattern of retrieving the nearest neighbors but only within an eligible subset.

We chose this dataset because it captures the core performance challenge we see in agentic and RAG-style systems in production: Vector similarity alone is not enough, retrieval is frequently constrained by filters, and the system must maintain high recall while keeping latency low under those constraints. Compared to smaller QA-style datasets, a 20M document corpus also better reflects the scale and candidate pressure that filtered ANN systems face in practice.

Conclusion

In modern AI architectures, especially those built around context engineering, vector search speed isn’t a minor implementation detail. It’s a multiplier. When agents and workflows iterate through retrieve → reason → retrieve, retrieval performance directly shapes end-to-end latency, throughput, and the quality of the context fed into the model.

In our benchmarks, Elasticsearch consistently delivered higher recall at lower latency than OpenSearch in scenarios where correctness depends on retrieving the right document, not just a similar vector. On a controlled dataset, the difference is clear, and in production those gains accumulate across large volumes of retrieval calls, improving responsiveness, increasing capacity headroom, and reducing infrastructure costs.

Further reading

1. What is context engineering?
2. The evolution of hybrid search and context engineering
3. The impact of relevance in context engineering for AI agents

We’re making this simple: Starting today, AutoOps is available at no cost for every self-managed Elasticsearch cluster through Elastic Cloud Connect. Whether you’re on Free, Basic, Platinum, or Enterprise, you get the same feature product. This isn’t a limited preview or a "lite" version. It’s the same product used by the largest deployments.

An investment in the self-managed community

Extending it to every user, including those on the free distribution, reflects Elastic's commitment to the success of the entire Elasticsearch community. By providing AutoOps for free, we’re investing in the stability and performance of the hundred of thousands of clusters that power the community's search and analytics workloads.

Elastic Cloud Connect enables self-managed clusters to consume Elastic Cloud services, such as AutoOps and the recently announced Elastic Inference Service (EIS), without the operational overhead of maintaining, patching, monitoring, and operating the services locally.

How AutoOps for self-managed works and what it provides

As clusters grow in complexity and size, you find yourself spending more time chasing configuration tweaks and trying to find the root cause of an issue. Monitoring tools show you metrics and leave the manual correlation to you and your favorite large language model (LLM) to find the root cause when the issue appears. AutoOps tells you what’s wrong, why, and exactly how to fix it, with real-time issue detection and specific resolution paths.

AutoOps runs on Elastic Cloud; there’s no infrastructure for you to provision or maintain. You simply run a lightweight agent on-premises to connect your cluster to the AutoOps service where operational metadata (such as node stats, cluster settings, and shard states) is shipped to AutoOps in real time to provide insights and recommendations. Your data never leaves your environment.

AutoOps vs. Stack Monitoring for self-managed users

Stack Monitoring provides the essential telemetry and basic monitoring for your nodes and indices, showing you the trend over time for various metrics, and alerts you when thresholds are crossed, yet it often leaves the diagnostic burden on the engineer. AutoOps offers a complete picture of cluster health by correlating all relevant metrics. This provides valuable insights and clear instructions on how to resolve issues when they occur.

Faster root cause analysis

Your cluster was humming along, but it suddenly got red in the middle of the night (and, as usual, nothing changed the days before).

• With Stack Monitoring: A built-in alert will notify you when your cluster health turns red. To find the cause, you need to dig into your cluster logs and turn to Dev Tools to look at your shard allocations to understand why that primary shard couldn’t be allocated. Looking at your alerts history, you see another one informing you that your hot nodes reached 80% disk utilization two days ago. You can’t find any disk usage charts to learn more about your disk fill rate, you only know that your disk reached 90% utilization in the meantime, and when one of your data streams needed to roll over, a new backing index was created, but no shards could be allocated to any of your nodes.
• With AutoOps: The system notifies you when your cluster turns red (1). Looking at the timeline, you immediately realize that this happened because you failed to take action on the previous watermark events that AutoOps raised, namely the high watermark events (2) that started happening recently, and the low watermark ones (3) that built up over the past few days. It is now straightforward for you to know what you need to do to get your cluster back to green.

Higher signal to noise ratio

Keeping your Elasticsearch cluster healthy is most probably your main concern. Yet, it’s not uncommon for the health status to sometimes flap between green and yellow (and sometimes red), and the cause is not always worthy of your time.

• With Stack Monitoring: The built-in “Cluster Health” alert will continue to be raised on each health transition from green to either yellow or red. In some situations, like frequent index creations, this can create a lot of repeated and undesired noise. Also, and more importantly, there’s no distinction between a yellow and a red status.
• With AutoOps: There are dedicated “Status Red” and “Status Yellow” events, with different severities. The latter can be customized in many different ways to fit your use case, as shown in the screenshot below:
  1. Since the cluster health can turn yellow only for a brief duration, you can decide for how long to ignore the yellow status before being notified (for example, five minutes, in the screenshot below).
  2. Furthermore, there are a lot of legitimate operations that Elasticsearch does all the time and that make the cluster turn yellow. You can pick any of the operations you don’t want to be notified about when your cluster turns yellow because of them (for example, adding replicas, relocating replicas, or closing or opening an index, among others).
  3. Finally, and most importantly, if you have several clusters to manage, you don’t need to configure this for all of them separately; you simply decide to which ones this configuration should be applied. Simple, powerful!

More insightful correlations and comparisons of node metrics

When your cluster runs on more than just a handful nodes, you often need to see how they perform against each other, especially when looking at search and indexing performance.

• With Stack Monitoring: The Nodes list doesn’t allow you to focus on a specific data tier and doesn’t provide any search or indexing performance metrics that you can sort on. These metrics are available, but only once you drill into a specific node, which doesn’t allow you to easily compare node performance against each other.
• With AutoOps: The Nodes view allows you to select nodes from a specific data tier. It also provides you with over 50 metrics visualizations, among them search and indexing performance, which give you exactly the visual cues that you need to understand how each node performs against each other and whether there are any struggling nodes that require your attention. In the screenshot below, we can see that some nodes are indexing at double the rate of some others and that search latency is building up at four times the latency of the fastest nodes.

Quick overview of the main differences

Here’s a glimpse at the notable differences between AutoOps and Stack Monitoring, but you can find a more detailed breakdown in our official documentation.

Start now: Five-minute installation

Connecting your cluster takes minutes, regardless of your license type:

1. Log in to your free Elastic Cloud account, or sign up for one.
2. Choose how to connect your cluster: Elastic Cloud on Kubernetes (ECK), Kubernetes, Docker, or Linux.
3. Enter your Elasticsearch cluster endpoint, and run the single command to install and run the lightweight Elastic agent.
4. Access AutoOps in your Elastic Cloud account.

For more details on AutoOps and instructions on connecting your self-managed cluster, read our product documentation.

Reach out if you have any questions

Feel free to reach out to us to share your questions and ideas via our Slack community, by posting on our Discuss forum, or by clicking the “Give Feedback” button on the AutoOps product page. If you’re connecting a paid self-managed Platinum or Enterprise cluster, you can contact support within your Elastic Cloud account.

Read more

If you’re interested to learn more about AutoOps and what it can do for you, please head to the official AutoOps documentation and the following Elastic Search Labs articles:

• AutoOps makes every Elasticsearch deployment simple(r) to manage
• AutoOps: A journey to simplify self-managed Elasticsearch management
• Leveraging AutoOps to detect long-running search queries

Start using AutoOps for free

By the end, you’ll have a working agent that can search your crawled pages, cite relevant passages, and answer questions grounded in your content, no custom chunking or embedding pipeline required.

In this guide, you’ll:

1. Start an Elasticsearch Serverless project.
2. Create an index using the new semantic_text field powered by Jina Embeddings v5.
3. Crawl any website using Elastic Crawler Control (a.k.a. Crawly) (an open source UI + API wrapper around the Elastic Open Web Crawler).
4. Chat with that data using the Elastic Agent Builder in Kibana.

What you’ll walk away with:

• A repeatable pattern you can point at any website/docs source.
• Chat that stays grounded in your content.

Prerequisites

• An Elasticsearch Serverless (Search) project + an API key with write permissions.
• Docker + Docker Compose (to run the crawler UI).
• git (to clone the repo).

1. Start an Elasticsearch Serverless project

First, we need a serverless project to host our data.

1. Log in to your Elastic Cloud Console.

2. Click Create project.

3. Select Search as the project type. (This type is optimized for vector search and retrieval.)

4. Give it a name (for example, es-labs-jina-guide), and click Create.

5.  Important: Save the Elasticsearch endpoint and API Key provided when the project is created. You’ll need these for the crawler.

2. Create the index

Elasticsearch Serverless supports semantic_text, which handles chunking and embedding generation automatically. We’ll use the .jina-embeddings-v5-text-small model that’s hosted on GPUs on Elastic Inference Service.

Create the index with the semantic_text field. This tells Elastic to automatically vectorize content put into the field property using the inference endpoint we just created.

In Kibana Dev tools run:

PUT furnirem-website
{
  "mappings": {
    "_meta": {
      "description": "Each document represents a web page with the following schema: 'title' and 'meta_description' provide high-level summaries; 'body' contains the full text content; 'headings' preserves the page hierarchy for semantic weighting. URL metadata is decomposed into 'url_host', 'url_path', and 'url_path_dir1/2/3' to allow for granular filtering by site section (e.g., 'blog' or 'tutorials'). 'links' contains extracted outbound URLs for discovery. Crawl timestamp: 2026-01-26T12:54:16.347907."
    },
    "properties": {
      "body_content": { 
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          },
          "semantic_multilingual": {
            "type": "semantic_text",
            "inference_id": ".jina-embeddings-v5-text-small" 
          }
        }
      },
      "headings": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          },
          "semantic_multilingual": {
            "type": "semantic_text",
            "inference_id": ".jina-embeddings-v5-text-small"
          }
        }
      },
      "title": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          },
          "semantic_multilingual": {
            "type": "semantic_text",
            "inference_id": ".jina-embeddings-v5-text-small"
          }
        }
      }
    }
  }
}

3. Run the Elastic Open Crawler

Crawly is one example of how an application can be constructed around the functionalities that the Open Web Crawler provides.

The application wraps the Elastic Open Crawler in a FastAPI service that manages crawler processes and persists execution data. A React front end provides the interface for configuring and monitoring crawls.

What happens under the hood is that the crawler service (check crawler.py) spawns JRuby processes via subprocess.Popen, allowing multiple concurrent crawls. Each execution's configuration, status, and logs are persisted to disk (for now).

Clone the repository:

git clone https://github.com/ugosan/elastic-crawler-control

Create an env.local file with your Elasticsearch credentials:

ES_URL=https://your-elasticsearch-endpoint.es.cloud
ES_API_KEY=your_api_key_here

Start the services:

docker-compose up

Access the UI at http://localhost:16700

You don’t necessarily need seed_urls unless you want to be specific, so your config can be as simple as below:

{
  "domains": [
    {
      "url": "https://furnirem.com"
    }
  ],
  "max_crawl_depth": 3,
  "max_unique_url_count": 500,
  "output_index": "furnirem-website"
}

From there, you can start a crawl on any website and check its progress:

Once it's finished, we’re ready to query the content in Elasticsearch directly or use the pages you just crawled for chatting with the website on Agent Builder.

4. Chat with data in Kibana

Now that the data is indexed and vectorized, we can start chatting with the data using the Elastic Agent Builder.

1. Open Kibana, and navigate to Agents (under the "Search" section).
2. Test the agent:
   • In the chat window, ask a question, like,"What is the difference between sparse and dense vectors?"

The agent will search your Jina-embedded data, retrieve the relevant snippets from the Search Labs blog posts, and generate an answer.

You can also chat with the data directly via Kibana API:

POST kbn://api/agent_builder/converse/async
{
  "input": "What is the difference between sparse and dense vectors?",
  "agent_id": "elastic-ai-agent",
  "conversation_id": ""
}

Use conversation_id to resume an existing conversation with an agent in Elastic Agent Builder. If you don’t provide it on the initial request, the API starts a new conversation and returns a newly generated ID in the streaming response.

Summary

You now have a working “chat with your website” stack: Your site gets crawled, indexed, auto-embedded with semantic_text + Jina v5, and surfaced through an agent in Kibana that answers questions grounded in your pages.

From here, you can point the same setup at docs, support content, or internal wikis and iterate on relevance in minutes.

The family includes two models:

• jina-embeddings-v5-text-small
• jina-embeddings-v5-text-nano

These models are the successful result of an innovative new training recipe for embedding models. They both outperform models many times their size, creating savings in memory and computing resources and responding faster to requests.

The jina-embeddings-v5-text-small model has 677M parameters, supports a 32768 token input context window, and produces 1024 dimension embeddings by default.

jina-embeddings-v5-text-nano weighs in at roughly a third of its sibling's size, with 239M parameters and a 8192 token input context window, yielding slender 768 dimension embeddings.

These two models are the best in class for overall MMTEB (Multilingual MTEB) benchmark performance. Among models with under 500M parameters, jina-embeddings-v5-text-nano is the top performer, despite having less than 250M parameters, and jina-embeddings-v5-text-small model is the leader among multilingual embedding models with under 750M parameters.

These models are available via Elastic Inference Service (EIS), via an online API, and available for local hosting. For instructions on how to access jina-embeddings-v5-text models, see the “Getting started” section, below.

Embedding models and semantic indexing dramatically increase the accuracy of search algorithms but also have a variety of other uses for tasks involving semantic similarity and meaning extraction, for example:

• Finding duplicate texts.
• Recognizing paraphrases and translations.
• Topic discovery.
• Recommendation engines.
• Sentiment and intention analysis.
• Spam filtering.
• And many others.

Features

This new model family has a number of features designed to improve relevance and reduce costs.

Task optimization

We’ve optimized the jina-embeddings-v5-text models for four broad task types:

Optimizing for one task usually means having to compromise on another, so most embedding models only have competitive performance for one kind of task. But jina-embeddings-v5-text models are able to specialize in all four areas without compromising by training task-specific Low-Rank Adaptation (LoRA) adapters.

LoRA adapters are a kind of plugin for an AI model that changes its behavior dramatically while only adding slightly to the total size. Instead of having an entire model for each task, each one with hundreds of millions of parameters, the jina-embeddings-v5-text model family lets you use just one model with a compact LoRA adapter for each task. This saves memory, storage space, and inference costs.

Truncating embeddings

We’ve trained the jina-embeddings-v5-text models using Matryoshka Representation Learning, which lets you cut your embeddings down to smaller sizes at a minimal cost to their quality.

By default, jina-embeddings-v5-text-small generates 1024-dimension embedding vectors, each represented by a 16-bit number, making every embedding 2KB in size. For a large collection of documents, this can be a lot of data to store, and searching in a vector database full of embeddings is proportional both to the size of the database and to the number of dimensions each stored vector has.

But you can just halve the size of the embeddings (throw away 512 of the 1024 dimensions), and take up half the space while doubling search speeds. This has an impact on performance. Throwing away information reduces precision. But as the graph below shows, even getting rid of half of the embedding only reduces performance slightly:

As long your embeddings are at least 256 dimensions, the loss in precision should remain fairly small. Below that level, however, relevance and accuracy deteriorate quickly.

Truncating embeddings like this empowers users to manage their own trade-offs between accuracy and computing costs. It gives you the tools to get big efficiency gains and large cost savings out of your search AI.

Robust quantization

Quantization is another way of reducing the size of embeddings. Instead of throwing away part of each embedding, quantization reduces the precision of the numbers in the embedding. The jina-embeddings-v5-text models generate embeddings with 16-bit numbers, but we can round those numbers off, reducing their precision and the number of bits needed to store them. In the most extreme case, we can reduce each number to one bit (0 or 1), compressing jina-embeddings-v5-text’s default 1024 dimension embeddings from 2 kilobytes to 128 bytes, a 94% reduction from binary quantization alone. Just like for truncation, this produces large savings in memory and computing costs. However, also like truncation, quantization makes embeddings less accurate.

We’ve trained the jina-embeddings-v5-text models to work with Elasticsearch’s Better Binary Quantization by minimizing that loss of accuracy, and benchmark tests of binarized embeddings from these models show performance almost equal to their non-binarized equivalents. Consult the technical report for detailed ablation studies of binarization performance.

Multilingual performance

Many embedding models are multilingual because they’ve been trained on materials that include large numbers of languages. But that doesn’t mean that they all perform equally well in all supported languages.

We identified 211 languages in the MMTEB multilingual benchmark and separated them so we could compare our models to similar models on a language-by-language basis. The image below summarizes our results as a heat map. Each patch is a language (identified by its ISO-639 code), and the greener it is, the better the model performed compared to the average of similar models:

Although accuracy varies between languages, the jina-embeddings-v5-text models are state-of-the-art or nearly so across most of the world’s languages.

For details about multilingual performance, see the jina-embeddings-v5-text technical report.

Jina in Elastic: State-of-the-art native AI for search

With jina-embeddings-v5-text models on EIS, you can run high-performance multilingual embedding models natively in Elasticsearch with fully managed, GPU-accelerated inference and no infrastructure to provision or scale. jina-embeddings-v5-text models extend the growing EIS model catalog with compact, multilingual models powered by the latest developments in AI. These models have state-of-the-art performance on information retrieval and standard data analysis benchmarks, and they offer unequaled, globe-spanning multilingual support.

With two models of vastly different sizes, users can determine which one is best suited for their applications and budgets. Furthermore, with robust embeddings that remain performant when truncated to smaller sizes or quantized to lower precision, jina-embeddings-v5-text models provide opportunities for further concrete savings in storage and computing costs as well as in processing latency.

With the jina-embeddings-v5-text family, Jina Reranker, and Elastic’s fast vector and BM25 search, users now have access to end-to-end, state-of-the-art hybrid search from Elastic. When you need the most relevant results, whether for retrieval augmented generation (RAG) pipelines, search applications, or data analysis, Elastic with Jina search AI models provides solid and cost-effective quality.

Getting started

The jina-embeddings-v5-text models are fully integrated into EIS, and you can use them by setting the type field to semantic_text when creating your index and specifying the model (jina-embeddings-v5-text-small or jina-embeddings-v5-text-nano) in the inference_id field, as in this example:

PUT multilingual-semantic-index
{
  "mappings": {
    "properties": {
      "content": {
        "type": "semantic_text",
        "inference_id": ".jina-embeddings-v5-text-small"
      }
    }
  }
}

# Ingest data about France
POST multilingual-semantic-index/_doc
{
  "content": "The capital of France is Paris"}

GET multilingual-semantic-index/_search
{
  "query": {
    "semantic": {
      "field": "content",
      "query": "What is the French capital?"
    }
  }
}

Elasticsearch automatically selects the appropriate LoRA adapter during indexing and retrieval. The embedding dimensions (see the “Truncating embeddings” section, above) can be set when creating a custom inference endpoint.

See the Elasticsearch documentation for more information on using jina-embeddings-v5-text models.

More information

To learn more about jina-embeddings-v5-text models, read the release notes on the Jina AI blog and the technical report, with more detailed technical information about performance and Jina AI’s innovative new training procedure. For information about downloading and running these models locally, visit the jina-embeddings-v5-text collection page on Hugging Face.

Jina AI models are available under a CC-BY-NC-4.0 license, so you are free to download them and try them out, but for commercial use, please contact Elastic sales.

In this article, you’ll learn how you can use the minimum score parameter to increase the precision of your semantic search results. If you’d like to test the examples provided in this blog post, go to the associated Jupyter notebook.

Background: Precision and recall

In search relevance, precision and recall are key concepts. Any reader not already familiar is highly encouraged to read up on them. Following is a summary.

• Precision: The fraction of returned search results that are relevant to the user.
• Recall: The fraction of all relevant documents in the corpus that are included in the search result set.

Or, in other words, precision is returning only relevant results; and recall is returning all relevant results. As you can imagine, these are often competing requirements. Semantic search tends to have very high recall but can struggle with precision. Keep reading to learn how to get around this property.

Introducing the minimum score parameter

The ‘min_score’ parameter allows us to improve precision by setting a minimum score, which will truncate the result set by removing any matches with a score less than the defined threshold. Following is a simple example:

GET search-movies/_search
{
  "retriever": {
    "linear": {
      "min_score": 4,
      "retrievers": [
        ...
      ]
    }
  }
}

Normalizing the score

Setting a minimum score is all well and good; however, not all semantic models return a score suitable for a static threshold. ELSER, for example, returns a score that is unbounded. Some dense model scores are densely clustered and only make sense in the context of the specific query.

For most semantic search cases, we recommend using a normalization approach before applying the ‘min_score’. The normalization ensures that the document score is within a defined interval. Elasticsearch retrievers provide two such normalizers, ‘l2_norm’ and ‘minmax’. The most commonly used is ‘minmax’, since it’s easy to understand and works well in many scenarios. Key properties of ‘minmax’ include:

• Document scores are distributed between 0–1.
• The highest scoring document is always scored as 1.
• The lowest scoring document is always scored as 0.
  • This can make it less suitable for keyword search. See “Hybrid search” section for further discussion.

Following is an example of a normalized semantic query with min_score. Rank window size has been increased to 500 to allow us to return a longer list of search results, starting at 100.

GET search-movies/_search
{
  "size": 100,
  "_source": [
    "title", "overview"
  ],
  "retriever": {
    "linear": {
      "rank_window_size": 500,
      "min_score": 0.25,
      "retrievers": [
        {
          "normalizer": "minmax",
          "retriever": {
            "standard": {
              "query": {
                "semantic": {
                  "field": "overview_vector",
                  "query": "superhero movie"
                }
              }
            }
          }
        }
      ]
    }
  }
}

The size has been set to a higher value than normally seen in production. This is so we can inspect the quality of search results and tune the results.

Hybrid search using the linear retriever

For hybrid search, the simplest approach is to normalize all scores, assign weights, and apply a minimum score. Note that by choosing weights with a sum of 1, you keep the total score within a range of 0–1. This makes it easy to make sense of the final scores and tune min_score. Following is an example:

GET search-movies/_search
{
  "size": 100,
  "_source": ["title", "overview","keywords"],
  "retriever": {
    "linear": {
      "rank_window_size": 500,
      "min_score": 0.25,
      "retrievers": [
        {
          "weight": 0.6,
          "normalizer": "minmax",
          "retriever": {
            "standard": {
              "query": {
                "semantic": {
                  "field": "overview_vector",
                  "query": "superhero movie"
                }
              }
            }
          }
        },
        {
          "weight": 0.4,
          "normalizer": "minmax",
          "retriever": {
            "standard": {
              "query": {
                "multi_match": {
                  "query": "superhero movie",
                  "fields": ["overview","keywords", "title"],
                  "type": "cross_fields",
                  "minimum_should_match": "2"
                }
              }
            }
          }
        }
      ]
    }
  }
}

Hybrid search using RRF

With BM25, we often control precision through other means, such as using the AND operator or minimum_should_match. In addition, queries consisting of single, precise, and rare terms will naturally cause search results with few search results, often all being highly relevant. This can lead to:

• Results further back in the result get assigned a low normalized score in the BM25 retriever, even if the absolute BM25 score is close to top scoring hits.
• When adding a very low BM25 score to the semantic score, the total can be approximated as the semantic score.
• The lack of BM25 score contribution can cause the document to be discarded by the min_score threshold.

As a solution, we can instead use reciprocal rank fusion (RRF) to combine BM25 and semantic results. RRF gets around the challenge of comparing scores from different search algorithms by instead focusing on the position in each result set. In this scenario, the min_score is only applied to the semantic retriever.

GET search-movies/_search
{
  "_source": ["title", "overview","keywords"],
  "retriever": {
    "rrf": {
      "rank_window_size": 500,
      "retrievers": [
        {
          "linear": {
            "rank_window_size": 500,
            "min_score": 0.25,
            "retrievers": [
              {
                "normalizer": "minmax",
                "retriever": {
                  "standard": {
                    "query": {
                      "semantic": {
                        "field": "overview_vector",
                        "query": "superhero movie"
                      }
                    }
                  }
                }
              }
            ]
          }
        },
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "superhero movie",
                "fields": ["overview", "keywords","title"],
                "type": "cross_fields",
                "minimum_should_match": "2"
              }
            }
          }
        }
      ]
    }
  }
}

Conclusion

By using min_score, we’ve shown how we can reduce the number of false positives in our result sets caused by the high recall of semantic search algorithms. To learn more about retrievers, please see this blog post and the Elasticsearch documentation.

Dependency management at Elastic

At Elastic, we have to manage hundreds or even thousands of repositories, both private and public. When a critical CVE is discovered, we need immediate answers and actions: Which repositories are vulnerable? How quickly can we patch them? Apart from security, productivity questions also arise: How can we quickly propagate the release of a new package version across all the repositories that depend on it without spending too much time on manual tasks?

The initial trigger for searching ways of doing dependency management was the need to establish a secure foundation with automated updates for reducing CVEs. After carefully considering solutions on dependency management, we first started working on a self-hosted infrastructure. We were using our own Kubernetes cluster to run Mend Renovate Community Self-Hosted. The idea was to be able to provide a dependency management platform that our users could access in a self-service manner.

The initial experiment was successful, so more and more teams started onboarding our platform and using it in their everyday repositories’ lifecycle for updates and CVE patching. This happened so fast that we soon hit the ceiling of our self-hosted installation.

The challenge: How can we scale a dependency management platform in a large organization with a significant number of repositories?

Our dependency management platform was processing one repository at a time and the sequential processing model couldn’t keep up, due to the large number of repositories that we own. We had already identified that the issue resided within the concept that a single instance of our dependency management tool could process our big and ever-growing list of repositories. Repositories waited in a queue, sometimes for many hours. More than 50% of our repositories were not even processed daily. That means that more than 50% of our repositories waited more than 24 hours between scans.

Large repositories created larger bottlenecks, due to their sizable codebases and their multiple open PRs. GitHub webhook events disrupted the sequence. Automerge became unreliable because scan timing was unpredictable. We had made a promise to our users for the frequency of scans, and we couldn’t fulfill it.

The decision to build in-house: Meeting Elastic's unique scale and security needs

While we considered commercial options, including Mend's Renovate Self-Hosted Enterprise Self-Hosted edition, internally at Elastic we had a few key initiatives ramping up.

Our decision to build an in-house platform was driven by the recognition that only a deeply customized solution could meet Elastic's specific, nonnegotiable requirements:

1. Investing in our internal developer platform: At the time, we had already started heavily investing in our internal developer platform. We were discussing and designing ways that each one of our services could fit into that. This meant that we wanted to test-drive our own rules and practices for our dependency management platform. On top of that, new guidelines were coming into play and we wanted to design the platform ahead of events.
2. Native integration and workflow customization: We required straightforward integration with our internal tooling and internal processes. For example, we wanted to centralize configuration as code with our Service Catalog (Backstage). We have specific needs around the usage of Backstage that we wanted to make our platform compatible with. So, although it would be possible to make use of the Renovate Self-Hosted APIs alongside our Backstage automation, this wouldn’t cover completely for our internal processes.
3. Elastic-specific defense-in-depth security: Our stringent security compliance required bespoke security mechanisms tailored to our ecosystem. We were working to harden our usage of “non-human identities.” The way this hardening of access worked meant that the nonstandard means to authenticate to GitHub wouldn’t work with an off-the-shelf tool that didn’t support this internal implementation.Our workflow included implementing a parent-child workflow secret encryption pattern and using transient, single-use GitHub tokens. Building in-house was the only practical way to embed these unique security layers and minimize the attack surface across our complex multicloud environment.

The solution: Workflow orchestration for dependency management

Our solution started from the fact that we wanted to build on the dependency management tool that we already used and not replace it and look for other solutions. It had shown signs of its potential, and its flexibility is important for different needs throughout our organization. We considered different solutions, and what helped us make up our minds was the big and sometimes special needs that we have to cover for. We decided to build a reliable and scalable dependency management platform, where each repository will be processed on its own, removing bottlenecks and setting us up for growth.

We designed the platform abiding to three core principles:

1. Parallel processing

Every repository gets its own dependency management processing environment. No more queues. Our concurrency is only limited by the number of resources we spend. We have also applied smart distributed scheduling to avoid getting rate limited by GitHub.

2. Self-serviceable

We use our Service Catalog (Backstage) to automatically onboard and manage any new repository. We use our own resource definition to give the end user the option to select how often a repository will be processed, how many resources they want to allocate to their schedules, and if they want to turn processing off or back on for any reason. We plan to add more options that way as our users’ needs evolve and they get more fluent with the new installation.

3. Reduced secret scope and namespace isolation

For increased security, we supply our dependency management pods with ephemeral GitHub tokens that are being generated at the start of each workflow. On top of that, we isolate our workloads in specific namespaces so they can be provided only the necessary secrets. We control what secrets can be accessed by each dependency management workflow using Kubernetes RBAC. We also use encryption to propagate the GitHub token from the parent to the child workflows.

We rebuilt our platform using Kubernetes and harnessing the power of Kubernetes, Argo Workflows powers the logic of our processes, and Renovate CLI is set up for scanning and processing one repository at a time.

The beauty: We’re using battle-tested open source projects in an original way, providing new working examples for all of those projects and, at the same time, amplifying development velocity and consolidating CVE reduction for our teams.

Dependency management architecture: Four microservices

The platform comprises four custom-built components:

Workflows Operator (Go/Kubebuilder)

A Kubernetes operator managing workflow lifecycle through three Custom Resource Definitions (CRDs):

• RepoConfig CRD: Single source of truth for repository configuration.

This is how RepoConfig is defined in the operator:

// RepoConfig is the Schema for the repoconfigs API
type RepoConfig struct {
	metav1.TypeMeta `json:",inline"`

	// metadata is a standard object metadata
	// +optional
	metav1.ObjectMeta `json:"metadata,omitempty,omitzero"`

	// spec defines the desired state of RepoConfig
	// +required
	Spec RepoConfigSpec `json:"spec"`

	// status defines the observed state of RepoConfig
	// +optional
	Status RepoConfigStatus `json:"status,omitempty,omitzero"`
}

And this is what an instance of RepoConfig would look like:

apiVersion: workflows.elastic.co/v1
kind: RepoConfig
metadata:
  generation: 3
  name: elastic-test-repo
  namespace: dependency-management-operator
spec:
  owner: group:my-team
  renovate:
    config:
      resourceGroup: SMALL
      runFrequency: 4h
    enabled: true
  repository: elastic/test-repo

• Parent CRD: Manages CronWorkflows for scheduled scans.

Inside the reconciliation loop of the parent controller, we make sure that workflow settings are created and kept up to date or even deleted if needed.

First, it gets some globally configured settings for workflows:

func (r *ParentReconciler) reconcileSubResources(ctx context.Context, req ctrl.Request, parent *workflowsv1.Parent) error {
	logger := logf.FromContext(ctx)
	logger.Info("Reconcile SubResources for Parent", "name", req.NamespacedName)
	wfSet := workflowsettings.WorkflowSettings{
		RunFrequency:   parent.Spec.RunFrequency,
		ResourceGroups: "parent",
	}

It makes sure a mutex configmap is up to date to prevent similar workflows from running together:

	cfMngr := resources.NewConfigMapManager(r.Client, r.Scheme, r.OperatorConfig.ParentNamespace)
	err := cfMngr.CreateOrUpdateSyncMutexConfigmap(ctx, fmt.Sprintf("%s%s", r.OperatorConfig.ResourcesPrefix, r.OperatorConfig.SyncMutexCfgMapName), strings.TrimPrefix(parent.Spec.Repository, "elastic/"), r.OperatorConfig.SemaphoreConcurrencyLimit)

Then it creates a Workflow Manager that’s the struct which will create or update the CronWorkflows and the Workflow Templates:

	wfMngr := resources.NewArgoWorkflowManager(r.Client,
		r.Scheme,
		curateResourceName(
			strings.ReplaceAll(parent.Spec.Repository, "/", "-"),
		),
		parent.Namespace,
		"parent-workflow",
		false).
		WithOrganization(r.OperatorConfig.GitHubOrg).
		WithRepoName(parent.Spec.Repository).
		Init(true, true).
		WithPrefix(r.OperatorConfig.ResourcesPrefix).
		WithWfTemplateName(r.OperatorConfig.ParentWorkflowTemplate).
		WithResources(wfSet.GetResourceCategory()).
		WithSchedule(wfSet.GetCronSchedule()).
		WithImagePullSecrets([]corev1.LocalObjectReference{{
			Name: r.OperatorConfig.WorkflowImagePullSecrets,
		}}).
		AddArgument(true, true, "extra_cli_args").
		SetArgument(true, false, "extra_cli_args", "none").
		AddTemplate(resources.NewParentDAGTemplateInstance()).
		AddTemplate(resources.NewWorkflowsTemplateInstance("check-child-workflows", r.OperatorConfig.WorkflowImagePullPolicy, r.OperatorConfig.WorkflowNodeSelector)).
		AddTemplate(resources.NewWorkflowsTemplateInstance("security", r.OperatorConfig.WorkflowImagePullPolicy, r.OperatorConfig.WorkflowNodeSelector)).
		AddTemplate(resources.NewWorkflowsTemplateInstance("submit-child-workflow", r.OperatorConfig.WorkflowImagePullPolicy, r.OperatorConfig.WorkflowNodeSelector))
	wfMngr.OverWriteCommand("submit-child-workflow", r.OperatorConfig.ChildNamespace)
	wfMngr.OverwriteWfTemplateName("parent-wftmpl")
	wfMngr.AddSynchronization(fmt.Sprintf("%s%s", r.OperatorConfig.ResourcesPrefix, r.OperatorConfig.SyncMutexCfgMapName), "{{workflow.parameters.repo_name}}")
	err = wfMngr.CreateOrUpdateCronWorkflow(ctx)
	if err != nil {
		return fmt.Errorf("failed to create or update cron workflow: %w", err)
	}
	err = wfMngr.CreateOrUpdateWorkflowTemplate(ctx)
	if err != nil {
		return fmt.Errorf("failed to create or update workflow template: %w", err)
	}
	return nil

• Child CRD: Manages WorkflowTemplates with per-repository resources.

The child controller has a similar reconciliation duty to the parent, but this time it’s responsible for workflow templates in the child namespace that will be triggered by the parent workflows.

func (r *ChildReconciler) reconcileSubResources(ctx context.Context, req ctrl.Request, child *workflowsv1.Child) error {
	logger := logf.FromContext(ctx)
	logger.Info("Reconcile SubResources for Child", "name", req.NamespacedName)
	wfSet := workflowsettings.WorkflowSettings{
		ResourceGroups: child.Spec.ResourceCategory,
	}
	wfMngr := resources.NewArgoWorkflowManager(r.Client,
		r.Scheme,
		curateResourceName(
			strings.ReplaceAll(child.Spec.Repository, "/", "-"),
		),
		child.Namespace,
		"runner",
		true).
		Init(false, true). // only manage workflow template
		WithPrefix(r.OperatorConfig.ResourcesPrefix).
		WithSuffix("-child-wftmpl").
		WithRepoName(child.Spec.Repository).
		WithOrganization(r.OperatorConfig.GitHubOrg).
		WithResources(wfSet.GetResourceCategory()). // will override resources of presets if set
		WithImagePullSecrets([]corev1.LocalObjectReference{{
			Name: r.OperatorConfig.WorkflowImagePullSecrets,
		}}).
		AddTemplate(resources.NewWorkflowsTemplateInstance("runner", r.OperatorConfig.WorkflowImagePullPolicy, r.OperatorConfig.WorkflowNodeSelector)).
		AddArgument(false, true, "repo_full_name").
		AddArgument(false, true, "repo_name").
		AddArgument(false, true, "encrypted_token").
		AddArgument(false, true, "extra_cli_args")
	wfMngr.OverWriteCommand("runner", r.OperatorConfig.ChildNamespace)
	err := wfMngr.CreateOrUpdateWorkflowTemplate(ctx)
	if err != nil {
		return fmt.Errorf("failed to create or update workflow template: %w", err)
	}
	return nil
}

The multi-controller pattern provides clear separation: RepoConfig Controller handles onboarding/offboarding, Parent Controller manages scheduling, and Child Controller handles execution templates.

GitHub Events Gateway (Go)

A secure webhook proxy that receives GitHub webhooks, verifies signatures, filters by organization/repository, and routes to Argo Events. We built 10 distinct sensors responding to dependency dashboard interactions, PR events, and package updates.

This gateway enables integration with GitHub Apps by:

• Verifying incoming GitHub webhook signatures for security.
• Forwarding valid events to the Argo Events EventSource with all relevant headers and authentication.
• We also configure an authSecret on the EventSource and provide this as a Bearer header in forwarded requests.
• Providing logging, metrics, and retry logic.

It performs various validations on each GitHub Event request.

It makes sure some HTTP attributes are present:

// ValidateRequestMethod checks if the request method is POST.
func ValidateRequestMethod(r *http.Request) error {
	if r.Method != http.MethodPost {
		return fmt.Errorf("method not allowed, only POST is accepted")
	}
	return nil
}

// ValidateRequiredHeaders checks for required GitHub headers.
func ValidateRequiredHeaders(r *http.Request) error {
	eventType := r.Header.Get("X-GitHub-Event")
	deliveryID := r.Header.Get("X-GitHub-Delivery")
	signature := r.Header.Get("X-Hub-Signature-256")
	if eventType == "" || deliveryID == "" || signature == "" {
		return fmt.Errorf("missing required GitHub headers")
	}
	return nil
}

// ValidateUserAgent checks that the User-Agent header starts with GitHub-Hookshot/
func ValidateUserAgent(r *http.Request) error {
	userAgent := r.Header.Get("User-Agent")
	if !strings.HasPrefix(userAgent, "GitHub-Hookshot/") {
		return fmt.Errorf("invalid User-Agent")
	}
	return nil
}

While it also validates the signature of each request and its organizsation:.

// ValidateSignature verifies the GitHub webhook signature.
func ValidateSignature(r *http.Request, secret string) ([]byte, error) {
	payload, err := GitHub.ValidatePayload(r, []byte(secret))
	if err != nil {
		return nil, fmt.Errorf("invalid GitHub signature: %w", err)
	}
	return payload, nil
}

// ValidateAllowedOwner checks if the organization login is in the allowed organizations list.
func ValidateAllowedOwner(payload []byte, allowedGitHubOrganizations []string) (string, error) {
	var orgLogin string
	var payloadMap map[string]any
	if err := json.Unmarshal(payload, &payloadMap); err == nil {
		if orgObj, ok := payloadMap["organization"].(map[string]any); ok {
			if login, ok := orgObj["login"].(string); ok {
				orgLogin = login
			} else if name, ok := orgObj["name"].(string); ok {
				orgLogin = name
			}
		}
	}
	if !slices.Contains(allowedGitHubOrganizations, orgLogin) {
		return orgLogin, fmt.Errorf("organization login not allowed")
	}
	return orgLogin, nil
}

Finally, it routes to Argo Events based on event type:

	// Map eventType to Argo `EventSource` path
	var endpoint string
	switch eventType {
	case "push":
		endpoint = "/push"
	case "issues":
		endpoint = "/issues"
	case "pull_request":
		endpoint = "/pull-requests"
	default:
		slog.Info("Ignoring unhandled event type", "event_type", eventType, "delivery_id", deliveryID)
		w.WriteHeader(http.StatusOK)
		_, _ = w.Write([]byte("ok"))
		return
	}
	forwardURL := h.config.ArgoEventSourceForwardURL + endpoint

On the Argo Events side of things, 10 sensors watch the Argo Events EventBus for new events:.

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: {{ .Values.sensors.packageUpdateOnDefaultBranch.name }}
  namespace: {{ .Release.Namespace }}
spec:
  eventBusName: {{ .Values.eventBus.name }}

Then the script applies each sensor’s logic:

script: |
          local e = event
          if not e or not e.body or not e.body.repository then
            return false
          end

          -- e.g., "refs/heads/main"
          local ref = e.body.ref
          local default_branch = e.body.repository.default_branch
          if not ref or not default_branch then
            return false
          end

          local expected = "refs/heads/" .. default_branch
          if ref ~= expected then
            return false
          end

        {{- if .Values.sensors.packageUpdateOnDefaultBranch.packageFiles }}
          patterns = { {{- range $i, $f := .Values.sensors.packageUpdateOnDefaultBranch.packageFiles }}{{ if $i }}, {{ end }}"{{ $f }}"{{- end }} }
        {{- end }}

          local function anyMatch(path)
            if type(path) ~= "string" then return false end
            for _, pat in ipairs(patterns) do
              -- match filename at repo root, or anywhere under subdirs
              if path:match(pat) or path:match(".+/" .. pat) then
                return true
              end
            end
            return false
          end

          local function filesContainPackage(paths)
            if type(paths) ~= "table" then return false end
            for _, p in ipairs(paths) do
              if anyMatch(p) then return true end
            end
            return false
          end

          -- Inspect all commits (GitHub includes added/modified/removed lists)
          local commits = e.body.commits
          if type(commits) ~= "table" then
            -- Fallback: some payloads include only head_commit
            commits = {}
            if type(e.body.head_commit) == "table" then
              table.insert(commits, e.body.head_commit)
            end
          end

          for _, c in ipairs(commits) do
            if filesContainPackage(c.added) or filesContainPackage(c.modified) or filesContainPackage(c.removed) then
              return true
            end
          end

          return false

Backstage Syncer (Go)

This polls our Service Catalog (Backstage) for Repository Real Resource Entities, transforms them into RepoConfig CRDs, and keeps the platform in sync with configuration changes. Changes apply within three minutes.

repoMap := make(map[string]map[string]interface{})
			for i := range entities {
				entity := &entities[i]
				if entity.Spec.Type != "GitHub-repository" {
					continue
				}

				implRaw, err := json.Marshal(entity.Spec.Implementation)
				if err != nil {
					logger.Error("Failed to marshal implementation", "error", err)
					continue
				}

				var implMap map[string]interface{}
				err = json.Unmarshal(implRaw, &implMap)
				if err != nil {
					logger.Error("Failed to unmarshal implementation map", "error", err)
					continue
				}
				var repoName string
				if specMap, ok := implMap["spec"].(map[string]interface{}); ok {
					if repo, ok := specMap["repository"].(string); ok {
						repoName = repo
					}
				}
				if repoName == "" {
					continue
				}

				var workflowsRaw []byte
				if v, ok := implMap["spec"].(map[string]interface{}); ok {
					if r, ok := v["renovate"]; ok {
						workflowsRaw, _ = json.Marshal(r)
					} else {
						workflowsRaw = []byte(`{}`)
					}
				} else {
					workflowsRaw = []byte(`{}`)
				}

				var workflowsWithDefaults schema.WorkflowsMetadata
				err = json.Unmarshal(workflowsRaw, &rworkflowsWithDefaults)
				if err != nil {
					logger.Error("Failed to unmarshal workflows config", "error", err)
					continue
				}

				workflowsMap := map[string]interface{}{
					"enabled":        workflowsWithDefaults.Enabled,
					"require_pr":     workflowsWithDefaults.RequirePr,
					"resource_group": string(workflowsWithDefaults.ResourceGroup),
					"run_frequency":  string(workflowsWithDefaults.RunFrequency),
				}
				repoMap[repoName] = map[string]interface{}{
					"renovate": workflowsMap,
					"owner":    entity.Spec.Owner,
				}
			}
			logger.Info("Fetched GitHub Repository data from Backstage", "repository_count", len(repoMap), "status_code", resp.StatusCode)

Finally, it writes that data into RepoConfig instances.

Workflows base (Mixed: JavaScript, Go, Helm)

The foundation layer contains Helm charts, JavaScript configs, a Go wrapper for Renovate CLI with encryption support, and a custom APK Indexer for Alpine packages.

Self-service configuration

Teams configure their repositories declaratively through Backstage:

spec:
  renovate:
    enabled: true
    config:
      resourceGroup: LARGE      # SMALL | MEDIUM | LARGE  
      runFrequency: "0 */4 * * *"  # Every 4 hours

Resource groups allocate CPU and memory based on repository size:

• SMALL: 500m CPU, 1Gi memory.
• MEDIUM: 1000m CPU, 2Gi memory.
• LARGE: 2000m CPU, 4Gi memory.

Configuration is version-controlled, auditable, and applies automatically.

The parent-child pattern

The execution model uses a parent-child workflow pattern:

• Parent workflow: Lightweight CronWorkflow running on schedule. Encrypts secrets, determines whether a scan should run, passes configuration to the child.
• Child workflow: Ephemeral pod where Renovate CLI runs. Allocated resources dynamically, decrypts secrets in isolation, terminates after completion.

This separation provides security (secrets encrypted at parent level), resource optimization (parents use minimal resources), and scalability (children run in parallel).

The results

Performance transformation

• Before: One repository at a time, some repositories would not get processed possibly even for a day or more, less than 1,000 scans per day.
• After: 100+ concurrent scans, usually 8,000 scans and up to 10,000 recorded scans per day, limited only by the amount of resources we’re willing to spend and how we handle GitHub rate limits.

Cost efficiency

However weird it may sound, running 8,000 pods a day can get you the same result much cheaper than having one long-running pod trying to achieve the same results.

In the previous setup, we were running a single instance that, on a good day, would perform 500–600 scans. At the same time, due to the fact that different kinds of repositories would be executed on the same pod, we needed to size the pod for the biggest ones. That sizing would be much bigger than our current extra large offering, using 8 CPUs for the pod and 16G of memory.

To meet the current daily output, the single pod would need to run for 12 days. So comparing the cost of that single pod running for 12 days to 8,000 pods of our “MEDIUM” size running each day, our new design is far more efficient for the same output of scans:

However, let’s take into consideration that our default for our workloads is set to “SMALL,” with the great majority running successfully with 0.5 CPU and 1G RAM, and only a few need to change to medium, large. Let’s see what happens if 60% of our workloads are running on “SMALL,” 30% at “MEDIUM,” and 10% at “LARGE,” which is closer to the truth.

We can see that, for the same output, we’re far more cost-efficient in our current setup.

Enhanced security

• Ephemeral GitHub tokens (minutes of exposure versus days).
• Namespace isolation with Role-Based Access Control (RBAC) boundaries.
• Secret encryption at rest in parent workflows.
• Removed direct vault access.

Predictable performance

With guaranteed scan frequency, we can finally set Service Level Objectives (SLOs). Automerge works reliably. Teams trust the platform to deliver what’s promised.

Key architectural decisions

Here are some of the milestone design decisions that shaped how the platform looks.

• Why parent-child workflows?

We adopted this pattern to enforce a defense-in-depth strategy. By restricting high-value credentials (such as GitHub App secrets) to a dedicated, locked-down namespace, we use RBAC to ensure that ephemeral execution pods cannot arbitrarily access sensitive data. Recent supply chain vulnerabilities (for example, the "Shai Hulud" continuous integration/continuous delivery [CI/CD] attacks) have demonstrated the criticality of isolating runtime environments that execute dynamic scripts from the credential store.

Simultaneously, this decoupling enables granular resource optimization. The "parent" workflows act as lightweight orchestrators with a minimal footprint, while the "child" workflows handle the compute-intensive dependency scanning. This separation simplifies lifecycle management by allowing us to apply distinct reconciliation logic to each layer, granting users control over execution parameters (child) while retaining administrative control over the scheduling and security infrastructure (parent).

• Why self-serviceable?

Eliminating our team as a bottleneck for repository configuration was a critical requirement. Our mission was to architect a scalable, self-service platform capable of supporting diverse use cases. We recognized that acting as gatekeepers for every configuration change was unsustainable, given the sheer volume of repositories. Instead, we adopted a philosophy of enablement: providing the “rails” (infrastructure and guardrails) while empowering users to drive the “trains” (execution and customization). We believe this shift toward team autonomy significantly enhances productivity by allowing users to tailor the system to their specific operational needs.

• Why Kubernetes Operator pattern?

As mentioned above, a foundational design principle was to ensure that the platform was fully self-serviceable. We required an automated mechanism to capture user intent (such as toggling scans, adjusting scheduling frequency, or tuning runtime resource limits) and instantly propagate those changes to the underlying workflows. Anticipating future requirements, the system also needed to be easily extensible.

To achieve this, we developed a custom Dependency Management Kubernetes Operator. By using CRDs as the interface for configuration, we established a Kubernetes-native reconciliation loop. This operator continuously monitors the desired state defined by the user and automatically orchestrates the necessary updates to the workflow infrastructure. This ensures an event-driven, seamless operation, where the platform logic handles all complexity behind the scenes.

• Why design a GitHub Events Gateway?

Adopting an event-driven architecture (EDA) was essential for the platform's responsiveness. While CronWorkflows provided a reliable baseline schedule, we required the agility to handle ad hoc executions, such as users manually triggering scans via the dashboard. To achieve this, we needed a dedicated ingestion gateway to validate payload integrity and route requests intelligently.

We evaluated existing solutions, including the native GitHub EventSource for Argo, but we identified significant risks regarding operational overhead and strict GitHub API quotas (for example, webhook limits per repository). Consequently, we built a custom gateway to decouple our infrastructure from these limitations.

Crucially, this gateway served as a strategic traffic control point during our migration. It acted as a switch, enabling us to perform a gradual, granular rollout (traffic shifting) from the legacy system to the new infrastructure. This ensured that onboarding thousands of repositories was a controlled, risk-free process rather than a “big bang” switchover.

Lessons learned

Some lessons that we learned go hand-in-hand with the Elastic Source Code:

1. Customer First: Platforms are built for users. So it’s important to take users’ needs as priority number one. This shapes the platform into efficiently designed infrastructure and applications that reduce friction with users, simplify the scaling of the platform and ease adoption.
2. Space, Time: Sometimes the path of least resistance leads to shifting sands. We initially tried to optimize the existing sequential processing model, but this failed to resolve our issues; in fact, it only introduced more complexity and loose ends. The bold decision to rearchitect the platform with parallel processing required significant up-front effort. However, it ultimately paved the way for sustainable platform growth and virtually eliminated tedious daily administrative work.
3. IT, Depends: A platform cannot operate in isolation; its success depends on how well it integrates with the broader ecosystem. In our case, integration with Backstage was critical, as it serves as the source of truth for seamless service onboarding. Similarly, connecting to Artifactory allowed us to manage private package updates efficiently, and the list of essential integrations goes on.
4. Progress, SIMPLE Perfection: Throughout the implementation, we constantly pressure-tested our initial assumptions and adapted to new barriers as they emerged. Rather than getting paralyzed by perfectionism, we adopted an iterative approach, tackling challenges one by one and adjusting our migration strategy to meet real-world conditions.

What’s next

The delivery of the platform enables us for more meaningful work that will help us improve the UX and efficiency of our platform. Some examples are:

• Increase and guardrail the adoption of auto-merge

The auto-merge feature significantly accelerates team velocity by eliminating tedious manual tasks. However, we need to make sure that strict guardrails are in place to ensure that this increased speed does not come at the expense of security.

• Improve observability around end-user experience

A critical priority for our roadmap is enhancing observability, not just at the platform level but also specifically from the end-user’s perspective. While capturing infrastructure metrics is straightforward, understanding the actual user experience requires deeper insights. We’re working to define core user-centric key performance indicators (KPIs) so our telemetry can detect friction points and performance issues before they escalate into user complaints.

• Remove barriers for greater adoption

Looking ahead, our priority is to identify and remove any barriers hindering platform adoption. Whether this requires developing new integrations or deploying specific feature sets, we’re committed to data-driven planning. We’ve successfully built a platform designed for scale; our focus now shifts to maximizing its potential.

The bigger picture

The dependency management workflows project demonstrates a broader principle: When you need to scale open source tools beyond their default deployment model, Kubernetes-native patterns provide a path forward.

By embracing:

• CRDs for configuration.
• Operators for lifecycle management.
• Event-driven architecture for responsiveness
• GitOps for deployment.

We built orchestration that scales independently of the number of repositories it manages. The performance of scanning one repository is the same whether we’re managing 100 or 1,000.

When a critical CVE is announced, we now have answers in minutes, not hours. That’s the difference between a bottleneck and a competitive advantage.

Acknowledgments

This platform builds on excellent open source tools:

• Kubebuilder: The open source framework we used to kick-start our Kubernetes Operators that bootstrap and orchestrate our workflows. [1][2]
• Backstage: The open source framework on which we’ve built our Service Catalog and which we use as our source of truth. [1][2]
• Argo Workflows and Argo Events: The open source suite we used to orchestrate complex processes and add dynamic processing based on events. [1][2][3][4]
• Renovate CLI: The open source dependency management tool processing our repositories. [1][2]

* The AWS Fargate pricing model was used as a reference for of the cost of a single pod, although our workloads are not running necessarily on AWS and are running on full- blown Kubernetes clusters.

But if you work with languages like Hebrew, Arabic, German, or Polish, you know that standard rule-based analyzers often fail. They either under-analyze (missing relevant matches) or overanalyze (returning garbage results).

For years, we’ve had to rely on complex dictionaries and fragile regex rules. Today, we can do better. By replacing rule-based logic with neural models for text analysis (small, efficient language models that understand context), we can drastically improve search quality.

Here’s how to solve the morphology challenge by using the Elasticsearch inference API and a custom model service.

The problem: Why rules fail

Most standard analyzers are context-free. They look at one word at a time and apply a static set of rules.

• Algorithmic analyzers (like Snowball) strip suffixes based on patterns.
• Dictionary analyzers (like Hunspell) look up words in a list.

This approach breaks down when the structure of a word (its root and affixes) changes based on the sentence it lives in.

1. The semitic ambiguity (roots versus prefixes)

Semitic languages, like Hebrew and Arabic, are built on root systems and often attach prepositions (such as, in, to, or from) directly to the word. This creates ambiguous tokens that rule-based systems cannot solve.

• Word: בצל (B-Tz-L).
• Context A: “The soup tastes better with onion (batzal).”
• Context B: “We sat in the shadow (ba-tzel) of the tree.”

In Context A, בצל is a noun (onion). In Context B, it’s a preposition ב (in) attached to the noun צל (shadow).

A standard analyzer is forced to guess. If it aggressively strips the ב prefix, it turns "onion" into "shadow." If it’s conservative and leaves it alone, a user searching for "shadow" (tzel) will fail to find documents containing "in the shadow" (batzel). Neural models solve this by reading the sentence to determine whether the ב is part of the root or a separate preposition.

2. The compound problem (German, Dutch, and more)

Languages like German, Dutch, Swedish, and Finnish concatenate nouns without spaces to form new concepts. This results in a theoretically infinite vocabulary. To search effectively, you must split (decompound) these words.

• Word: Wachstube.
• Split A: Wach (guard) + Stube (room) = guardroom.
• Split B: Wachs (wax) + Tube (tube) = wax tube.

A dictionary-based decompounder acts blindly. If both “Wach” and “Wachs” are in its dictionary, it might pick the wrong split, polluting your index with irrelevant tokens.

To see this problem in English: A naive algorithm might split “carpet” into “car” + “pet.” Without understanding meaning, rules fail.

The solution: “Neural analyzers” (neural models for text analysis)

We don’t need to abandon the inverted index. We just need to feed it better tokens.

Instead of a regex rule, we use a neural model (like BERT or T5) to perform the analysis. Because these models are trained on massive datasets, they understand context. They look at the surrounding words to decide whether בצל means "onion" or "in shadow" or if Wachstube belongs in a military or cosmetic context.

Architecture: The inference sidecar

We can integrate these Python-based models directly into the Elasticsearch ingestion pipeline using the inference API.

The pattern:

1. External model service: A simple Python service (for example, FastAPI) hosts the model.
2. Elasticsearch inference API: Defines this service as a custom model within Elasticsearch.
3. Ingest pipeline: Sends text to the inference processor, which calls your Python service.
4. Index mapping: Create a whitespace target field for the analyzed text.
5. Indexing: The service returns the cleaned text, which Elasticsearch stores in the target field.
6. Search: Queries are analyzed via the inference API before matching.

Implementation guide

Let’s build this for Hebrew (using DictaBERT) and German (using CompoundPiece).

To follow along, you’ll need:

• Python 3.10+.
• Elasticsearch 8.9.x+.

Install the Python dependencies:

pip3 install fastapi uvicorn torch transformers

Step 1: External model service

To connect Elasticsearch to our neural model, we need a simple API service that:

1. Receives text from the Elasticsearch inference API.
2. Passes it through the neural model.
3. Returns analyzed text in a format Elasticsearch understands.

This service interfaces Elasticsearch with the neural model. At ingest time, the Elasticsearch pipeline calls this API to analyze and store document fields; at search time, the application calls it to process the user's query. You can deploy this on any infrastructure, including EC2, Lambda, or SageMaker.

The code below loads both models at startup and exposes /analyze/hebrew and /analyze/german endpoints:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List, Union
from transformers import AutoTokenizer, AutoModel, AutoModelForSeq2SeqLM
from contextlib import asynccontextmanager
import torch

# Global models (loaded once at startup)
he_model = None
he_tokenizer = None
de_model = None
de_tokenizer = None


@asynccontextmanager
async def lifespan(app: FastAPI):
   """Load models at startup."""
   global he_model, he_tokenizer, de_model, de_tokenizer

   print("Loading Hebrew model (DictaBERT-Lex)...")
   he_tokenizer = AutoTokenizer.from_pretrained("dicta-il/dictabert-lex")
   he_model = AutoModel.from_pretrained("dicta-il/dictabert-lex", trust_remote_code=True)
   he_model.eval()

   print("Loading German model (CompoundPiece)...")
   de_tokenizer = AutoTokenizer.from_pretrained("benjamin/compoundpiece")
   de_model = AutoModelForSeq2SeqLM.from_pretrained("benjamin/compoundpiece")

   if torch.cuda.is_available():
       he_model.to("cuda")
       de_model.to("cuda")

   print("Models loaded successfully!")
   yield
   print("Shutting down...")


app = FastAPI(
   title="Neural Text Analyzer",
   description="Multi-language text normalization service",
   version="1.0.0",
   lifespan=lifespan
)


class InferenceRequest(BaseModel):
   """ES Inference API sends: {"input": ["text1", "text2"]} or {"input": "text"}"""
   input: Union[str, List[str]]


def format_response(normalized_text: str) -> dict:
   """
   Normalize output to OpenAI-compatible format for ES Inference API.
   ES extracts: $.choices[*].message.content You do not need to stick
   with the OpenAI output format.
   Using it here for consistency reasons, since using the completions API.
   """
   return {
       "choices": [
           {"message": {"content": normalized_text}}
       ]
   }


@app.post("/analyze/hebrew")
async def analyze_hebrew(request: InferenceRequest):
   """Hebrew lemmatization using DictaBERT-Lex."""
   global he_model, he_tokenizer

   if he_model is None:
       raise HTTPException(status_code=503, detail="Model not loaded")

   # Handle input (can be string or list)
   if isinstance(request.input, str):
       texts = [request.input]
   else:
       texts = request.input

   # Run prediction
   with torch.no_grad():
       results = he_model.predict(texts, he_tokenizer)

   # results format: [[[word, lemma], [word, lemma], ...]]
   if results and results[0]:
       lemmas = []
       for word, lemma in results[0]:
           if lemma == '[BLANK]':
               lemma = word
           lemmas.append(lemma)
       normalized = " ".join(lemmas)
   else:
       normalized = ""

   return format_response(normalized)


@app.post("/analyze/german")
async def analyze_german(request: InferenceRequest):
   """German decompounding using CompoundPiece (supports 56 languages)."""
   global de_model, de_tokenizer

   if de_model is None:
       raise HTTPException(status_code=503, detail="Model not loaded")

   # Handle input
   if isinstance(request.input, str):
       text = request.input
   else:
       text = request.input[0] if request.input else ""

   # Format: "de: " for German
   input_text = f"de: {text}"

   inputs = de_tokenizer(input_text, return_tensors="pt")
   if torch.cuda.is_available():
       inputs = {k: v.to("cuda") for k, v in inputs.items()}

   with torch.no_grad():
       outputs = de_model.generate(**inputs, max_length=128)

   # IMPORTANT: decode outputs[0], not outputs
   result = de_tokenizer.decode(outputs[0], skip_special_tokens=True)

   # Clean up: "de: Donau-Dampf-Schiff" -> "Donau Dampf Schiff"
   # Note: model returns "de: " (with space after colon)
   if result.startswith("de: "):
       clean_result = result[4:].replace("-", " ")
   elif result.startswith("de:-"):
       clean_result = result[4:].replace("-", " ")
   elif result.startswith("de:"):
       clean_result = result[3:].replace("-", " ")
   else:
       clean_result = result.replace("-", " ")

   return format_response(clean_result.strip())


@app.get("/health")
async def health():
   return {"status": "healthy"}

Save the code above to a file (for example, analyzer_service.py), and run:

python3 -m uvicorn analyzer_service:app --port 8000

Wait for “Models loaded successfully!” (takes ~30–60 seconds for models to download on first run).

Test locally:

#Hebrew
curl -X POST http://localhost:8000/analyze/hebrew \
 -H "Content-Type: application/json" \
 -d '{"input": "הילדים אכלו גלידה בגינה"}'

#German
curl -X POST http://localhost:8000/analyze/german \
 -H "Content-Type: application/json" \
 -d '{"input": "Donaudampfschifffahrt"}'

Expected output:

- Hebrew: `{"choices":[{"message":{"content":"ילד אוכל גלידה גינה"}}]}`
- German: `{"choices":[{"message":{"content":"Donau Dampf Schiff Fahrt"}}]}`

Step 2: Configure Elasticsearch inference API

We’ll use the custom inference endpoint. This allows us to define exactly how Elasticsearch talks to our Python endpoint.

Note: Use response.json_parser to extract the content from our normalized JSON structure. You do not need to stick with the OpenAI output format. We’re using it here for consistency reasons, since we’re using the completion task type, which is text to text.

Exposing your local service

For testing, we’ll use ngrok to expose the local Python service to the internet. This allows any Elasticsearch deployment (self-managed, Elastic Cloud, or Elastic Cloud Serverless) to reach your service.

Install and run ngrok:

# Install ngrok (macOS) (Or download from https://ngrok.com/download)
brew install ngrok

Expose your local service:

ngrok http 8000

ngrok will display a forwarding URL like:

Forwarding https://abc123.ngrok.io -> http://localhost:8000

Copy the HTTPS URL. You’ll use this in the Elasticsearch configuration.

Configure the inference endpoint

 PUT _inference/completion/hebrew-analyzer                           
 {                                  
   "service": "custom",                                              
   "service_settings": {                             
     "url": "https://abc123.ngrok.io/analyze/hebrew",  
     "headers": {                    
       "Content-Type": "application/json"               
     },                                                
     "request": "{\"input\": ${input}}",                     
     "response": {                                
       "json_parser": {                         
         "completion_result": "$.choices[*].message.content"     
       }                               
     }                                 
   }                                   
 }

Replace https://abc123.ngrok.io with your actual ngrok URL.

Note: ngrok is used here for fast testing and development. The free tier has request limits, and URLs change on restart. For production, deploy your service to a persistent infrastructure.

For production (with API Gateway)

In production, deploy your Python service to a secure, persistent endpoint (such as AWS API Gateway + Lambda, EC2, ECS, or any cloud provider). Use secret_parameters to securely store API keys:

 PUT _inference/completion/hebrew-analyzer                        
 {                                     
   "service": "custom",                  
   "service_settings": {                
     "url": "https://your-api-gateway.execute-api.region.amazonaws.com/prod/analyze/hebrew",                 
     "headers": {                      
       "x-api-key": "${api_key}",       
       "Content-Type": "application/json"  
     },                              
     "secret_parameters": {           
       "api_key": "YOUR-API-KEY"     
     },                           
     "request": "{\"input\": ${input}}",      
     "response": {                    
       "json_parser": {               
         "completion_result": "$.choices[*].message.content"  
       }                             
     }                               
   }                                 
 }

Step 3: Ingest pipeline

Create a pipeline that passes the raw text field to our model and stores the result in a new field.

PUT _ingest/pipeline/hebrew_analysis_pipeline
{
 "description": "Lemmatizes Hebrew text using a custom inference endpoint",
 "processors": [
   {
     "inference": {
       "model_id": "hebrew-analyzer",
       "input_output": {
         "input_field": "content",
         "output_field": "content_analyzed"
       }
     }
   }
 ]
}

Step 4: Index mapping

This is the most critical step. The output from our neural model is already analyzed. We do not want a standard analyzer to mess it up again. We use the whitespace analyzer to simply tokenize the text we received.

PUT /my-hebrew-index
{
 "mappings": {
   "properties": {
     "content": {
       "type": "text",
       "analyzer": "standard"
     },
     "content_analyzed": {
       "type": "text",
       "analyzer": "whitespace"
     }
   }
 }
}

Step 5: Indexing

Option A: Single document.

POST /my-hebrew-index/_doc?pipeline=hebrew_analysis_pipeline
{
"content": "הילדים אכלו גלידה בגינה"
}

Option B: Reindex existing data.

If you have existing data in another index, reindex it through the pipeline:

POST _reindex
{
 "source": {
   "index": "my-old-index"
 },
 "dest": {
   "index": "my-hebrew-index",
   "pipeline": "hebrew_analysis_pipeline"
 }
}

Option C: Set pipeline as default for index.

Make all future documents automatically use the pipeline:

PUT /my-hebrew-index/_settings
{
"index.default_pipeline": "hebrew_analysis_pipeline"
}

Then index normally (no ?pipeline= needed):

POST /my-hebrew-index/_doc
{
"content": "הילדים אכלו גלידה בגינה"
}

Step 6: Search

Search using a neural analyzer in Elasticsearch is a two-step process, so analyze the query first using the inference API, and then search with the result:

A. Analyze the query.

 POST _inference/completion/hebrew-analyzer
 {
   "input": "הילדים אכלו גלידה בגינה"
 }

B. Search with the result.

 GET /my-hebrew-index/_search
 {
   "query": {
     "match": {
       "content_analyzed": "ילד אוכל גלידה גינה"
     }
   }
 }

In production, wrap these two calls in your application code for a seamless experience.

Available models

The architecture above works for any language. You simply swap the Python model and adjust the post-processing of the output. Here are verified models for common complex languages:

• Hebrew: Context-aware lemmatization. Handles prefix ambiguity (ב, ה, ל, and more) dicta-il/dictabert-lex.
• German: Generative decompounding. Supports 56 languages, including Dutch, Swedish, Finnish, and Turkish. benjamin/compoundpiece.
• Arabic: BERT-based disambiguation and lemmatization for Modern Standard Arabic. CAMeL Tools.
• Polish: Case-sensitive lemmatization for Polish inflections. amu-cai/polemma-large.

Conclusion

You don’t need to choose between the precision of lexical search and the intelligence of AI. By moving the “smart” part of the process into the analysis phase using the inference API, you fix the root cause of poor search relevance in complex languages.

The tools are here. The models are open-source. The pipelines are configurable. It’s time to teach our search engines to read.

Code

All code snippets from this article are available at https://github.com/noamschwartz/neural-text-analyzer.

References:

• https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-inference-put-custom
• https://www.elastic.co/docs/manage-data/ingest/transform-enrich/ingest-pipelines
• https://ngrok.com
• https://huggingface.co/dicta-il/dictabert-lex
• https://huggingface.co/benjamin/compoundpiece
• https://arxiv.org/pdf/2305.14214

Quick start

Download the relevant release or build and install (Linux build script generates Elasticsearch‑compatible zip):

./scripts/build_plugin_linux.sh

Install in Elasticsearch:

/path/to/elasticsearch/bin/elasticsearch-plugin install file:///path/to/heb-lemmas-embedded-plugin-.zip

Test:

curl -k -X POST "https://localhost:9200/_analyze" \
  -H "Content-Type: application/json" \
  -u "elastic:" \
  -d '{"tokenizer":"whitespace","filter":["heb_lemmas","heb_stopwords"],"text":"הילדים אוכלים את הבננות"}'

Why Hebrew search is different

Hebrew is morphologically rich: Prefixes, suffixes, inflection, and clitics all collapse into a single surface form. That makes naive tokenization insufficient. Without true lemmatization, search quality suffers; users miss relevant results due to simple variations in form. This project tackles that by embedding a Hebrew lemmatization model inside the analyzer itself, so every token passes through a neural model before indexing and querying.

Example

Users may search for the lemma “בית” (house), but documents might contain:

• בית (a house)
• בבית (in the house)
• לבית (to the house)
• בבתים (in houses)
• לבתים (to houses)

Without lemmatization, these become different surface tokens; lemmatization normalizes them toward the same lemma (בית), improving recall:

What this plugin does

Rather than relying on rule-based stemming, the analyzer runs a Hebrew lemmatization model as part of the Elasticsearch analysis chain and emits one normalized lemma per token. Because the model is neural, it can use local context within each analyzed segment to choose a lemma in ambiguous cases—while still producing stable tokens that work well for indexing and querying. The analyzer:

• Runs a Hebrew lemmatization model inside Elasticsearch.
• Produces better normalized tokens for Hebrew text.
• Supports stopwords and standard analyzer pipelines.

The result: Fast, reliable lemmatization

This analyzer is optimized for real‑world throughput:

• ONNX Runtime in‑process inference.
• INT8-quantized model for lower latency and memory footprint.
• Java Foreign Function Interface (FFI) for high‑performance native inference.

The result: fast, reliable lemmatization with predictable operational behavior.

To evaluate performance, we ran a benchmark in a Docker container (4 cores, 12 GB RAM) on 1 million large documents (5.7 GB of data) from the Hebrew Wikipedia dataset. You’ll find the results below:

Open source and Elastic‑ready

The plugin is fully open source and works on:

• Elastic open‑source distributions.
• Elastic Cloud.

You can build it yourself or download prebuilt releases and install it like any other plugin.

To upload the analyzer plugin to Elastic Cloud, navigate to the Extensions section within your Elastic Cloud console and proceed with the upload.

Credits

This project is a fork of the Korra ai Hebrew analysis plugin (MIT), which was implemented by Korra.ai with funding and guidance from the National NLP Program led by MAFAT and the Israel Innovation Authority.

This fork focuses on Elasticsearch 9.x compatibility and running lemmatization fully in-process via ONNX Runtime, using an INT8‑quantized model and bundled Hebrew stopwords. Lemmatization is powered by DictaBERT dicta-il/dictabert-lex (CC‑BY‑4.0).

Huge thanks to the Dicta team for making high-quality Hebrew natural language processing (NLP) models available to the community.

Links

• GitHub repo
• Releases

Storage formats in dense_vector fields

Prior to 9.3, dense_vector fields support vectors of single bits, 1-byte integers, and 4-byte floats. We store the original vectors on top of any quantization and/or hierarchical navigable small world (HNSW) graph used for indexing, and the original vectors make up the vast majority of the required disk space of the vector indices. If your vectors are floating point, then the only option versions of Elasticsearch prior to 9.3 provide is to store 4 bytes per vector value: That’s 4kB for a single 1024-dimensional vector.

There are other floating-point sizes available, of course: IEEE-754 specifies floating-point sizes of many different lengths, including the 4-byte float32 and 8-byte float64 used by Java float and double types. It also specifies a float16 format, which only uses 2 bytes per value. However, this only has a maximum value of 65,504, compared to the 3.4x1038 of 4-byte float32 values, and the conversion between the two involves several arithmetic operations.

As an alternative, many machine learning (ML) applications now use bfloat16, which is a modification of IEEE-754 float32 to only use 2 bytes. It does this by discarding the lowest 2 bytes of the fractional part of the value, leaving the sign and exponent unchanged. This effectively reduces the precision of the floating-point value without a corresponding reduction in range. The conversion from float32 to bfloat16 is a simple bitwise truncation on the float32 value, with a bit of jiggling to account for rounding.

bfloat16 in Elasticsearch 9.3

Elasticsearch 9.3 now supports storing vector element types as bfloat16. In memory, it will still process every vector value as a 4-byte float32, as Java does not have built-in support for bfloat16. As it writes vector data to disk, it will simply truncate and round each float32 value to a 2-byte bfloat16, and zero-expand each bfloat16 value back to float32 on reading the value into memory.

This effectively halves your vector index sizes, as it uses 2 bytes per value rather than 4 bytes. There may be a small performance cost during reading and writing data as Elasticsearch performs the necessary conversions, but this is often counterbalanced by a significant reduction in the I/O required, as the OS now has to read half as much data. And, for most datasets, there is a minimal effect on search recall.

As an example, this is the difference in sizes for bfloat16 on our dense_vector dataset:

So, if your input vectors are already at bfloat16 precision, then happy days! Elasticsearch accepts raw bfloat16 vectors as float values, and as Base64-encoded vectors. The vectors are persisted to disk with the same precision as your original source data, immediately halving your data storage requirements.

If your input vectors are at 4-byte precision, then you can also use bfloat16 format to halve your index data sizes. Elasticsearch will truncate and round each value to 2-byte precision, throwing away the least significant bits of the fraction. This means that the vector values you get back from Elasticsearch won’t be exactly the same as what you originally indexed, so don’t use bfloat16 if you need to maintain the full 4-byte precision of float32.

Starting in Elasticsearch 9.3, and on Elasticsearch Serverless, you can specify element_type: bfloat16 with all dense_vector index types on any newly created indices. If you wish to use bfloat16 with existing indices, you can reindex into an index with element_type: bfloat16 and Elasticsearch will automatically convert your existing float vectors to bfloat16.

At a recent DevFest, we demonstrated how to bridge the gap between natural language requests and structured IT workflows. By combining Elastic Agent Builder with Elastic Workflows, we can create AI assistants that not only answer questions but also perform complex actions.

In this post, we’ll dive into the architecture from that talk, specifically looking at how we built an automated "Laptop Refresh" workflow. We’ll demonstrate how to configure an agent that collects user requirements and triggers a server-side automation to interact directly with ServiceNow APIs.

Watch the full breakdown: This post is based on our presentation at Google DevFest. You can watch the full session here to see the demo in action.

The architecture: From chat to fulfillment

Note: The technical implementation described in this document is a streamlined version of the full production environment. While the architecture diagram provided serves as an accurate structural reference for the actual deployment, the accompanying text and code snippets have been simplified for illustrative purposes and may differ from the final, complex configurations used in the live implementation.

The goal is to move from a manual, form-heavy process to a conversational interface. Instead of a user navigating a catalog, they simply tell the AI assistant that they’re due for a laptop upgrade.

As illustrated above, the flow consists of three distinct layers:

1. Interaction layer (ElasticGPT/Agent Builder): The user interacts naturally with an interface powered by ElasticGPT. Behind the scenes, Agent Builder processes this conversation, handling intent detection and slot filling, to structure the data and orchestrate interactions with other internal systems.

• Intent detection
  • Mechanism: System prompt instruction.
  • Implementation: The agent is explicitly told its single purpose in the MISSION statement. It doesn’t need to "detect" other intents because it’s scoped strictly to IT provisioning.
    • Code reference: MISSION: You are a specialized agent designed to collect complete employee onboarding information...
  • Constraint: If a user asks about non-IT topics (for example, "What is the weather?"), the MISSION implies that the agent should pivot back to data collection or decline, depending on the large language model’s (LLM's) default safety alignment.
• Slot filling (data collection)
  • Mechanism: Phased conversation flow.
  • Implementation: Instead of asking for all slots at once, the DATA COLLECTION STRATEGY breaks the slots into five logical phases. This prevents the context switching fatigue mentioned above.
    • Code reference: PHASE 1: Personal information, PHASE 2: Employment Details, and so on.
  • Validation: The prompt enforces immediate validation (for example, Validate inputs immediately), acting as a gatekeeper before moving to the next slot.

2. Automation layer ( Workflows): Once the agent has the data, it triggers a workflow. This workflow handles the logic: checking device eligibility, enforcing policy (for example, "Is the laptop > 3 years old?"), and making API calls.

3. System of record (ServiceNow): The workflow reads and writes directly to the ITSM tool to maintain audit trails and initiate fulfillment.

Step 1: Configuring the agent

The first step is defining the "brain" of the operation using Agent Builder. We need an agent that acts strictly within the bounds of IT provisioning. We don't want a general chatbot; we want a data collection machine that feels like a helpful colleague.

We achieve this via a robust system prompt. The prompt dictates the agent's operating protocol, enforcing a step-by-step data collection strategy.

Here’s the refined structure of the prompt we used. Notice how it enforces validation and logically groups questions to avoid overwhelming the user:

MISSION: You are a specialized agent designed to collect complete employee onboarding information for IT equipment provisioning.

OPERATING PROTOCOL:
0. On every new chat, send a welcome message, and directly jump to data collection.

1. DATA COLLECTION STRATEGY:
   - Use a step-by-step approach across 5 clear phases
   - Validate inputs immediately

2. CONVERSATION FLOW:
   PHASE 1: Personal Information (Name, Email, Phone)
   PHASE 2: Employment Details (Job Title, Department, Manager)
   PHASE 3: Location & Shipping (Address, Country)
   PHASE 4: Technical Setup (Laptop Type, Accessories)
   PHASE 5: Confirmation

...

6. SUCCESS COMPLETION:
   After all data is collected and validated, invoke the tool "laptoprefreshworkflow" with the JSON payload.

For a sample system prompt or instructions, please refer here.

By explicitly instructing the agent to send the data in a specific JSON format at the end of the conversation, we ensure that the input matches exactly what our automation layer expects.

Step 2: The automation layer (Workflows)

The agent provides the intent and the data, but Workflows provides the muscle.

We define a workflow using a YAML configuration. This workflow acts as the bridge between the AI agent and the ServiceNow REST APIs. It handles authentication, data retrieval, and the ordering process.

Below is the workflow definition. We’ve refined the code to use secure variable handling for credentials rather than hardcoding them.

Workflow inputs

First, we define the inputs the workflow expects to receive from the agent:

YAML
version: "1"
name: Submit Laptop Refresh Request
enabled: true
triggers:
  - type: manual
inputs:
  - name: userid
    type: string
  - name: preferred-address
    type: string
  - name: laptop-choice
    default: Macbook latest
    type: string
  - name: laptop-keep-or-return
    default: return
    type: string

Interacting with ServiceNow

The workflow executes a series of HTTP steps. Crucially, we first need to identify the user's current asset to link the refresh request correctly.

1. Fetching computer data

We query the cmdb_ci_computer table in ServiceNow to find the asset currently assigned to the user.

YAML
steps:
  - name: snow_get_computer_data
    type: http
    with:
      url: https://elasticdev.service-now.com/api/now/table/ci_computer?assigned_to={{ inputs.userid }}
      method: GET
      headers:
        Accept: application/json
        Content-Type: application/json
        # Best Practice: Use secrets for authorization headers
        Authorization: Basic {{ secrets.servicenow_creds }}
      timeout: 30s

2. Adding to cart

Once we have the asset details and the user's preferences, we don't just create a generic ticket. We use the ServiceNow Service Catalog API to programmatically add the specific item to a cart.

YAML
  - name: snow_post_add_item_to_cart
    type: http
    with:
      url: https://elasticdev.service-now.com/example
      method: POST
      headers:
        Accept: application/json
        Content-Type: application/json
        Authorization: Basic {{ secrets.servicenow_creds }}
      body: |
        {
            "sysparm_quantity": 1,
            "variables": {
              "caller_id_common": "{{ inputs.userid }}",
              "current_device": "{{ steps.snow_get_asset.output.data.result.sys_id }}",
              "laptop_keep_or_return": "{{ inputs.laptop-keep-or-return }}",
              "choose_your_laptop": "{{ inputs.laptop-choice }}",
              "shipping_address": "{{ inputs.preferred-address }}"
            }
        }

3. Indexing the transaction

Finally, we want to keep a record of this transaction within Elasticsearch for analytics and future reference. We use the elasticsearch.index step to store the request details immediately after submission.

YAML

  - name: index-submission-record
    type: elasticsearch.index
    with:
      index: laptop-refresh-submission-data
      id: "{{ steps.snow_post_submit_order.output.data.result.request_id }}"
      document:
        request-id: "{{ steps.snow_post_submit_order.output.data.result.request_id }}"
        user-id: "{{ inputs.userid }}"
        configuration-item: "{{ steps.snow_get_computer_data.output.data.result[0].sys_id }}"
        laptop-choice: "{{ inputs.laptop-choice }}"
        timestamp: "{{ steps.snow_post_submit_order.output.data.result.sys_created_on }}"

For detailed workflow yaml, please refer here.

The result

By stitching these components together, we create a seamless experience:

1. The user chats naturally with the agent to provide details.
2. The agent structures this unstructured conversation into a JSON object.
3. Workflow receives the JSON, validates the user's current hardware via ServiceNow, creates the order, and indexes the result.

This approach reduces a process that traditionally took users 5–10 minutes of form navigation into a quick conversation, while ensuring that IT operations retains full visibility and control.

Video demo:

Ready to build?

This pattern, using an agent for the interface and using Workflows for the execution, can be applied to almost any ITSM task, from password resets to software provisioning.

If you’re interested in trying this out, be sure to watch the DevFest talk for the full context, and check out the Elastic AI Agent Builder documentation to get started building your own agents today.

When you see that headline, what do you think of? For a developer, it's a call to action, time to dive into new syntax, concurrency models, and bug fixes for the Swift programming language. For a music fan, it's a completely different story, a signal that Taylor Swift has just dropped a new album or is making a major announcement.

Your brain, in a fraction of a second, performs a remarkable feat of natural language processing (NLP). It doesn't just read the word "Swift" in isolation; it uses the surrounding context (the headline's source, your personal interests, and more) to resolve that single, ambiguous word to a unique, real-world entity.

In NLP, we call this ability to disambiguate named entity resolution (NER), and it's something humans do all the time. Natural language is inherently ambiguous, so we need to be able to map entities like "Bill Gates" to "the founder of Microsoft" and "The Eras Tour" to "Taylor Swift's concert tour". For humans, these connections come easily; for computers, not so much. Think how disappointed a Swiftie would be when they find out the article that their smart assistant recommended is actually about programming.

This same challenge becomes critical when you're monitoring news articles for mentions of specific people or organizations. Imagine you're tracking sanctioned entities or monitoring mentions of specific companies. You have a watch list with “Sakura Shipping Group” on it, and you want to know when articles mention the company. Simple enough, right? But what happens when an article refers to “Sakura Shipping” instead of the full legal name? Or uses an abbreviation like “SSG”? Or describes it indirectly as “a major Japanese maritime logistics firm”? Or mentions the company in Japanese, as “さくら海運グループ”? Your simple text matching won’t find these mentions, even though they all refer to the same organization. For compliance and risk monitoring use cases, missing a mention could have serious consequences. You need to catch every variation, every alias, every way an entity might be referred to.

This is the problem of entity resolution: identifying when different mentions in text refer to the same real-world entity and determining which entity that is. To solve this, we need a system that can handle semantic search (understanding meaning, not just keywords), named entity recognition (extracting entities from text), and fast, scalable matching across millions of documents. That's why we built this prototype on Elasticsearch. It provides built-in semantic search capabilities, integrated NER models, and the scalability needed for entity resolution.

In this series, we present an educational prototype for intelligent entity resolution that deliberately separates retrieval from judgment and explanation. Elasticsearch is used to efficiently narrow the search space by combining keyword, alias, and semantic (hybrid) search. Once plausible entity candidates are identified, a large language model (LLM) is used to determine whether a candidate refers to the same real-world entity, and the model’s rationale is provided in natural language.

This division of responsibilities avoids treating LLMs as black-box retrievers, preserves explainability for sensitive use cases, and demonstrates a reusable design pattern for building transparent, Elasticsearch-native systems. We examine why this pattern is particularly effective for entity resolution, where ambiguity is common and explainability matters. The goal is not to present a production-ready solution but to teach the architectural principles behind building transparent entity resolution systems.

Important note: This series presents an educational prototype that teaches Elasticsearch-native entity resolution using LLM judgments. We've made some simplifying choices (such as using Wikipedia for entity enrichment) to keep the system accessible for learning. Production systems might use different data sources, additional validation steps, or more sophisticated enrichment pipelines. The goal here is to demonstrate the core concepts and architecture, not to provide a production-ready system.

This series shows how we can help computers make these necessary connections while working with a 100% Elasticsearch-native architecture. We'll explore three major innovations:

• Enhancing entities with contextual information.
• Recognizing basic and complex entities with comprehensive NER.
• Providing transparent reasoning through Elasticsearch candidate matching and LLM-powered explanations.

We'll also evaluate the system and identify an important optimization that improves the overall performance of the educational prototype.

In this first post of a four-part series, we’ll focus on preparing both sides of the entity resolution equation: your watch list and the articles you want to search.

The problem: Why entity resolution requires preparation

Entity resolution is hard because we face challenges on both sides of the matching equation. On one side, entities can be mentioned in many different ways. A company might be referred to as "Microsoft", "Microsoft Corporation", "MSFT", or even "the Redmond-based tech giant", depending on the context and writing style. On the other hand, we need to find these mentions in articles, even when they're not obvious, such as when an article uses "the Russian President" or "F.D.R." instead of a full name.

Why we can't just match names directly: Without proper preparation, matching becomes unreliable. You might think, "But I can just search for 'Tim Cook' in the text, right?" Well, yes, if the article always mentions him by that exact name. But what about when it says "Apple CEO" instead? Or "Timothy D. Cook" (his full name)? Your simple text search won't find those mentions, even though they all refer to the same person.

Without entity preparation, we can't match "the Russian President" to "Vladimir Putin" because we don't know what "the Russian President" means without context. We can't match "J.R.R. Tolkien" to "John Ronald Reuel Tolkien" because we don't know that they're aliases for the same person. We can't match "Apple CEO" to "Tim Cook" because we can't understand the semantic relationship. Without indexing, finding matches means checking every entity in your watch list individually. This doesn't scale: With thousands of entities, every match becomes slow and expensive. For sanctioned individuals monitoring, this means missing critical mentions that use aliases or alternate spellings, a failure that could have serious consequences.

Why we can't just search text directly: Entity extraction is hard for the same reason entity resolution is hard: Entities can be mentioned in many different ways. The same person might be referred to as "J.R.R. Tolkien", "the author of The Lord of the Rings", or just "Tolkien", depending on the context. Without proper extraction, we can't find these mentions in the text. We'd have to manually identify every entity mention, which doesn't scale. We'd miss entities mentioned in nonstandard ways (for example, titles or abbreviations). We also wouldn't capture the context around entity mentions, which is crucial for accurate matching.

The solution is a two-phase system that prepares both your watch list and the articles you want to search.

The solution: Two-phase preparation system

To solve entity resolution, we need to prepare both sides of the matching equation. First, we enrich and index our watch list entities to enable semantic search. Second, we extract entity mentions from articles using hybrid techniques that capture explicit and implicit references. Together, these phases create the foundation for intelligent entity matching.

Phase 1: Preparing your watch list

The solution to preparing entities is to enrich them with meaningful contexts. This enables our entity matching system to work effectively. We'll explain how context helps in a bit, but let's walk through the prototype's simple implementation first.

Our watch list of entities may be provided in multiple formats. The Office of Foreign Assets Control (OFAC) provides sanctions lists that include first and last names, addresses, and identifying information, such as passport numbers, date and place of birth, and nationality information [1]. While this provides a good amount of context, in practice many of these fields are omitted when the values are unknown for the given entity. Some lists may be just a set of names. The most helpful lists for our purposes come out of the box with rich descriptions, as is often the case with commercial or curated datasets.

The three-component system used in the prototype starts by managing our entities and organizing their metadata. Since entity lists can vary in the amount of information they contain, our prototype is designed to work with whatever it receives. The JSON format supports entities with minimal information (just a name and type) or full information (with aliases, descriptions, metadata, and more). For example, an entity might be as simple as:

{
  "name": "J.R.R. Tolkien",
  "entity_type": "person"
}

Or it might include additional context:

{
  "name": "J.R.R. Tolkien",
  "entity_type": "person",
  "description": "English writer and philologist, author of The Lord of the Rings",
  "aliases": ["John Ronald Reuel Tolkien", "J.R.R. Tolkien", "Tolkien"],
  "priority": "medium"
}

The system handles both cases gracefully during enrichment. For the prototype, the enrichment process adds context from Wikipedia (specifically, the first paragraph of the entity's Wikipedia page) for entities that don't already have context [2]. This Wikipedia context helps with semantic matching, but it doesn't add other fields, like aliases or full names; those must come from the original dataset. (In production, you might use other approaches for enrichment, including an agentic system that figures out where to find the context information for a given entity. This is beyond the scope of our prototype, but it’s an exciting feature we could add in the future.) Finally, we index the entities in Elasticsearch with semantic search capabilities, creating a searchable index that understands meaning rather than just text.

Key concepts: Semantic search and indexing

What is semantic search? Semantics refers to the meanings of words and phrases. Figuring out meaning is usually easy for humans, but it's much more challenging for computers to "get" because it requires a depth of understanding that’s difficult to program. Semantic search works by turning this challenge into a math problem, something that computers are very good at [3].

Think of semantic search like map coordinates for meaning. Just as latitude and longitude tell you where something is on a map, semantic embeddings tell you where something is in "meaning space." Whereas traditional keyword search requires exact matches, semantic search relies on describing that "location" in a multidimensional vector space. For example, you might have the coordinates for a specific "big red building". When you search for a "small red building", semantic search looks in the "neighborhood" for similar concepts in the vector space. Your big red building might appear as a nearest neighbor, but the relevance score will be lower because parts of the meaning don't match.

Getting back to our example, when you search for "Apple CEO", semantic search can find "Tim Cook" because the semantic embeddings capture the meaning that both refer to the same person, even though they use completely different words. This capability is invaluable when monitoring for sanctioned individuals, as aliases and code names may be used to evade detection.

Why Elasticsearch for entity indexing? Elasticsearch has built-in semantic search capabilities using embedding models, like EmbEddings from bidirEctional Encoder rEpresentations (E5) [4]. This means we can create an index that understands meaning, not just text. When we index our enriched entities, Elasticsearch creates semantic embeddings that capture each entity's meaning, enabling intelligent matching later.

What is the mapping schema? The mapping schema defines how we structure entity data in Elasticsearch. Our schema includes several field types optimized for different search strategies, including:

• Keyword fields (id, name.keyword, aliases.keyword): For exact matching on entity names and aliases.
• Text fields (name, name_lower, context, aliases): For traditional, case-normalized full-text search with BM25 scoring.
• Semantic text fields (name_semantic, context_semantic): For vector-based similarity search using the multilingual-e5-small model.

This hybrid mapping enables multiple search strategies: exact matching for precise names, keyword search for aliases, and semantic search for meaning-based matching. Even better, Elasticsearch supports hybrid search, allowing us to use both keyword and semantic search simultaneously.

Before and after entity preparation

Before entity preparation, you have a simple list without much context, possibly nothing more than a name: "J.R.R. Tolkien". That's it. You can only match exact text matches, which means you'll miss "John Ronald Reuel Tolkien", "Tolkien", and any other variations. For sanctioned individuals, this means missing critical mentions that use aliases or alternate spellings.

After entity preparation, you have a rich, searchable index. "Vladimir Putin" is now enriched with Wikipedia context, and if your original dataset included aliases, like "Путин" or "Vladimir Vladimirovich Putin", those are indexed as well. The entity also has semantic embeddings that capture its meaning. The Wikipedia context helps semantic search understand that "The Russian President" refers to Vladimir Putin, enabling that match. If "Путин" was provided as an alias in your original dataset, exact matching handles that. Semantic variations work because your semantic embeddings understand meaning. For sanctioned individuals, this comprehensive preparation ensures you catch every mention, regardless of how the name is written or what alternative name is used.

Phase 2: Extracting entities from articles

Now that we have a searchable watch list, we need to extract entity mentions from articles. This is where article processing comes in.

Remember: This is an educational prototype designed to teach entity extraction concepts. Production systems might use different NER models, custom extraction rules, or specialized extraction pipelines tailored to specific domains or languages.

We extract entities from articles using a hybrid NER approach that combines machine learning with pattern-based extraction. First, we process articles to prepare them for extraction. Then, we extract entities using a hybrid extraction approach that combines NER performed in Elasticsearch (using a deployed XLM-RoBERTa model) with pattern-based extraction to catch entities that NER might miss.

This hybrid extraction approach provides several benefits. NER automatically finds entity mentions in text, even when they're not obvious. Pattern-based extraction catches entities that NER might miss, like titles and compound entities. We preserve the context around each entity mention, which helps with matching decisions later. The approach scales well, allowing us to process thousands of articles automatically, not just a few manually.

Key concepts: NER, pattern-based extraction, and hybrid extraction approach

What is NER? Named entity recognition is a machine learning technique that identifies named entities in text. When we run NER on an article, it finds mentions like "Microsoft", "Seattle", and "Washington" and labels them as organization, location, or person entities.

Why use NER in Elasticsearch? Using NER in Elasticsearch maintains our 100% Elasticsearch-native architecture, which simplifies the entity resolution prototype's design. Instead of managing separate services for entity extraction and search, everything runs in one system. You can perform NER during document ingestion using inference pipelines, and the extracted entities are immediately available for indexing and searching. This unified approach reduces complexity, eliminates network calls between services, and makes deployment and management easier. The XLM-RoBERTa model is trained to recognize entities in multiple languages, so we can extract entities from articles in different languages without needing separate models for each language. For information on deploying NER models in Elasticsearch, see the Elasticsearch NER documentation.

What is pattern-based extraction? Pattern-based extraction uses rules and patterns to find entities that NER might miss. For example, NER might not recognize "the author of The Lord of the Rings" as an entity mention, but pattern-based extraction can catch titles and roles like "the CEO" or "the President". However, pattern-based extraction is language-specific. The patterns need to be defined for each language you want to support. This is a significant drawback for multilingual systems, but it's acceptable for our educational prototype, which focuses on demonstrating the core concepts. Production systems might use language-specific pattern sets or alternative approaches for multilingual support.

How do they work together? The hybrid extraction approach combines both techniques. NER finds obvious entity mentions like "J.R.R. Tolkien", while pattern-based extraction catches variations that NER might miss, such as "the author of The Lord of the Rings". Together, they provide comprehensive coverage of entity mentions in text.

When we extract entities from an article mentioning "the author of The Lord of the Rings", we get:

• Text: "author of The Lord of the Rings"
• Type: PERSON (from pattern-based extraction)
• Confidence: 0.85
• Context: "The author of The Lord of the Rings published a new edition"

Before and after entity extraction

With NER-only extraction, we might find "J.R.R. Tolkien" and "The Lord of the Rings" in the article, but we'd miss "the author of The Lord of the Rings" because NER doesn't recognize descriptive phrases as entity mentions.

With hybrid extraction, we find both "J.R.R. Tolkien" (from NER) and "the author of The Lord of the Rings" (from pattern-based extraction). This comprehensive coverage enables better matching later, since we can match both the name and the descriptive phrase to our watch list.

What's next: Matching entities to our watch list

Now that we've prepared both sides of the entity resolution equation, we have everything we need for intelligent matching:

• A searchable watch list enriched with context and indexed for semantic search.
• Extracted entity mentions from articles using hybrid NER.

Preparation gives us the raw ingredients, but it doesn’t tell us which entity a mention actually refers to. In the next post, we'll explore how to match these extracted entities to our watch list using semantic search and LLM-powered judgment that handles ambiguity and context transparently.

Try it yourself

Want to see the preparation process in action? Check out these notebooks for complete walkthroughs with real implementations, detailed explanations, and hands-on examples:

• Entity preparation notebook: Shows you exactly how to enrich entities with Wikipedia context, create semantic search indexes, and prepare your watch list for intelligent matching.
• Article processing notebook: Shows you exactly how to extract entities from articles using hybrid NER, handle multilingual content, and process compound entities.

Remember: This is an educational prototype designed to teach the concepts. When building production systems, consider additional factors, like data source reliability, validation pipelines, error handling, monitoring, compliance requirements, domain-specific NER models, custom extraction rules, and quality validation that aren't covered in this learning-focused prototype.

References

1. OFAC Sanctions List Search
2. The datasets used for the prototype also use a special field, 'explicit_context', in lieu of getting the context from Wikipedia. We do this to control for the entity preparation step when we're testing other components such as entity matching.
3. The big ideas behind retrieval augmented generation
4. E5 in Elasticsearch

We’re also planning to support hybrid search in the community-driven Java integration very soon.

What is hybrid search?

Hybrid search is an information retrieval approach that combines keyword-based full-text search (lexical matching) with semantic search (vector similarity). Practically, it means a query can match documents because they contain the right terms and/or because they express the right meaning (even if the wording differs).In simple terms, you can think of it like this:

• Lexical retrieval: “Do these documents contain the words I typed (or related words)?”
• Semantic retrieval: “Do these documents mean something similar to what I typed?”

These two retrieval methods produce scores on different scales, so hybrid search systems typically use a fusion strategy to merge them into one ranking, for example, using reciprocal rank fusion (RRF).

In the figure above, we show an example: BM25 (keyword search) returns Docs A, B, and C, while semantic search returns Docs X, A, and B. The RRF algorithm then combines these two result lists into the final ranking: Doc A, Doc B, Doc X, and Doc C. With hybrid search, Doc C is included in the results thanks to BM25.

Why hybrid search matters

If you’ve built search or retrieval-augmented generation (RAG) features in production, you’ve probably seen the same failure modes show up again and again:

• Keyword search can be too literal. If the user doesn’t use the exact terms that appear in your documents, relevant content gets buried or missed.
• Semantic search can be too fuzzy. It’s great at meaning, but it can also return results that feel related while missing a critical constraint, like a product name, an error code, or a specific phrase the user actually typed.

Hybrid search exists because real user queries in production environments usually need both.

Next we’ll dive into how you get started with hybrid search in the LangChain integration for Python and JavaScript. If you want to read more about hybrid search, check out What is hybrid search? and When hybrid search truly shines.

Setting up a local Elasticsearch instance

Before running the examples, you'll need Elasticsearch running locally. The easiest way is using the start-local script:

curl -fsSL https://elastic.co/start-local | sh

After starting, you'll have:

• Elasticsearch at http://localhost:9200.
• Kibana at http://localhost:5601.

Your API key is stored in the .env file (under the elastic-start-local folder) as ES_LOCAL_API_KEY.

Getting started with hybrid search in LangChain (Python and JavaScript)

The dataset is a CSV with information on 1,000 science fiction movies, taken from an IMDb dataset on Kaggle. This demo uses a subset of the data, which has been cleaned. You can download the dataset used for this article from our GitHub gist, along with the full code for this demo.

Step 1: Install what you need.

First you’ll need the LangChain Elasticsearch integration and Ollama for embeddings. (You can also use some other embedding model if you wish.)

In Python:

pip install langchain-elasticsearch langchain-ollama

In JavaScript:

npm install @langchain/community @langchain/ollama @elastic/elasticsearch csv-parse

Step 2: Configure your connection and dataset path.

In Python:

At the top of the script, we set:

• Where Elasticsearch is (ES_LOCAL_URL).
• How to authenticate (ES_LOCAL_API_KEY).
• Which demo index name to use (INDEX_NAME).
• Which CSV file we’ll ingest (scifi_1000.csv).

ES_URL = os.getenv("ES_LOCAL_URL", "http://localhost:9200") 
ES_API_KEY = os.getenv("ES_LOCAL_API_KEY")
INDEX_NAME = "scifi-movies-hybrid-demo" 
CSV_PATH = Path(__file__).with_name("scifi_1000.csv")

In JavaScript:

Notes for JavaScript:

• JavaScript uses process.env instead of os.getenv.
• Path resolution requires fileURLToPath and dirname for Elasticsearch modules.
• The class is called ElasticVectorSearch (not ElasticsearchStore as in Python).

import { Client } from "@elastic/elasticsearch";
import { OllamaEmbeddings } from "@langchain/ollama";
import {
  ElasticVectorSearch,
  HybridRetrievalStrategy,
} from "@langchain/community/vectorstores/elasticsearch";
import { parse } from "csv-parse/sync";
import { readFileSync } from "fs";
import { dirname, join } from "path";
import { fileURLToPath } from "url";

const __dirname = dirname(fileURLToPath(import.meta.url));

const ES_URL = process.env.ES_LOCAL_URL || "http://localhost:9200";
const ES_API_KEY = process.env.ES_LOCAL_API_KEY;
const INDEX_NAME = "scifi-movies-hybrid-demo";
const CSV_PATH = join(__dirname, "scifi_1000.csv");

We can now also create the client.

In Python:

es = Elasticsearch(ES_URL, api_key=ES_LOCAL_API_KEY)

In JavaScript:

const client = new Client({
  node: ES_URL,
  auth: ES_API_KEY ? { apiKey: ES_LOCAL_API_KEY } : undefined,
});

Step 3: Ingest the dataset, and then compare vector-only vs. hybrid.

Step 3a: Read the CSV and build what we index.

We build three lists:

• texts: The actual text that will be embedded + searched.
• metadata: Structured fields stored alongside the document.
• ids: Stable IDs (so Elasticsearch can dedupe if needed).

In Python:

# --- Ingest dataset ---
texts: list[str] = []
metadatas: list[dict] = []
ids: list[str] = []

with CSV_PATH.open(newline="", encoding="utf-8") as f:
    for row in csv.DictReader(f):
        movie_id = (row.get("movie_id") or "").strip()
        movie_name = (row.get("movie_name") or "").strip()
        year = (row.get("year") or "").strip()
        genre = (row.get("genre") or "").strip()
        description = (row.get("description") or "").strip()
        director = (row.get("director") or "").strip()

        # This text is both:
        #  - embedded (vector search)
        #  - keyword-matched (BM25 in hybrid mode)
        text = "\n".join(
            [
                f"{movie_name} ({year})" if year else movie_name,
                f"Director: {director}" if director else "Director: (unknown)",
                f"Genres: {genre}" if genre else "Genres: (unknown)",
                f"Description: {description}" if description else "Description: (missing)",
            ]
        )
        texts.append(text)
        metadatas.append(
            {
                "movie_id": movie_id or None,
                "movie_name": movie_name or None,
                "year": year or None,
                "genre": genre or None,
                "director": director or None,
            }
        )
        ids.append(movie_id or movie_name)

In JavaScript:

async function main() {
  // --- Ingest dataset ---
  const texts = [];
  const metadatas = [];
  const ids = [];

  const csvContent = readFileSync(CSV_PATH, "utf-8");
  const records = parse(csvContent, {
    columns: true,
    skip_empty_lines: true,
  });

  for (const row of records) {
    const movieId = (row.movie_id || "").trim();
    const movieName = (row.movie_name || "").trim();
    const year = (row.year || "").trim();
    const genre = (row.genre || "").trim();
    const description = (row.description || "").trim();
    const director = (row.director || "").trim();

    // This text is both:
    //  - embedded (vector search)
    //  - keyword-matched (BM25 in hybrid mode)
    const text = [
      year ? `${movieName} (${year})` : movieName,
      director ? `Director: ${director}` : "Director: (unknown)",
      genre ? `Genres: ${genre}` : "Genres: (unknown)",
      description ? `Description: ${description}` : "Description: (missing)",
    ].join("\n");

    texts.push(text);
    metadatas.push({
      movie_id: movieId || null,
      movie_name: movieName || null,
      year: year || null,
      genre: genre || null,
      director: director || null,
    });
    ids.push(movieId || movieName);
  }

What’s important here:

• We don’t embed only the description. We embed a combined text block (title/year + director + genre + description). That makes results easier to print and sometimes improves retrieval.
• The same text is what the lexical side uses, too (in hybrid mode), because it’s indexed as searchable text.

Step 3b: Add texts to Elasticsearch using LangChain.

This is the indexing step. Here we embed texts and write them to Elasticsearch.

For asynchronous applications, please use AsyncElasticsearchStore with the same API.

You can find our reference docs for both the sync and async versions of ElasticsearchStore, along with more parameters for advanced fine-tuning RRF.

In Python:

print(f"Ingesting {len(texts)} movies into '{INDEX_NAME}' from '{CSV_PATH.name}'...") 

vector_store = ElasticsearchStore(
    index_name=INDEX_NAME,
    embedding=OllamaEmbeddings(model="llama3"),
    es_url=ES_LOCAL_URL,
    es_api_key=ES_LOCAL_API_KEY,
    strategy=ElasticsearchStore.ApproxRetrievalStrategy(hybrid=False),
)

#This is the indexing step. We embed the texts and add them to Elasticsearch
vectore_store.add_texts(texts=texts, metadatas=metadatas, ids=ids)

In JavaScript:

  console.log(
    `Ingesting ${texts.length} movies into '${INDEX_NAME}' from 'scifi_1000.csv'...`
  );

  const embeddings = new OllamaEmbeddings({ model: "llama3" });

  // Vector-only store (no hybrid)
  const vectorStore = new ElasticVectorSearch(embeddings, {
    client,
    indexName: INDEX_NAME,
  });

  // This is the indexing step. We embed the texts and add them to Elasticsearch
  await vectorStore.addDocuments(
    texts.map((text, i) => ({
      pageContent: text,
      metadata: metadatas[i],
    })),
    { ids }
  );

Step 3c: Create another store for hybrid search.

We create another ElasticsearchStore object pointing at the same index but with different retrieval behavior: hybrid=False is vector-only search and hybrid=True is hybrid search (BM25 + kNN, fused with RRF).

In Python:

# Since we are using the same INDEX_NAME we can avoid adding texts again 
# This ElasticsearchStore will be used for hybrid search

hybrid_store = ElasticsearchStore(
    index_name=INDEX_NAME,
    embedding=OllamaEmbeddings(model="llama3"),
    es_url=ES_LOCAL_URL,
    es_api_key=ES_LOCAL_API_KEY,
    strategy=ElasticsearchStore.ApproxRetrievalStrategy(hybrid=True),
)

In JavaScript:

  // Since we are using the same INDEX_NAME we can avoid adding texts again
  // This ElasticVectorSearch will be used for hybrid search
  const hybridStore = new ElasticVectorSearch(embeddings, {
    client,
    indexName: INDEX_NAME,
    strategy: new HybridRetrievalStrategy(),
  });

  // With custom RRF parameters
  const hybridStoreCustom = new ElasticVectorSearch(embeddings, {
    client,
    indexName: INDEX_NAME,
    strategy: new HybridRetrievalStrategy({
      rankWindowSize: 100,  // default: 100
      rankConstant: 60,     // default: 60
      textField: "text",    // default: "text"
    }),
  });

Step 3d: Run the same query both ways, and print results.

As an example, let’s run the query “Find movies where the main character is stuck in a time loop and reliving the same day." and compare the results from hybrid search and vector search.

In Python:

query = "Find movies where the main character is stuck in a time loop and reliving the same day."
k = 5

print(f"\n=== Query: {query} ===")

vec_docs = vector_store.similarity_search(query, k=k)
hyb_docs = hybrid_store.similarity_search(query, k=k)

print("\nVector search (kNN) top results:")
for i, doc in enumerate(vec_docs, start=1):
    print(f"{i}. {(doc.page_content or '').splitlines()[0]}")

print("\nHybrid search (BM25 + kNN + RRF) top results:")
for i, doc in enumerate(hyb_docs, start=1):
    print(f"{i}. {(doc.page_content or '').splitlines()[0]}")

In JavaScript:

  const query =
    "Find movies where the main character is stuck in a time loop and reliving the same day.";
  const k = 5;

  console.log(`\n=== Query: ${query} ===`);

  const vecDocs = await vectorStore.similaritySearch(query, k);
  const hybDocs = await hybridStore.similaritySearch(query, k);

  console.log("\nVector search (kNN) top results:");
  vecDocs.forEach((doc, i) => {
    console.log(`${i + 1}. ${(doc.pageContent || "").split("\n")[0]}`);
  });

  console.log("\nHybrid search (BM25 + kNN + RRF) top results:");
  hybDocs.forEach((doc, i) => {
    console.log(`${i + 1}. ${(doc.pageContent || "").split("\n")[0]}`);
  });
}

main().catch(console.error);

Example output

Ingesting 1000 movies into 'scifi-movies-hybrid-demo' from 'scifi_1000.csv'...

=== Query: Find movies where main character is stuck in a time loop and reliving the same day. ===

Vector search (kNN) top results:
1. The Witch: Part 1 - The Subversion (20  18)
2. Divinity (2023)
3. The Maze Runner (2014)
4. Spider-Man (2002)
5. Spider-Man: Into the Spider-Verse (2018)

Hybrid search (BM25 + kNN + RRF) top results:
1. Edge of Tomorrow (2014)
2. The Witch: Part 1 - The Subversion (2018)
3. Boss Level (2020)
4. Divinity (2023)
5. The Maze Runner (2014)

Why these results?

This query (“time loop / reliving the same day”) is a great case where hybrid search tends to shine because the dataset contains literal phrases that BM25 can match and vectors can still capture meaning.

• Vector-only (kNN) embeds the query and tries to find semantically similar plots. Using a broad sci‑fi dataset, this can drift into “trapped / altered reality / memory loss / high-stakes sci‑fi” even when there’s no time-loop concept. That’s why results like “The Witch: Part 1 – The Subversion” (amnesia) and “The Maze Runner” (trapped/escape) can appear.
• Hybrid (BM25 + kNN + RRF) rewards documents that match both keywords and meaning. Movies whose descriptions explicitly mention “time loop” or “relive the same day” get a strong lexical boost, so titles like “Edge of Tomorrow” (relive the same day over and over again…) and “Boss Level” (trapped in a time loop that constantly repeats the day…) rise to the top.

Hybrid search doesn’t guarantee that every result is perfect. It balances lexical and semantic signals so you may still see some non-time-loop sci‑fi in the tail of the top‑k.

The main takeaway is that hybrid search helps anchor semantic retrieval with exact textual evidence when the dataset contains those keywords.

Full code example

You can find our full demo code in Python and JavaScript, as well as the dataset used, hosted on GitHub gist.

Conclusion

Hybrid search provides a pragmatic and powerful retrieval strategy by combining traditional BM25 keyword search with modern vector similarity into a single, unified ranking. Instead of choosing between lexical precision and semantic understanding, you get the best of both worlds, without adding significant complexity to your application.

In real-world datasets, this approach consistently yields results that feel more intuitively correct. Exact term matches help anchor results to the user’s explicit intent, while embeddings ensure robustness against paraphrasing, synonyms, and incomplete queries. This balance is especially valuable for noisy, heterogeneous, or user-generated content, where relying on only one retrieval method often falls short.

In this article, we demonstrated how to use hybrid search in LangChain through its Elasticsearch integrations, with complete examples in both Python and JavaScript. We’re also contributing to other open-source projects, such as LangChain4j, to extend hybrid search support with Elasticsearch.

We believe hybrid search will be a key capability for generative AI (GenAI) and agentic AI applications, and we plan to continue collaborating with libraries, frameworks, and programming languages across the ecosystem to make high-quality retrieval more accessible and robust.

The reality is precisely the opposite. Context engineering, the practice of managing what information reaches your LLM, is more critical than ever. Large context windows don’t eliminate the need for precision; they amplify it. With more context comes exponentially more opportunities for error, hallucinations, and irrelevant information to contaminate your LLM reasoning process.

Whether you’re using retrieval-augmented generation (RAG) retrieval, tool outputs, or memory systems, effective context engineering isn’t about providing more information but about providing the right information. That’s where Elasticsearch comes in, serving as your context engineering platform.

In this article, we’ll explore what context poisoning is, how it manifests across different types of memory, and how Elasticsearch RAG capabilities provide defense at every stage of the retrieval pipeline, from ingestion to composition, ensuring your LLM receives clean, relevant, and reliable context.

What is context poisoning?

Context poisoning occurs when compromised, outdated, or irrelevant information enters an LLM’s context window, leading to degraded responses, hallucinations, or perpetuated errors. Once corrupted or incorrect information enters the context window, it propagates into answers. The LLM references it as truth, creating cascading errors across the conversation.

This poisoning can happen at multiple stages of the LLM lifecycle (like in training), but our focus is on the retrieval and composition stages. Although adversarial attacks, like prompt injection, also pose risks, this article focuses on the operational patterns that teams encounter most frequently in production environments.

Operational understanding

Context poisoning often happens for reasons like:

• Context rot: Information becomes outdated but remains in your knowledge base without being updated or deleted.
• Context overflow: Too much information overwhelms the LLM's attention to the real important and relevant context, leading to missing relevant information from answers.
• Conflicting information: Multiple sources provide contradictory data, confusing the model.
• Semantic noise: Vectorial similar but contextually irrelevant content dilutes relevance.
• Malicious injection: Content deliberately inserted by attackers into knowledge bases, including prompt injections or manipulated data.

Understanding these patterns is the first step toward building robust defenses. Let’s examine each pattern and how Elasticsearch helps you address them. You can follow along with the supporting notebook.

Types of context poisoning

Temporal degradation

Over time, information in your knowledge base becomes outdated, and without proper management, stale content continues to be retrieved and presented to your LLM as current truth. This is especially problematic in industries where information changes frequently, like product documentation, pricing, regulations, or news.

Impact

Your LLM provides outdated advice, references to deprecated features, or contradictions to current reality, disengaging user trust.

Solutions: Temporal filtering in hybrid search

Elasticsearch’s date-based query capabilities ensure your RAG system prioritizes recent and relevant information through explicit temporal filters.

Example: Product documentation search with time filtering

A user asks your chatbot about authentication setup. Six months ago, the authentication had a significant change, so it’s important to only return documents from six months or earlier.

Without temporal filtering

POST product-docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "semantic": {
                "field": "content_semantic",
                "query": "how to configure OAuth authentication"
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "configure OAuth authentication",
                "fields": ["title^2", "content"]
              }
            }
          }
        }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  },
  "_source": ["title", "last_updated", "version", "content_snippet"]
}

Response without filtering: Contradictory results

The LLM receives three different methods for OAuth configuration: the current security API (9.x), legacy realm settings (7.x), and the deprecated shield plugin (6.x). This contradictory context leads to confused or misleading responses:

{
  "hits": {
    "total": { "value": 23 },
    "max_score": 24.5,
    "hits": [
      {
        "_id": "doc-oauth-2025",
        "_score": 24.5,
        "_source": {
          "title": "OAuth 2.0 Authentication Setup",
          "last_updated": "2025-10-15",
          "version": "9.x",
          "content_snippet": "To configure OAuth 2.0 authentication in Elasticsearch 9.x, use the new security API..."
        }
      },
      {
        "_id": "doc-oauth-2023",
        "_score": 23.8,
        "_source": {
          "title": "OAuth Authentication Configuration",
          "last_updated": "2023-04-20",
          "version": "7.x",
          "content_snippet": "Configure OAuth using the legacy realm settings in elasticsearch.yml..."
        }
      },
      {
        "_id": "doc-oauth-deprecated",
        "_score": 22.9,
        "_source": {
          "title": "Setting Up OAuth (Deprecated)",
          "last_updated": "2022-11-10",
          "version": "6.x",
          "content_snippet": "Use the shield plugin to configure OAuth authentication..."
        }
      }
    ]
  }
}

With temporal filtering

Add a filter to restrict results to documents updated within the last six months:

POST product-docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "semantic": {
                "field": "content_semantic",
                "query": "how to configure OAuth authentication"
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "configure OAuth authentication",
                "fields": ["title^2", "content"]
              }
            }
          }
        }
      ],
      "filter": [
        {"range": {"last_updated": {"gte": "now-6M"}}},
        {"term": {"status": "published"}}
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  },
  "_source": ["title", "last_updated", "version", "content_snippet"],
  "size": 5
}

This hybrid search query

• Semantic search (semantic) captures related concepts and context using the content_semantic field.
• Lexical search (multi_match) matches exact keywords like “OAuth” with field boosting title^2.
• Reciprocal rank fusion (RRF) combines both results sets with balanced reranking, retrieving the most relevant results.
• Temporal filter ensures only documents updated within the last six months are retrieved.
• Status filter restricts results to published documents, excluding drafts or deprecated content.

Response with temporal filtering: Consistent results

The temporal filtering eliminated outdated documents, leaving only current documentation for version 9.x. The LLM now receives consistent context and generates confident, accurate responses:

{
  "hits": {
    "hits": [
      {
        "_source": {
          "title": "OAuth 2.0 Authentication Setup",
          "last_updated": "2026-01-15",
          "version": "9.x",
          "content_snippet": "Configure OAuth 2.0 in Elasticsearch 9.x using the security API via Stack Management > Security."
        }
      },
      {
        "_source": {
          "title": "OAuth Provider Configuration",
          "last_updated": "2025-12-20",
          "version": "9.x",
          "content_snippet": "Configure Okta, Azure AD, Auth0 via security API with OIDC auto-discovery."
        }
      }
    ]
  }
}

Relative versus absolute time filters

Relative filtering (recommended for most use cases):

"filter": [
  {
    "range": {
      "last_updated": {
        "gte": "now-1y"
      }
    }
  }
]

Absolute filtering (for specific time ranges):

"filter": [
  {
    "range": {
      "last_updated": {
        "gte": "2025-01-01",
        "lte": "2025-12-31"
      }
    }
  }
]

Impact on LLM response quality

• Without filtering: LLM receives contradictory guidance from 2023–2025, producing uncertain responses mixing deprecated and current methods.
• With temporal filtering: LLM receives only recent documentation, generating confident responses based on current best practices.

Information conflicts

When your RAG system retrieves documentation for features that behave differently across deployment types, versions, or configurations, conflicting information can confuse the LLM about which guidance applies to the user’s specific context.

Impact

The LLM has to use more resources and tokens to understand and determine which information is correct, becoming more prone to errors and hallucinations.

Solutions: Hybrid search with metadata boosting

Elasticsearch’s bool query with a should clause allows you to boost values to prioritize documents matching specific metadata, ensuring deployment-specific or version-specific documentation appears first in the context window. For query syntax details, refer to Bool query reference.

Example: Deployment-specific feature documentation

A user asks, “How do I configure custom users in serverless?” Your knowledge base contains information about cloud, self-hosted, and managed deployments. With proper metadata prioritization, the LLM retrieves signals about feature availability and provides correct guidance:

POST platform-docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "bool": {
                "must": [
                  {
                    "multi_match": {
                      "query": "How do I configure custom users in serverless?",
                      "fields": ["title^2", "content"]
                    }
                  }
                ],
                "should": [
                  {"term": {"deployment_type": {"value": "serverless", "boost": 3.0}}},
                  {"term": {"doc_status": {"value": "current", "boost": 2.0}}}
                ]
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "semantic": {
                "field": "content_semantic",
                "query": "How do I configure custom users in serverless?"
              }
            }
          }
        }
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  },
  "_source": ["title", "deployment_type", "feature_supported", "content_snippet"],
  "size": 5
}

What this query does

• must clause: All documents must match “How do I configure custom users in serverless?”
• should clauses with explicit boosting:
  • Documents with deployment_type: “serverless” receive 3x boost.
  • Documents with doc_status: “current” receive 2x boost.
• Semantic search runs in parallel to capture conceptual matches.
• RRF combines lexical (with metadata boosting) and semantic results to get the best of both approaches.

Expected response:

{
  "hits": {
    "hits": [
      {
        "_source": {
          "title": "Authentication in Serverless",
          "deployment_type": "serverless",
          "feature_supported": false,
          "content_snippet": "Custom authentication not available in Serverless. Use SSO with your identity provider."
        }
      },
      {
        "_source": {
          "title": "User Management in Serverless",
          "deployment_type": "serverless",
          "feature_supported": false,
          "content_snippet": "Direct user creation not supported in Serverless. Use your organization's IdP."
        }
      },
      {
        "_source": {
          "title": "Role-Based Access in Serverless",
          "deployment_type": "serverless",
          "feature_supported": true,
          "content_snippet": "Configure roles in Serverless console. Roles sync with SSO provider groups."
        }
      },
      {
        "_source": {
          "title": "API Keys in Serverless",
          "deployment_type": "serverless",
          "feature_supported": true,
          "content_snippet": "Create API keys for programmatic Serverless access. Keys inherit user permissions."
        }
      },
      {
        "_source": {
          "title": "SSO Configuration for Serverless",
          "deployment_type": "serverless",
          "feature_supported": true,
          "content_snippet": "Configure SSO in Serverless via Cloud console with SAML 2.0 or OIDC."
        }
      }
    ]
  }
}

How metadata boosting resolves conflicts

Impact on LLM response quality

• Without metadata boosting: The context window receives equal-weight documents from all deployment types. The LLM produces vague responses that hedge between possibilities, failing to clearly state deployment-specific limitations.
• With metadata boosting (3x): Managed-specific documentation dominates the top results. The LLM generates direct answers about feature unavailability and provides actionable alternatives while maintaining the cross-deployment context for follow-up questions.

Semantic noise

Vector similarity search can retrieve documents that are semantically related but contextually irrelevant to the user’s need. This “semantic drift” occurs when embeddings capture a similarity without understanding the query intent. So when your context window fills with irrelevant information, the LLM's ability to generate precise answers declines.

Impact

The LLM receives correct information that doesn’t answer the question, wasting the context window and lowering the quality of the provided answer.

Solution: Hybrid search

Elasticsearch hybrid search combines lexical precision with semantic understanding, using explicit product filters to eliminate cross-product drift while maintaining conceptual recall.

Example: Technical documentation search

A developer searches for “Elastic Agent configuration,” and your knowledge base contains both the Elastic Agent (Elastic Observability) and the Elastic Agent Builder documentation. Both use the word "agent" prominently, making them semantically similar.

Let’s search for agent configuration documentation:

POST elastic-docs/_search
{
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "multi_match": {
                "query": "agent configuration logs metrics collection",
                "fields": ["title^3", "content", "tags^2"],
                "type": "best_fields"
              }
            }
          }
        },
        {
          "standard": {
            "query": {
              "semantic": {
                "field": "content_semantic",
                "query": "configuring agents to collect logs and metrics from hosts"
              }
            }
          }
        }
      ],
      "filter": [
        {"terms": {"product": ["observability", "elastic-agent"]}},
        {"term": {"doc_type": "configuration"}}
      ],
      "rank_window_size": 50,
      "rank_constant": 20
    }
  },
  "_source": ["title", "product", "tags", "url"],
  "size": 5
}

This hybrid query:

• Lexical component (multi_match) ensures exact keyword matches for "agent", "configuration", "logs", "metrics", and “collection”.
• Field boosting (title^3, tags^2) prioritizes documents where terms appear in important fields.
• Semantic component captures conceptual relationships and the intent about “configuring data collection agents”.
• RRF merges both result sets with balanced ranking using rank_constant: 20.
• Product filter restricts results to Elastic Observability and Elastic Agent domains, eliminating Agent Builder docs entirely.
• Category filter restricts results to "observability" and "elastic-agent" domains, eliminating semantic drift to other domains.

Expected response:

{
  "hits": {
    "hits": [
      {
        "_source": {
          "title": "Elastic Agent Input Configuration",
          "product": "elastic-agent",
          "tags": ["inputs", "logs", "metrics", "configuration"],
          "url": "/docs/elastic-agent/inputs"
        }
      },
      {
        "_source": {
          "title": "Configure Elastic Agent for Log and Metric Collection",
          "product": "elastic-agent",
          "tags": ["configuration", "logs", "metrics", "observability"],
          "url": "/docs/elastic-agent/configure"
        }
      },
      {
        "_source": {
          "title": "Agent Policies and Integrations",
          "product": "observability",
          "tags": ["policies", "integrations", "fleet"],
          "url": "/docs/fleet/policies"
        }
      },
      {
        "_source": {
          "title": "Configuring Agent Outputs",
          "product": "elastic-agent",
          "tags": ["outputs", "elasticsearch", "logstash"],
          "url": "/docs/elastic-agent/outputs"
        }
      },
      {
        "_source": {
          "title": "Manage Elastic Agents with Fleet",
          "product": "observability",
          "tags": ["fleet", "agent-management", "deployment"],
          "url": "/docs/fleet/manage-agents"
        }
      }
    ]
  }
}

Why hybrid search works

Before and after: LLM response comparison

Elasticsearch RAG best practices

Following these best practices optimizes your context engineering and significantly reduces the risk of context poisoning in your RAG systems. By implementing the following strategies, you ensure that every token in your context window contributes to relevant, accurate, and trustworthy LLM responses.

1. Choose the right search strategy for your data:
   Select your search approach based on your data characteristics and query patterns. Choose between lexical, semantic, or hybrid search. For more details, refer to Search approaches | Elastic Docs.
2. Implement temporal awareness
   Time-sensitive information requires active management to prevent outdated content from contaminating your context window. Use range queries with relative time filters (like now-6M or now-1y) for content that changes frequently, ensuring your RAG system prioritizes recent content. For more details, refer to Range query | Reference.
3. Use metadata boosting
   When your knowledge base contains similar content across different contexts, such as multiple product versions, deployment types, or user roles, metadata boosting helps prioritize contextually relevant results. For more details, refer to Boolean query | Reference.
4. Apply reranking when needed
   For complex or high-priority queries where precision is critical, consider implementing a reranker solution where the model can significantly improve the search result quality by reordering results based on the semantic understanding of the queries and documents. For more details, refer to Ranking and reranking | Elastic Docs.
5. Optimize chunking strategies
   Chunking is the process of breaking down large text into smaller “chunks.” Document chunking strategy affects both semantic representation and retrieval precision. Smaller chunks provide more granularity, but you may lose context; and larger chunks preserve more context, but you reduce retrieval precision. For more details, refer to Understanding chunking strategies in Elasticsearch.
6. Filter the data before it reaches the LLM
   Vector similarity search can retrieve semantically related but contextually irrelevant documents. Apply explicit filters on product, category, or domain fields to constrain results to the appropriate context before delivering the results to the LLM. For more details, refer to RAG pipelines in production: Operationalize your GenAI project - Elasticsearch Labs.
7. Calibrate your retrieval volume (k)
   Finding the "Goldilocks zone" for the number of documents retrieved is essential. Too few results lead to incomplete answers, while too many can cause the LLM to miss key facts. Balance your token budget against the depth of the model's window. For more details, refer to kNN search in Elasticsearch | Elastic Docs.
8. Consider summarization for large documents
   When retrieved content exceeds your context budget, summarization techniques help retain essential information while reducing token count. For more details, refer to Adding AI summaries to your site with Elastic - Elasticsearch Labs.
9. Monitor and iterate
   Over time, as knowledge bases grow and content evolves, we recommend that you implement monitoring to track relevance score distributions, temporal patterns in retrieved results, and user feedback signals. Watch for signs like outdated documents, declining user satisfaction scores, or a growing number of “no relevant results” queries. For more details, refer to Elastic Observability: Streams Data Quality and Failure Store Insights.

Conclusion

The new era of million-token context windows has not made context management obsolete; it has made context engineering more critical than ever. As context windows grow, so does the potential for poisoning from any source retrieval, tools, or memory.

The patterns shown in this article apply beyond just RAG. Temporal filtering, metadata boosting, and hybrid search are foundational techniques that improve context quality, regardless of source.

By implementing these strategies, you maintain control over what information reaches your LLM, ensuring relevance, accuracy, and trust at scale.

jina-vlm opens up new application areas:

• Accessibility: Creating descriptions of images for visually impaired people.
• Indexing: Generating detailed textual information to improve search for images and documents that contain images.
• Querying: Matching natural language queries to images through direct analysis of image content.
• Visual content analysis: Automatically inspecting images for classification and analysis.
• Retrieval-augmented generation (RAG): Using visual information as a knowledge source in RAG systems.

This article will introduce you to VLMs and show you how to use jina-vlm to bring multimodal AI to your own solutions.

Vision-language models

jina-vlm is a decoder-only language model with additional features and training to answer questions or create general descriptions of images.

We adapted the Qwen3-1.7B-Base open weight language model as the backbone of jina-vlm. What makes it a vision language model is that we’ve connected it to a custom fine-tuned image encoder model, derived from the SigLIP2-So400M/14-384 model, which converts images into a form that the language model can understand and trained it specifically for VQA and description generation.

To understand how this works, think about how language models “read” your prompts. Input texts are split into tokens (words or parts of words), which are replaced with embedding vectors from a lookup table and then inputted to a neural network. The model never sees the actual letters you typed.

As seen in the image, each unique token is substituted with a number that, in turn, corresponds to an embedding vector stored in the model’s vocabulary. This enables the model to process whatever text you give it, even if the words aren’t in its stored vocabulary.

This is why language models struggle to count letters in obscure words, like this example with Claude Haiku 4.5:

This model correctly recognizes what Qaqortoq is; it knows that the word means a town in Greenland, and it can tell you all about it, but it has no idea which actual letters compose the name. There are ways to supplement language models to address this shortcoming, so some models (like ChatGPT 5, Gemini, and Claude Sonnet) don’t generally make this kind of mistake.

However, this architectural shortcoming makes it possible to create multimodal language models.

The actual input to generative language models is a sequence of semantic embeddings, and there’s no reason those embeddings have to come from texts. They could come from anything, as long as they’re compatible with the model’s training. We can make any language model handle images, provided we can generate vectors for those images in the same embedding space as the tokens.

jina-vlm contains an image embedding model and adapter trained to do just that. The image (with some preprocessing into overlapping patches) is the input to the model, and it outputs a sequence of embeddings in the same semantic embedding space as the language model’s text token embeddings. These embeddings are then treated just like the text input embeddings. The full language model has access to your text and to the image at the same time, and it treats both as the same kind of input.

Figure 3 is a schematic of the jina-vlm model, showing the dual track processing of text and images together.

This architecture has some important benefits. Language models learn a lot of implicit information about the world from their training data, and the VLM architecture can use that information to help understand the content of images. Furthermore, some of the bias problems present in other multimodal architectures are absent in this one, notably the media “gap” that troubles CLIP-style models.

What can jina-vlm do?

jina-vlm has three overlapping core competencies:

• Describing images and generating captions.
• Understanding informationally rich images.
• Answering questions about the content of images.

Describing and captioning pictures

Providing good descriptions of images can be a boon for accessibility. There are increasing legal and informal requirements to provide alternative text (alt-text) for digitized images on the internet. Search engine optimization (SEO) strategies in particular often demand high-quality image captions and alt-text. Manually composing such descriptions can be expensive and time consuming.

jina-vlm can help. For example:

You can input this image to jina-vlm with the text:

• Input Instruction: Describe this image in detail.

The result is the following:

• Output: The image captures a vibrant scene of a white plate filled with a multitude of colorful jelly beans. The jelly beans, in hues of green, yellow, orange, purple, red, and blue, are scattered across the plate, creating a rainbow-like effect. The plate itself is placed on a gray surface, providing a stark contrast to the colorful jelly beans. The image does not contain any discernible text. The relative positions of the jelly beans vary, with some clustered together and others spread out, giving the image a sense of depth and texture. The overall composition of the image is simple yet striking, with the colorful jelly beans being the central focus.

This description is a little wordy but quite detailed.

jina-vlm treats the text you give it as a question or instruction, and some prompt engineering may improve the result a lot. For example, using the same picture of jelly beans:

• Input Instruction: Describe this image. Be concise.
• Output: A white plate is filled with a variety of colorful jelly beans, including green, yellow, red, purple, and blue, arranged in a scattered manner.

Or:

• Input Instruction: Provide a caption for this picture.
• Output: A white plate full of colorful jelly beans.

Asking specifically for a caption tends to produce very short results. Most captions are brief, so jina-vlm is trained to generate brief captions.

You can also use jina-vlm to identify and list items in images:

As a relatively small model, jina-vlm has limits to its object recognition. For common categories, it’s very good. For example:

It’s able to handle content like “red cars”, but ask jina-vlm a question involving detailed categories and it can become unreliable very quickly. For example, using the same image:

• Input Instruction: What model of car is this?
• Output: The car in the image is a classic French car, specifically a Citroën DS. This model is known for its distinctive rounded body shape, which was a hallmark of the DS series. The DS was produced from 1955 to 1969 and was a popular choice for its combination of style, performance, and practicality. It was designed to be a luxury car that could be driven on city streets as well as on the roads, making it a versatile vehicle for its time. The DS was also notable for its innovative features, such as the rear spoiler and the front grille with the Citroën emblem.

This answer is largely hallucinated. The car in the picture is one of the most recognizable cars in history: a Citroën 2CV, manufactured from 1948 to 1990. The Citroën DS was made from 1955 to 1975 and looked like this:

In short, when pressed for details beyond its capacity, jina-vlm hallucinates without warning.

It’s especially important to keep this in mind with images of people:

Tom Cruise is famous enough, and present enough in the training data, for jina-vlm to recognize him. But, using the same picture:

• Input Instruction: Provide a caption for this picture.
• Output: A man with a green shirt and a big smile.

This caption never mentions Tom Cruise, even though it’s the most important fact about the picture. Asking for a caption produces very little hallucination but also very little information.

In contrast, let’s do the same thing with this image:

This is terribly wrong. Ronald Reagan looks nothing like Richard Nixon.

When you ask jina-vlm to identify persons, it’s likely to hallucinate because it doesn’t have very deep world knowledge. Only people incidentally included many times in the training data might be correctly identified.

jina-vlm hallucinates much less when asked to write a caption. For example, using the same picture of Richard Nixon:

• Input Instruction: Provide a caption for this picture.
• Output: A man in a suit sits in a chair with a bookshelf behind him.

The most effective way to reduce hallucination is to prompt the model to say less. Any use or deployment of jina-vlm should take this into account.

Understanding informational content

jina-vlm truly shines at understanding informational graphics and synthetic images, like diagrams, graphs, screenshots, scans of printed pages, advertising copy, and presentation slides. The examples below show only part of jina-vlm’s range.

Charts

jina-vlm can report on the contents of all sorts of data presentations.

Bar charts:

Pie charts:

Area charts:

Line graphs:

Table:

Multilingual content

jina-vlm’s multilingual support means that it can recognize languages and describe informational content in a different language from its presentation:

It can also list the languages when there’s more than one:

jina-vlm may answer in a different language than you asked for, if the image content is in another language or strongly associated with it:

To fix this, specify the language in the query. With the same image:

• Input Instruction: Summarize the content of this document in English.
• Output: The image is a promotional poster for a KFC Thailand event. The poster is predominantly red with white and black text. At the top, there is a logo of KFC, which is a red and white square with a black silhouette of Colonel Sanders. Below the logo, there are three young men, presumably the main participants or hosts of the event. They are dressed in casual attire, with two of them wearing red jackets and one in a white shirt with a black stripe.
  
  The central part of the poster features a large text in white that reads, “เปิดตัวใหม่! ร่วมกับ KFC Thailand พบกับความสนุกสนานที่ไม่เหมือนใคร!” which translates to “Launching New! Join KFC Thailand and experience something unique!” Below this, there is a smaller text that says, “ทุกคนมีส่วนร่วม! วันนี้ 8 พฤศจิกายน 64 เวลา 12:00 น.” indicating that the event is happening on November 8, 2021, at 12:00 PM.
  
  The bottom section of the poster includes social media icons and a Facebook page link, suggesting that the event will be live on Facebook. There are also icons of a chicken drumstick and a bucket, which are associated with KFC, reinforcing the brand’s identity.
  
  The overall design of the poster is vibrant and eye-catching, with a clear focus on the event’s theme and the brand’s presence. The use of red and white colors is typical for KFC branding, which is known for its bold and energetic visual style. The poster is designed to attract attention and inform viewers about the upcoming event.

Complex information displays

We specifically trained jina-vlm to understand complex information displays and to be able to summarize them in natural language.

Infographics:

Technical and educational diagrams:

Maps:

Screenshots

jina-vlm has been trained to parse and understand screenshots:

It can even parse video games:

Scans

jina-vlm excels at materials like scans and images of printed text:

Pre-digital materials

Robust support for scans also makes pre-digital materials accessible to digital search technologies:

Social media

Since much of the internet is “five giant websites filled with screenshots of text from the other four,” jina-vlm is especially helpful in processing the kind of content that the web is full of:

jina-vlm can sometimes even explain memes:

Visual question-answering

“A picture is worth a thousand words.” As cliché as that saying is, multimodal AI can prove it.

VQA is the domain where jina-vlm excels and the one with the most potential for new and revolutionary applications. jina-vlm answers questions about the content of images just like the way a language model can answer questions about texts. This means it can serve as part of a RAG system that assembles responses from mixtures of images and texts stored in a database.

The infographic below is very information rich, and with jina-vlm you can ask natural language questions about its content:

Or, using the same image:

• Input Instruction: What is the crew module of the SLS called?
• Output: Orion spacecraft

This capability makes it possible to query formal schematics and diagrams:

With the same image:

• Input Instruction: How many bathrooms are there?
• Output: 4

If you use photographs or natural scenes, jina-vlm is typically good with high-visibility characteristics:

But if you’re asking about finer, less visible details, the answers might get spotty:

Thank you, Captain Obvious.

jina-vlm is also not well-suited to resolving optical illusions:

You can ask questions about text in photographs:

And jina-vlm is ideally suited to understand traditional magazine and newspaper advertisements:

Visual question-answering also works cross-lingually:

Using the same image:

• Input Instruction: How much do grapes cost per kilo?
• Output: The cost of grapes per kilogram is 3.98.

It answers questions from manual pages:

It can also answer questions about the content of posters and artistic works:

We showed in the previous section that jina-vlm is not very good at general face recognition, outside of a few very famous people. However, it can use the text in images to answer questions it can’t answer from just a picture:

Best practices

jina-vlm is a generative language model. It’s not a database. It’s prone to error and hallucination. It’s very important to take that into account in your use case.

To minimize errors, we offer the following advice:

Test your prompts

If you depend on getting certain kinds of results consistently, you should do thorough quality testing on your queries. Getting a few good results with one prompt does not guarantee the same quality of result with different images. You can minimize this risk by manually reviewing the results of queries over a broad collection of images that are genuinely representative of what you plan to process.

Specify the language

Add to your query the language you expect the answer to be in. Like all language models, jina-vlm cannot easily distinguish between your instructions and the data it’s supposed to apply your instructions to. The only way for it to know how to answer is to look at the languages it recognizes in its input or to follow your explicit instructions.

Don’t ask leading questions

It’s very easy to get jina-vlm to hallucinate by asking leading questions. For example:

Prompt for short answers

jina-vlm can be very wordy sometimes. Using words like “briefly” and “concise” reduces the size of the output.

Compare:

And with the same image:

• Input Instruction: What kind of vehicle is this? Be very concise.
• Output: This is a Massey Ferguson tractor.