Elasticsearch vector search is up to 8x faster than OpenSearch

Why search speed matters for AI agents and context engineering

Our benchmarks on a 20M document corpus show that Elasticsearch delivers up to 8x higher throughput than OpenSearch for filtered vector search, while also achieving higher Recall@100 across the configurations we tested. Context engineering depends on more than fast vector retrieval. Teams also need strong relevance controls, like hybrid search and filtering, operational simplicity, and predictable performance, as workflows iterate. But because agents often run retrieve, reason, retrieve loops many times per request, retrieval latency becomes a multiplier, so improvements here translate directly into better end-to-end responsiveness and lower cost.

For context engineering, retrieval isn’t a one-time step. Agents and applications repeatedly run loops, such as retrieve → reason → retrieve, to refine queries, verify facts, assemble grounded context, and complete tasks. This pattern is common in agentic workflows and iterative retrieval augmented generation (RAG). Because retrieval may be invoked many times per user request, it adds delay to the response and/or increases infrastructure costs.

Why is vector search performance critical?

Imagine a shopping assistant answering the question, “I need a carry-on backpack under $60 that fits a 15-inch laptop, is water resistant, and can arrive by Friday.”

In production, the assistant rarely issues one vector query and stops. It runs a retrieval loop to build the right context, and each step is typically constrained by filters, like availability, region, shipping promise, brand rules, and policy eligibility.

Step 1: Interpret intent and translate to constraints.

The agent turns the request into structured filters and a semantic query, such as:

• Filters: In stock, deliverable to the user’s postcode, delivery by Friday, price under $60, valid listing

• Vector query: “Carry-on backpack 15-inch laptop water resistant”

Step 2: Retrieve candidates, and then refine.

It often repeats retrieval with variations to avoid missing good matches:

• “travel backpack carry on laptop sleeve”

• “water resistant commuter backpack 15 inch”

• “lightweight cabin backpack”

Each query uses the same eligibility filters, because retrieving irrelevant or unavailable items is wasted context.

Step 3: Expand to confirm details and reduce risk.

The agent then retrieves again to verify key attributes that affect the final answer:

• Material and water resistance wording

• Dimensions and laptop compartment fit

• Return policy or warranty constraints

• Alternate options if inventory is low

This is multistep context engineering: Retrieve, reason, retrieve, assemble.

Why latency and recall matter for context engineering

These interactions can involve dozens of filtered retrieval calls per user session. That makes per-call latency a direct multiplier on end-to-end response time, and low recall forces extra retries or causes the agent to miss eligible items, degrading answer quality.

Takeaway: In context-engineered systems, filtered approximate nearest neighbors (ANN) isn’t a single lookup. It’s a repeated operation under constraints, so vector search performance shows up immediately in latency, throughput, and cost, even when the large language model (LLM) is the most visible component.

Benchmarking

Results

In Graph 2, each dot represents one test configuration. The best results appear toward the top left, meaning higher recall with lower latency. Elasticsearch’s results are consistently closer to the top left than OpenSearch’s, indicating better speed and accuracy under the same workload settings.

Some key insights

• s_n_r_value: Shorthand for size_numCandidates_rescoreOversample (k and numCandidates set equal to numCandidates in these tests), for example, 100_500_1 means size=100, numCandidates=500 and k=500, rescore oversample=1

• Recall: Measured Recall@100 for that configuration

• Avg latency (ms): Average end-to-end latency per query

• Throughput: Queries per second

• Recall %: Relative recall lift of Elasticsearch versus OpenSearch (Elasticsearch minus OpenSearch) / OpenSearch

• Latency Xs: OpenSearch average latency divided by Elasticsearch average latency

• Throughput Xs: Elasticsearch throughput divided by OpenSearch throughput

For example, at 100_9000_1, OpenSearch averages 687 milliseconds per retrieval versus 90 milliseconds on Elasticsearch, and in a 10-step retrieval loop that’s about 10 x (687 - 90) = six seconds of additional waiting time.

See the full results.

Methodology

Using Python to send the queries and track the response timing and other statistics, we sent the following queries to the engines. Bear in mind that the performance of any vector search engine depends on how you tune its core parameters: how many candidates to consider, how aggressively to rescore, and how much context to return. These settings directly affect both recall (the likelihood of finding the right answer) and latency (how fast you get results).

In our benchmarks, we used the same candidate, rescore, and result-size settings you’d typically tune in an agentic retrieval loop, and we measured how Elasticsearch performs under that workload. We then ran OpenSearch with the same settings as a reference.

OpenSearch

GET <INDEX_NAME>/_search
{
  "query": {
    "knn": {
      "<DENSE_VECTOR_FIELD_NAME>": {
        "vector": [...],
        "k": <NUMBER_OF_CANDIDATES>,
        "method_parameters": {
          "ef_search": <NUMBER_OF_CANDIDATES>
        },
        "rescore": {
          "oversample_factor": <OVERSAMPLE>
        },
        "filter": {
          <SOME_FILTER>
        }
      }
    }
  },
  "size": <RESULT_SIZE>,
  "_source": {
    "excludes": [
      "<DENSE_VECTOR_FIELD_NAME>"
    ]
  }
}

• "size": <RESULT_SIZE>: Number of hits returned to the client. In this benchmark, result size is 100 to compute Recall@100.

• "k": <NUMBER_OF_CANDIDATES>: The number of nearest neighbor candidates.

• "ef_search": <NUMBER_OF_CANDIDATES>: The number of vectors to examine.

• "oversample_factor": <OVERSAMPLE>: How many candidate vectors are retrieved before rescoring.

Elasticsearch

GET <INDEX_NAME>/_search
{
  "query": {
    "knn": {
      "field": "<DENSE_VECTOR_FIELD_NAME>",
      "query_vector": [...],
      "k": <NUMBER_OF_CANDIDATES>,
      "num_candidates": <NUMBER_OF_CANDIDATES>,
      "rescore_vector": {
        "oversample": <OVERSAMPLE>
      },
      "filter": {
        <SOME_FILTER>
      }
    }
  },
  "size": <RESULT_SIZE>,
  "_source": {
    "excludes": [
      "<DENSE_VECTOR_FIELD_NAME>"
    ]
  }
}

• "size": <RESULT_SIZE>: Number of hits returned to the client. In this benchmark, result size is 100 to compute Recall@100.

• "k": <NUMBER_OF_CANDIDATES>: Number of nearest neighbors to return from each shard.

• "num_candidates": <NUMBER_OF_CANDIDATES>: Number of nearest neighbor candidates to consider per shard while doing knn search.

• "oversample": <OVERSAMPLE>: How many candidate vectors are retrieved before rescoring.

Example

Knn query, (100_500_1), would be as follows:

OpenSearch

GET search_catalog_128/_search
{
  "query": {
    "knn": {
      "search_catalog_embedding": {
        "vector": [...],
        "k": 500,
        "method_parameters": {
          "ef_search": 500
        },
        "rescore": {
          "oversample_factor": 1
        },
        "filter": {
          "term": {
            "valid": true
          }
        }
      }
    }
  },
  "size": 100,
  "_source": {
    "excludes": [
      "search_catalog_embedding"
    ]
  }
}

Elasticsearch

GET search_catalog_128/_search
{
  "query": {
    "knn": {
      "field": "search_catalog_embedding",
      "query_vector": [...],
      "k": 500,
      "num_candidates": 500,
      "rescore_vector": {
        "oversample": 1
      },
      "filter": {
        "term": {
          "valid": true
        }
      }
    }
  },
  "size": 100,
  "_source": {
    "excludes": [
      "search_catalog_embedding"
    ]
  }
}

The full configuration, alongside Terraform scripts, Kubernetes manifests and the benchmarking code is available in this repository in the folder es-9.3-vs-os-3.5-vector-search.

Cluster setup

We ran our tests on six e2-standard-16 cloud servers, each with 16 vCPUs and 64 GB RAM. On each server, we allocated 15 vCPUs and 56 GB RAM to each Kubernetes pod running the search engine node, with 28 GB reserved for the JVM heap.

The clusters ran Elasticsearch 9.3.0 and OpenSearch 3.5.0 (Lucene 10.3.2). Because both systems use the same Lucene version in this benchmark, the throughput and latency differences we observe cannot be attributed to Lucene alone and instead reflect differences in how each engine integrates and executes filtered k-nearest neighbor (kNN) retrieval and rescoring. We used a single index with three primary shards and one replica (so 6 shards total, 1 per node).

We also used a separate server in the same region to run the benchmark client and collect timing statistics.

The dataset

For this benchmark, we used a large-scale ecommerce-style catalog embedding dataset with 20 million documents, designed to reflect real-world filtered vector retrieval at scale.

Each document represents a catalog item and includes:

• A 128-dimensional dense vector embedding used for approximate kNN retrieval.

• Structured metadata fields used for filtering (for example, item validity and availability plus other catalog constraints) enabling the common production pattern of retrieving the nearest neighbors but only within an eligible subset.

We chose this dataset because it captures the core performance challenge we see in agentic and RAG-style systems in production: Vector similarity alone is not enough, retrieval is frequently constrained by filters, and the system must maintain high recall while keeping latency low under those constraints. Compared to smaller QA-style datasets, a 20M document corpus also better reflects the scale and candidate pressure that filtered ANN systems face in practice.

Conclusion

In modern AI architectures, especially those built around context engineering, vector search speed isn’t a minor implementation detail. It’s a multiplier. When agents and workflows iterate through retrieve → reason → retrieve, retrieval performance directly shapes end-to-end latency, throughput, and the quality of the context fed into the model.

In our benchmarks, Elasticsearch consistently delivered higher recall at lower latency than OpenSearch in scenarios where correctness depends on retrieving the right document, not just a similar vector. On a controlled dataset, the difference is clear, and in production those gains accumulate across large volumes of retrieval calls, improving responsiveness, increasing capacity headroom, and reducing infrastructure costs.

Further reading

1. What is context engineering?

2. The evolution of hybrid search and context engineering

3. The impact of relevance in context engineering for AI agents