HyDE in Elasticsearch: 50% better semantic search precision

Elasticsearch has native integrations with the industry-leading Gen AI tools and providers. Check out our webinars on going Beyond RAG Basics, or building prod-ready apps with the Elastic vector database.

To build the best search solutions for your use case, start a free cloud trial or try Elastic on your local machine now.

Hypothetical Document Embeddings (HyDE) improves semantic search precision and recall by 50% on short, casual queries against formal document corpora, without reindexing or changing your embedding model. The technique works by asking an LLM to generate a hypothetical document that matches your query, then using that document's embedding as the search vector. The fake document gets discarded; only real results come back. This post shows you how to implement HyDE in Elasticsearch using the Inference API and semantic_text, measure it against a baseline with precision, recall and MRR, and decide when it's worth the extra LLM round trip.

Prerequisites

Elastic Cloud cluster or Elasticsearch 9.x+ (start a free trial)
Python 3.9+
An OpenAI API key for hypothetical document generation.

What is HyDE, and why does it work?

HyDE closes the embedding gap between short queries and formal documents by generating a full-length hypothetical document in the same register as the corpus before embedding.

An embedding doesn't just encode a topic. It encodes topic, register, vocabulary density, and sentence structure, all packed into one vector. A nine-word casual query, like "adamw vs Adam transformers", carries a handful of semantic signals, so its embedding sits in a vague region of the vector space. A 200-word abstract covering the same subject packs dozens of reinforcing terms (weight decay, gradient update, convergence, generalization) that anchor its embedding much more precisely. The result: A semantic search for the short query can drift toward loosely related papers instead of landing on the best matches.

HyDE closes this gap by asking an LLM to write a document in the same style as the corpus before embedding, a strategy that complements other query rewriting techniques for improving search quality. The flow looks like this:

A flowchart showing a query “why adamw transformers” leading to a hypothetical document about AdamW and transformer architectures and then embedding into a vector search that returns three search results titled “Understanding AdamW through Proximal Methods,” “Trainable Weight Averaging for Fast Convergence,” and “Subformer: Exploring Weight Sharing.”

The hypothetical document doesn’t need to be factually correct. It may contain false details, and that’s fine, because the document is thrown away after its embedding is extracted. The user only ever sees real documents from the index.

This technique was introduced in Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao et al., 2022).

How to set up HyDE in Elasticsearch

All the code snippets will be available in the companion notebook, if you prefer to run everything at once.

Load the ML arXiv dataset

We sample 5,000 papers from CShorten/ML-ArXiv-Papers: machine-learning arXiv titles and abstracts. This gives us a corpus of formal, technical documents, the kind where short queries struggle most.

Index with semantic_text

Elasticsearch as a vector database lets us handle embedding generation, storage, and search in a single system. We use the .jina-embeddings-v5-text-small inference endpoint, preconfigured in Elastic Cloud. The copy_to parameter lets a single semantic_text field cover both title and abstract, so both fields are embedded and searchable through one vector. This walkthrough of the semantic_text GA release covers the full capabilities of this field type, including semantic highlighting.

Bulk-index all 5,000 papers:

Create the chat completion inference endpoint

We register an OpenAI gpt-4o-mini endpoint via the Elasticsearch Inference API. Routing the LLM call through Elasticsearch (instead of calling OpenAI directly) keeps API key handling, retries, and observability inside your cluster:

Baseline semantic search in Elasticsearch

Before introducing HyDE, we need a reference point. Here’s a simple semantic search function using the semantic query:

Results:

The top result is relevant, but results 3–5 drift toward "adversarial training" and "rotation equivariant geometry," both off-topic. The baseline query is too short to anchor the embedding in the right neighborhood.

Generate a hypothetical document with the Inference API

Now we ask the LLM to write a hypothetical abstract that would be a perfect match for our query. The prompt asks for formal academic register, 150–200 words, matching the density of real arXiv abstracts:

This hypothetical abstract is dense with transformer-specific optimization vocabulary: weight decay, gradient updates, convergence, generalization, ablation studies. That vocabulary is what will pull the embedding toward the right neighborhood.

HyDE search

Now we search with the hypothetical document instead of the original query:

The results shifted. "VectorAdam" and "adversarial training" papers are gone, replaced by papers about weight averaging for convergence, parameter efficiency in transformers, and optimizer quantization, all closer to what the original query was really asking about.

Does HyDE improve semantic search? Evaluation results

HyDE improved precision and recall by 50% on two of the three test queries. The third query already named both optimizers explicitly, so the baseline embedding was already well-anchored.

We define a hand-curated judgment set for three queries, listing the papers considered relevant, and measure standard retrieval metrics:

Precision@5: Fraction of the top 5 that are in the relevance set.
Recall@5: Fraction of the relevance set captured in the top 5
MRR: Reciprocal rank of the first relevant hit (0 if none).

Run both methods across all three queries, and collect results:

Results

Two bar charts titled “HyDE vs baseline on a hand‑curated judgment set.” The top chart shows precision@5 scores and the bottom chart shows recall@5 scores for three queries about AdamW, mixture of experts, and catastrophic forgetting. Orange bars for HyDE are equal to or higher than blue bars for baseline.

HyDE improved precision and recall on queries 2 and 3 but tied with the baseline on query 1. Query 1 ("why does adamw train transformers better than plain adam") already names both optimizers explicitly, so the baseline embedding lands close to the right papers without extra help. The hypothetical abstract adds density but also commits to specific framing, which can swap one relevant paper for another without a net gain. Queries 2 and 3 are vaguer and give the baseline less to anchor on. Here the hypothetical document fills the gap with domain vocabulary the short query lacks, pulling the embedding into a more precise region of the vector space.

When to use HyDE (and when not to)

HyDE is not a universal upgrade. Here’s when it helps and when to be careful:

Good candidates for HyDE:

Short, casual queries against a corpus of formal documents (academic papers, legal filings, technical reports).
Domain-specific corpora where the register gap between how users ask and how documents are written is large.
Retrieval pipelines where precision or recall are not good enough, as a low-cost technique to improve results without reindexing or changing your embedding model.

Practical considerations:

Use a small, fast model since the hypothetical only needs to be topically correct, not factually perfect.
Cache hypothetical documents for repeated or similar query patterns to avoid redundant LLM calls.
Consider running HyDE selectively: Use it for short queries (under ~10 words), and skip it for longer, more specific queries that already carry enough semantic signal.

Conclusion

HyDE closes the embedding distribution gap between short queries and formal documents by using an LLM-generated hypothetical document as the search vector. The Elasticsearch Inference API handles both the LLM generation and the embedding step without leaving the cluster, keeping the implementation compact.

On our dataset, HyDE improved precision and recall by 50% on two out of three test queries and tied on the third. That said, it isn’t free: It adds an LLM round trip to every query, and when the model commits to one interpretation of an ambiguous question, it can narrow retrieval instead of broadening it. It’s an interesting alternative, but first evaluate your own data before adopting it as a default.