Florian Bernd - Elasticsearch Labs

LINQ to Elasticsearch ES|QL: Write C#, query Elasticsearch

Florian Bernd,Martijn Laarman — Wed, 01 Apr 2026 00:00:00 GMT

Starting with v9.3.4 and v8.19.18, the Elasticsearch .NET client includes a Language Integrated Query (LINQ) provider that translates C# LINQ expressions into Elasticsearch Query Language (ES|QL) queries at runtime. Instead of writing ES|QL strings by hand, you compose queries using Where, Select, OrderBy, GroupBy, and other standard operators. The provider takes care of translation, parameterization, and result deserialization, including per-row streaming that keeps memory usage constant, regardless of result set size.

Your first query

Start by defining a plain old CLR object (POCO) that maps to your Elasticsearch index. Property names are resolved to ES|QL column names through standard System.Text.Json attributes, like [JsonPropertyName], or through a configured JsonNamingPolicy. The same source serialization rules that apply across the rest of the client apply here as well.

using System.Text.Json.Serialization; public class Product { [JsonPropertyName("product_id")] public string Id { get; set; } public string Name { get; set; } public string Brand { get; set; } [JsonPropertyName("price_usd")] public double Price { get; set; } [JsonPropertyName("in_stock")] public bool InStock { get; set; } }

With the type in place, a query looks like this:

The provider translates this into the following ES|QL:

A few details to note:

Property name resolution: p.Price becomes price_usd because of the [JsonPropertyName] attribute, and p.Brand becomes brand following the default camelCase naming policy.
Parameter capturing: The C# variables minPrice and brand are captured as named parameters (?minPrice, ?brand). They’re sent separately from the query string in the JSON payload, which prevents injection and enables server-side query plan caching.
Streaming: QueryAsync returns IAsyncEnumerable. Rows are materialized one at a time as they arrive from Elasticsearch.

You can also inspect the generated query and its parameters without executing it:

var query = client.Esql.CreateQuery() .Where(p => p.InStock && p.Price >= minPrice && p.Brand == brand) .OrderByDescending(p => p.Price) .Take(10); Console.WriteLine(query.ToEsqlString()); // FROM products | WHERE (in_stock == true AND price_usd >= 100) | SORT price_usd DESC | LIMIT 10 Console.WriteLine(query.ToEsqlString(inlineParameters: false)); // FROM products | WHERE (in_stock == true AND price_usd >= ?minPrice AND brand == ?brand) | SORT price_usd DESC | LIMIT 10 var parameters = query.GetParameters(); // { "minPrice": 100.0, "brand": "TechCorp" }

How does this work? A quick LINQ refresher

The mechanism that makes LINQ providers possible is the distinction between IEnumerable and IQueryable.

When you call .Where(p => p.Price > 100) on an IEnumerable, the lambda compiles to a Func, a regular delegate that the runtime executes in-process. This is LINQ-to-Objects.

When you call the same method on an IQueryable, the C# compiler wraps the lambda in an Expression> instead. This is a data structure that represents the structure of the code rather than its executable form. The expression tree can be inspected, analyzed, and translated into another language at runtime.

// IEnumerable: the lambda is a compiled delegate IEnumerable local = products.Where(p => p.Price > 100); // IQueryable: the lambda is an expression tree, a data structure IQueryable remote = queryable.Where(p => p.Price > 100);

The IQueryProvider interface is the extension point. Any provider can implement CreateQuery and Execute to translate these expression trees into a target language. Entity Framework uses this to emit SQL. The LINQ to ES|QL provider uses it to emit ES|QL.

The expression tree for the query above looks like this:

Expression tree for the example query.

The tree is nested inside out: Take wraps OrderByDescending, which wraps Where, which wraps From, which wraps the root EsqlQueryable constant. The Where predicate is itself a subtree of BinaryExpression nodes for the &&, >=, and == operators, with MemberExpression leaves for property accesses and closure captures for the minPrice and brand variables. This is the data structure that the provider walks to produce the final ES|QL.

Under the hood: The translation pipeline

The path from a LINQ expression to query results follows a six-stage pipeline:

Translation pipeline overview.

1. Expression tree capture

When you chain .Where(), .OrderBy(), .Take() and other operators on an IQueryable, the standard LINQ infrastructure builds an expression tree. EsqlQueryable implements IQueryable and delegates to EsqlQueryProvider.

2. Translation

When the query is executed (by enumerating, calling ToList(), or using await foreach), the EsqlExpressionVisitor walks the expression tree inside out. It dispatches each LINQ method call to a specialized visitor:

Visitor

Translates

Into

WhereClauseVisitor

.Where(predicate)

WHERE condition

SelectProjectionVisitor

.Select(selector)

EVAL + KEEP + RENAME

GroupByVisitor

.GroupBy().Select()

STATS ... BY

OrderByVisitor

.OrderBy() / .ThenBy()

SORT field [ASC\|DESC]

EsqlFunctionTranslator

EsqlFunctions.*, Math.*, string methods

80+ ES|QL functions

During translation, C# variables referenced in expressions are captured as named parameters.

3. Query model

The visitors don’t produce strings directly. Instead, they produce QueryCommand objects, an immutable intermediate representation. A FromCommand, a WhereCommand, a SortCommand, and a LimitCommand, each representing one ES|QL processing command. These are collected into an EsqlQuery model.

Query model and command pattern.

This intermediate model is decoupled from both the expression tree and the output format. It can be inspected, intercepted (via IEsqlQueryInterceptor), or modified before formatting.

4. Formatting

EsqlFormatter visits each QueryCommand in order and produces the final ES|QL string. Each command becomes one line, separated by the pipe (|) operator that ES|QL uses to chain processing commands. Identifiers containing special characters are automatically escaped with backticks.

5. Execution

The formatted ES|QL string and captured parameters are sent to Elasticsearch’s /_query endpoint as a JSON payload. The IEsqlQueryExecutor interface abstracts the transport layer, which is where the layered package architecture comes into play.

6. Materialization

EsqlResponseReader streams the JSON response without buffering the entire result set into memory. A ColumnLayout tree, precomputed once per query, maps flat ES|QL column names (like address.street, address.city) to nested POCO properties. Each row is assembled into a T instance and yielded one at a time via IEnumerable or IAsyncEnumerable.

The layered architecture

The LINQ to ES|QL functionality is split across three packages:

Package architecture. Elastic.Esql is the pure translation engine. It has zero HTTP dependencies and contains the expression visitors, query model, formatter, and response reader. You can use it stand alone to build and inspect ES|QL queries without an Elasticsearch connection, which is useful for testing, query logging, or building your own execution layer.

// Translation-only: no Elasticsearch connection needed var provider = new EsqlQueryProvider(); var query = new EsqlQueryable(provider) .From("products") .Where(p => p.InStock) .OrderByDescending(p => p.Price); Console.WriteLine(query.ToEsqlString()); // FROM products | WHERE in_stock == true | SORT price_usd DESC

Elastic.Clients.Esql is a lightweight stand-alone ES|QL client. It adds HTTP execution on top of Elastic.Esql via Elastic.Transport. If your application only needs ES|QL and none of the other Elasticsearch APIs, this is the minimal dependency option.

Elastic.Clients.Elasticsearch is the full Elasticsearch .NET client. It also builds on Elastic.Esql and exposes the LINQ provider through the client.Esql namespace. This is the recommended entry point for most applications.

Both execution-layer packages provide their own implementation of IEsqlQueryExecutor, the strategy interface that bridges translation and transport.

All three packages are compatible with Native AOT when used with a source-generated JsonSerializerContext. For the full client, see the Native AOT documentation.

Beyond the basics

The example above covered filtering, sorting, and pagination. The provider supports a broader set of operations.

Aggregations

GroupBy, combined with aggregate functions in Select, translates to ES|QL STATS ... BY:

var stats = client.Esql.Query(q => q .GroupBy(p => p.Brand) .Select(g => new { Brand = g.Key, Count = g.Count(), AvgPrice = g.Average(p => p.Price), MaxPrice = g.Max(p => p.Price) })); // -> FROM products | STATS COUNT(*), AVG(price_usd), MAX(price_usd) BY brand

Projections

Select, with anonymous types generates EVAL, KEEP, and RENAME commands:

var query = client.Esql.CreateQuery() .Select(p => new { ProductName = p.Name, p.Price, p.InStock }); // -> FROM products | KEEP name, price_usd, in_stock | RENAME name AS ProductName

Rich function library

Over 80 ES|QL functions are available through the EsqlFunctions class, covering date/time, string, math, IP, pattern matching, and scoring. Standard Math.* and string.* methods are also translated:

.Where(p => p.Name.Contains("Pro")) // -> WHERE name LIKE "*Pro*" .Where(p => EsqlFunctions.CidrMatch( // -> WHERE CIDR_MATCH(ip, "10.0.0.0/8") p.IpAddress, "10.0.0.0/8"))

LOOKUP JOIN

Cross-index lookups translate to ES|QL LOOKUP JOIN:

var enriched = client.Esql.Query(q => q .LookupJoin( "category-lookup-index", product => product.Id, category => category.CategoryId, (product, category) => new { product.Name, category!.CategoryLabel }));

Raw ES|QL escape hatch

For ES|QL features not yet covered by the LINQ provider, you can append raw fragments:

var results = client.Esql.Query(q => q .Where(p => p.InStock) .RawEsql("| EVAL discounted = price_usd * 0.9"));

Server-side async queries

For long-running queries, submit them for background processing on the server:

await using var asyncQuery = await client.Esql.SubmitAsyncQueryAsync( q => q.Where(p => p.InStock), asyncQueryOptions: new EsqlAsyncQueryOptions { WaitForCompletionTimeout = TimeSpan.FromSeconds(5), KeepAlive = TimeSpan.FromMinutes(10) }); await asyncQuery.WaitForCompletionAsync(); await foreach (var product in asyncQuery.AsAsyncEnumerable()) Console.WriteLine(product.Name);

Server-side async queries are especially useful for long-running analytical queries / large dataset processing that might exceed typical timeout thresholds, or in timeout-sensitive environments with load balancers, API gateways, or proxies that enforce strict HTTP timeouts. Async queries avoid connection drops by decoupling submission from result retrieval.

Getting started

LINQ to ES|QL is available starting from:

Elastic.Clients.Elasticsearch v9.3.4 (9.x branch)
Elastic.Clients.Elasticsearch v8.19.18 (8.x branch)

Install from NuGet:

dotnet add package Elastic.Clients.Elasticsearch

The entry points are on client.Esql:

Method

Returns

Use case

Query(...)

IEnumerable

Synchronous execution

QueryAsync(...)

IAsyncEnumerable

Async streaming

CreateQuery()

IEsqlQueryable

Advanced composition and inspection

SubmitAsyncQueryAsync(...)

EsqlAsyncQuery

Long-running server-side queries

For the full feature reference, including query options, multifield access, nested objects, and multivalue field handling, see the LINQ to ES|QL documentation.

Conclusion

LINQ to ES|QL brings the full expressiveness of C# LINQ to Elasticsearch's ES|QL query language, letting you write strongly typed, composable queries without handcrafting query strings. With automatic parameter capturing, streaming materialization, and a layered package architecture that scales from stand-alone translation to the full Elasticsearch client, it fits naturally into .NET applications of any size. Install the latest client, point your LINQ expressions at an index, and let the provider handle the rest.

How to use Elasticsearch Vector Store Connector for Microsoft Semantic Kernel for AI Agent development

Florian Bernd,Srikanth Manvi — Fri, 06 Dec 2024 00:00:00 GMT

In collaboration with the Microsoft Semantic Kernel team, we are announcing the availability of Semantic Kernel Elasticsearch Vector Store Connector, for Microsoft Semantic Kernel (.NET) users. Semantic Kernel simplifies building enterprise-grade AI agents, including the capability to enhance large language models (LLMs) with more relevant, data-driven responses from a Vector Store. Semantic Kernel provides a seamless abstraction layer for interacting with Vector Stores like Elasticsearch, offering essential features such as creating, listing, and deleting collections of records and uploading, retrieving, deleting individual records.

The out-of-the-box Semantic Kernel Elasticsearch Vector Store Connector supports the Semantic Kernel vector store abstractions which make it very easy for developers to plugin Elasticsearch as a vector store while building AI agents.

Elasticsearch has a strong foundation in the open-source community and recently adopted the AGPL license. Combined with the open-source Microsoft Semantic Kernel, these tools offer a powerful, enterprise-ready solution. You can get started locally by spinning up Elasticsearch in a few minutes by running this command curl -fsSL https://elastic.co/start-local | sh (refer start-local for details) and move to cloud-hosted or self-hosted versions while productionizing your AI agents.

In this blog we look at how to use Semantic Kernel Elasticsearch Vector Store Connector when using Semantic Kernel. A Python version of the connector will be made available in the future.

High-level scenario: Building a RAG app with Semantic Kernel & Elasticsearch

In the following section we go through an example. At a high-level we are building a RAG (Retrieval Augmented Generation) application which takes a user's question as input and returns an answer. We will use Azure OpenAI (local LLM can be used as well) as the LLM, Elasticsearch as the vector store and Semantic Kernel (.net) as the framework to tie all components together.

If you are not familiar with RAG architectures, you can have a quick introduction with this article: https://www.elastic.co/search-labs/blog/retrieval-augmented-generation-rag.

The answer is generated by the LLM which is fed with context, relevant to the question, retrieved from Elasticsearch vectorstore. The response also includes the source that was used as the context by the LLM.

RAG example

In this specific example, we build an application that allows users to ask questions about hotels stored in an internal hotel database. The user could e.g. search for a specific hotel, based on different criteria, or ask for a list of hotels.

For the example database, we generated a list of hotels containing 100 entries. The sample size is intentionally small to allow you to try out the connector demo as easily as possible. In a real-world application, the Elasticsearch connector would show its advantages over other options, such as the `InMemory` vector store implementation, especially when working with extremely large amounts of data.

The complete demo application can be found in the Elasticsearch vector store connector repository.

Let’s start with adding the required NuGet packages and using directives to our project:

dotnet add package "Elastic.Clients.Elasticsearch" -v 8.16.2 dotnet add package "Elastic.SemanticKernel.Connectors.Elasticsearch" -v 0.1.2 dotnet add package "Microsoft.Extensions.Hosting" -v 9.0.0 dotnet add package "Microsoft.SemanticKernel.Connectors.AzureOpenAI" -v 1.30.0 dotnet add package "Microsoft.SemanticKernel.PromptTemplates.Handlebars" -v 1.30.0using System; using System.IO; using System.Linq; using System.Threading.Tasks; using Elastic.Clients.Elasticsearch; using Elastic.Transport; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.VectorData; using Microsoft.SemanticKernel; using Microsoft.SemanticKernel.Data; using Microsoft.SemanticKernel.Embeddings; using Microsoft.SemanticKernel.PromptTemplates.Handlebars;

We can now create our data model and provide it with Semantic Kernel specific attributes to define the storage model schema and some hints for the text search:

///

/// Data model for storing a "hotel" with a name, a description, a description embedding and an optional reference link. ///

public sealed record Hotel { [VectorStoreRecordKey] public required string HotelId { get; set; } [TextSearchResultName] [VectorStoreRecordData(IsFilterable = true)] public required string HotelName { get; set; } [TextSearchResultValue] [VectorStoreRecordData(IsFullTextSearchable = true)] public required string Description { get; set; } [VectorStoreRecordVector(Dimensions: 1536, DistanceFunction.CosineSimilarity, IndexKind.Hnsw)] public ReadOnlyMemory? DescriptionEmbedding { get; set; } [TextSearchResultLink] [VectorStoreRecordData] public string? ReferenceLink { get; set; } }

The Storage Model Schema attributes (`VectorStore*`) are most relevant for the actual use of the Elasticsearch Vector Store Connector, namely:

VectorStoreRecordKey to mark a property on a record class as the key under which the record is stored in a vector store.
VectorStoreRecordData to mark a property on a record class as 'data'.
VectorStoreRecordVector to mark a property on a record class as a vector.

All of these attributes accept various optional parameters that can be used to further customize the storage model. In the case of VectorStoreRecordKey , for example, it is possible to specify a different distance function or a different index type.

The text search attributes (TextSearch*) will be important in the last step of this example. We will come back to them later.

In the next step, we initialize the Semantic Kernel engine and obtain references to the core services. In a real world application, dependency injection should be used instead of directly accessing the service collection. The same thing applies to the hardcoded configuration and secrets, which should be read using a configuration provider instead:

var builder = Host.CreateApplicationBuilder(args); // Register AI services. var kernelBuilder = builder.Services.AddKernel(); kernelBuilder.AddAzureOpenAIChatCompletion("gpt-4o", "https://my-service.openai.azure.com", "my_token"); kernelBuilder.AddAzureOpenAITextEmbeddingGeneration("ada-002", "https://my-service.openai.azure.com", "my_token"); // Register text search service. kernelBuilder.AddVectorStoreTextSearch(); // Register Elasticsearch vector store. var elasticsearchClientSettings = new ElasticsearchClientSettings(new Uri("https://my-elasticsearch-instance.cloud")) .Authentication(new BasicAuthentication("elastic", "my_password")); kernelBuilder.AddElasticsearchVectorStoreRecordCollection("skhotels", elasticsearchClientSettings); // Build the host. using var host = builder.Build(); // For demo purposes, we access the services directly without using a DI context. var kernel = host.Services.GetService()!; var embeddings = host.Services.GetService()!; var vectorStoreCollection = host.Services.GetService>()!; // Register search plugin. var textSearch = host.Services.GetService>()!; kernel.Plugins.Add(textSearch.CreateWithGetTextSearchResults("SearchPlugin"));

The vectorStoreCollection service can now be used to create the collection and to ingest a few demo records:

await vectorStoreCollection.CreateCollectionIfNotExistsAsync(); // CSV format: ID;Hotel Name;Description;Reference Link var hotels = (await File.ReadAllLinesAsync("hotels.csv")) .Select(x => x.Split(';')); foreach (var chunk in hotels.Chunk(25)) { var descriptionEmbeddings = await embeddings.GenerateEmbeddingsAsync(chunk.Select(x => x[2]).ToArray()); for (var i = 0; i < chunk.Length; ++i) { var hotel = chunk[i]; await vectorStoreCollection.UpsertAsync(new Hotel { HotelId = hotel[0], HotelName = hotel[1], Description = hotel[2], DescriptionEmbedding = descriptionEmbeddings[i], ReferenceLink = hotel[3] }); } }

This shows how Semantic Kernel reduces the use of a vector store with all its complexity to a few simple method calls.

Under the hood, a new index is created in Elasticsearch and all the necessary property mappings are created. Our data set is then mapped completely transparently into the storage model and finally stored in the index. Below is how the mappings look in Elasticsearch.

{ "mappings": { "properties": { "descriptionEmbedding": { "dims": 1536, "index": true, "index_options": { "type": "hnsw" }, "similarity": "cosine", "type": "dense_vector" }, "hotelName": { "type": "keyword" }, "description": { "type": "text" } } } }

The embeddings.GenerateEmbeddingsAsync() calls transparently called the configured Azure AI Embeddings Generation service.

Even more magic can be observed in the last step of this demo.

With just a single call to InvokePromptAsync, all of the following operations are performed when the user asks a question about the data:

1. An embedding for the user's question is generated

2. The vector store is searched for relevant entries

3. The results of the query are inserted into a prompt template

4. The actual query in the form of the final prompt is sent to the AI chat completion service

// Invoke the LLM with a template that uses the search plugin to // 1. get related information to the user query from the vector store // 2. add the information to the LLM prompt. var response = await kernel.InvokePromptAsync( promptTemplate: """ Please use this information to answer the question: {{#with (SearchPlugin-GetTextSearchResults question)}} {{#each this}} Name: {{Name}} Value: {{Value}} Source: {{Link}} ----------------- {{/each}} {{/with}} Include the source of relevant information in the response. Question: {{question}} """, arguments: new KernelArguments { { "question", "Please show me all hotels that have a rooftop bar." }, }, templateFormat: "handlebars", promptTemplateFactory: new HandlebarsPromptTemplateFactory());

Remember the TextSearch* attributes, we previously defined on our data model? These attributes enable us to use corresponding placeholders in our prompt template which are automatically populated with the information from our entries in the vector store.

The final response to our question "Please show me all hotels that have a rooftop bar." is as follows:

Console.WriteLine(response.ToString()); // > The hotel that has a rooftop bar is Skyline Suites. You can find more information about this hotel [here](https://example.com/yz567).

The answer correctly refers to the following entry in our hotels.csv

9; Skyline Suites; Offering panoramic city views from every suite, this hotel is perfect for those who love the urban landscape. Enjoy luxurious amenities, a rooftop bar, and close proximity to attractions. Luxurious and contemporary.; https://example.com/yz567

This example shows very well how the use of Microsoft Semantic Kernel achieves a significant reduction in complexity through its well thought abstractions, as well as enabling a very high level of flexibility. By changing a single line of code, for example, the vector store or the AI services used can be replaced without having to refactor any other part of the code.

At the same time, the framework provides an enormous set of high-level functionality, such as the `InvokePrompt` function, or the template or search plugin system.

The complete demo application can be found in the Elasticsearch vector store connector repository.

What else is possible with Elasticsearch

Elasticsearch & Semantic Kernel: What's next?

We showed how the Elasticsearch vector store can be easily plugged into Semantic Kernel while building GenAI applications in .NET. Stay tuned for a Python integration next.
As Semantic Kernel builds abstractions for advanced search features like hybrid search, the Elasticsearch connect will enable .NET developers to easily implement them while using Semantic Kernel.