Tech Topics

Introducing Elasticsearch.Net and NEST 1.0.0-beta1

It’s been 4 months since NEST 0.12.0 was released and a lot has happened since then. Elasticsearch released 1.0 and even version 1.1. With this release announcement, I’m pleased to say NEST has finally caught up and brings (almost) all the awesome that Elasticsearch 1.0 and 1.1 bring to the .NET world. Even more exciting – as of today, Elasticsearch.Net and NEST are both officially supported .Net clients for Elasticsearch!

(For those just learning about Elasticsearch.Net and NEST, Elasticsearch.Net is the low level client and NEST is the high level client for users of .Net and Elasticsearch.)

Before diving in all the technical details of the NEST 1.0 beta release, I would first like to express a sincere thank you to all of you who’ve reached out on Twitter, GitHub issues and via email. Your kind words made all the difference these past 4 months as I busily switched jobs, refactored the client and implemented 1.0/1.1 features. I am happy to say that I have formally joined Elasticsearch Inc and will now be fully dedicated to making Elasticsearch in the .NET world awesome!

So, what does ‘beta’ mean? All tests are passing, but we are still waiting for at least one new feature to land. Additionally, beta means that some new features may change before general release. Most importantly, this beta period is intended to solicit as much feedback as possible from everyone on breaking changes/oversights/bugs.

Breaking Changes

With this 1.0 release, we will have many breaking changes. The 0.* NEST releases have been around since 2010 and the internals were showing their age. This 1.0 release represents an almost complete refactoring of NEST.

A separate section that lists the breaking changes can be found here. If you find any breaking changes that are not documented, we’d love to hear about them in elasticsearch-net’s github issues.

Not one but two clients

NEST 0.12.0 introduced a separate, completely generated, client interface called IRawElasticClient which allowed you to build your own requests and responses without having to worry about building endpoints.
Much of the work for this 1.0 beta release has been around refactoring this functionality out of the NEST assembly and making NEST’s strongly typed IElasticClient use IRawElasticClient internally.

Elasticsearch.Net

Elasticsearch.Net’s client is now called IElasticsearchClient and is almost completely generated from the client API specification, exposing all the Elasticsearch 1.1 endpoints. It brings a client that’s well aligned in spirit and architecture with the other official Elasticsearch client libraries.

Elasticsearch.Net leaves all the request and response mapping up to you although it comes with support for mapping responses into a dynamic container (a slightly modified DynamicDictionary from NancyFx) out of the box.

Elasticsearch.Net is very much intended to be used by other high level clients.

Another big new feature is built-in cluster failover/connection pooling support support. This allows the client to retry failed connections on different nodes and sniff the cluster for live nodes when a node fails or at certain configurable intervals.

Elasticsearch is elastic, and the client should be, too.

Elasticsearch 1.0.0-beta is a prerelease package so make sure you adjust your filter inside the package manager:

or pass the -PreRelease flag to nuget from the console:

PM> Install-Package Elasticsearch.Net -PreRelease

Please read the new documentation for Elasticsearch.Net to find out more.

NEST

NEST’s client classes have completely been rewritten and internally use and still expose the Elasticsearch.Net client. It benefits from it’s cluster failover support and it no longer has complicated internal path builders.

NEST is highly opinionated on how you should build and consume Elasticsearch responses, and comes with a strongly typed Query DSL. NEST has really high coverage of mapped endpoints, and the unmapped API’s are all identified and documented on the roadmap.

PM> Install-Package NEST -PreRelease

Changelog

This release consists of 300+ commits over a 4 month timespan so I will just list the highlights:

Aggregations

All of Elasticsearch’s (1.0 and 1.1!) aggregations have been mapped in NEST 1.0.

var results = this._client.Search<ElasticsearchProject>(s=>s
    .Size(0)
    .Aggregations(a=>a
        .Nested("contributors", n=>n
            .Path(p=>p.Contributors)
            .Aggregations(t=>t
                .Average("avg_age", m=>m
                    .Field(p=>p.Contributors.First().Age)
                )
            )
        )
    )
);

var bucket = results.Aggs.Nested("contributors");
var averageAge = bucket.Average("avg_age");

Snapshot and Restore

You can now script your create/delete/get repositories/snapshotting/restoring with NEST.

New query/filter types

The common terms query and simple_query_string have all been mapped, along with other new query and filter types.

Lots of new API’s are mapped

Count percolations, clearing scrolls, external versioning, suggests and more are now all mapped in NEST. The goal is to get as close to a 100% API coverage as possible before the final 1.0 release goes out. Just to reiterate: Elasticsearch.Net, the low level client, already has 100% API coverage but building and handling the responses is out of its scope.

Conditionless() query construct

NEST comes with a powerful feature called conditionless queries which greatly simplifies writing queries with optional parts.

Consider this example:

.Query(q=>q.Term("this_term_is_conditionless", ""))

NEST will remove the term query and default to not sending a query at all in the body (equivalent to match_all)

But what if you want a different query to be run in instead of match_all? This is where the Conditionless() construct comes into play:

.Query(q=>q
    .Conditionless(qs=>qs
        .Query(qcq=>qcq.Term("this_term_is_conditionless", ""))
        .Fallback(qcf=>qcf.Term("name", "do_me_instead")
    )
)

Now we’ll fallback to a term query on name instead. Note that fallbacks themselves can be conditionless and Conditionless() constructs can be nested inside fallbacks as well.

Also if you want to disable conditionless queries you can specify .Strict() on individual parts in the query or globally on the search descriptor.

.Query(q=>q.Strict().Term("this_term_is_conditionless", ""))

The previous example will throw an exception because the term misses a value. To enable conditionless only for certain parts in your query, you can also do:

.Strict()
.Query(q=>
    q.Term("must", "exist") 
    && q.Strict(false).Term("this_term_is_conditionless", "")
)

The previous example will just send a term query on the field must, and NEST is smart enough to infer the bool is no longer needed.

Finally, if you really intend to send a conditionless query to Elasticsearch you can use .Verbatim()

.Query(q=>q.Verbatim().Term("this_term_is_conditionless", ""))

The Strict() and Verbatim() constructs are not new but worth repeating in this context.

Community

As with every release the level of community engagement never ceases to amaze me. Whoever proclaimed open source in .Net is dead?

A huge thank you goes out to the folks who submitted the following pull requests.

  • #412 Add random_score support for FunctionScore(). ty @vovikdrg!
  • #419 Add script score support for FunctionScore(). ty @V1tOr!
  • #419 Added support for index status API. ty @richclement!
  • #430 Allow any object to be used for a partial update. ty @Plasma!
  • #431 Add Type as constructor parameter to CustomAnalyzer. ty @sschlesier!
  • #440 Support dictionary key evaluation in expressions. ty @azubanov!
  • #444 Renamed QueryString() to Query() MatchQueryDescriptor. ty @gmarz!
  • #445 Renamed QueryString() to Query() MultiMatchQueryDescriptor. ty @gmarz!
  • #446 Renamed QueryString() to Query() TextQueryDescriptor. ty @gmarz!
  • #447 Added Params() support to script on RangeFacetDescriptor. ty @angielamb!
  • #448 Added filter option in function_score queries. ty @gmarz!
  • #479 Added detail response mapped to _explain. ty @andreabalducci!
  • #488 Added nested field attribute mapped back in. ty @lukapor!
  • #490 Added size parameter to CompletionSuggestDescriptor. ty @bgiromini!
  • #540 Added missing sort options. ty @Grastveit!
  • #536 Worlds smallest pull request (typo). ty @ChrisMcKee!
  • #541 FunctionScoreQueryDescriptor boost mode. ty @azubanov!
  • #542 Allow fields boosting when using expressions in multimatch query. ty @azubanov!
  • #545 Show requests/response as json string in debug mode when available. ty @azubanov!
  • #548 Create cache directory for code generation if it does not exist. ty @gmarz!
  • #550 Fixed typos, formatting, verbiage, naming conventions in the documentation. ty @paigecook!
  • #563 Added support for new text query types on the multi_match query. ty @azubanov!
  • #565 Thrift connection does not clean up properly after going into faulted state. ty @danp60!

In addition a huge thank you goes out to all the folks who gave continuous feedback on the state of master while it underwent major refactoring.

A special shout out to @synhershko and @icanhasjonas for opening #522 and #537

Seeing tweets like this is awesome, thank you tonariman & luigiberrettini!

Moving forward

The goal is to come to a final Elasticsearch.Net and NEST 1.0 release as soon as possible and with your help we can!

Please don’t forget to post your issues and feedback in elasticsearch-net’s GitHub issues!