Introducing Elasticsearch.Net and NEST 1.0.0-beta1
It’s been 4 months since NEST 0.12.0 was released and a lot has happened since then. Elasticsearch released 1.0 and even version 1.1. With this release announcement, I’m pleased to say NEST has finally caught up and brings (almost) all the awesome that Elasticsearch 1.0 and 1.1 bring to the .NET world. Even more exciting – as of today, Elasticsearch.Net and NEST are both officially supported .Net clients for Elasticsearch!
(For those just learning about Elasticsearch.Net and NEST, Elasticsearch.Net is the low level client and NEST is the high level client for users of .Net and Elasticsearch.)
Before diving in all the technical details of the NEST 1.0 beta release, I would first like to express a sincere thank you to all of you who’ve reached out on Twitter, GitHub issues and via email. Your kind words made all the difference these past 4 months as I busily switched jobs, refactored the client and implemented 1.0/1.1 features. I am happy to say that I have formally joined Elasticsearch Inc and will now be fully dedicated to making Elasticsearch in the .NET world awesome!
So, what does ‘beta’ mean? All tests are passing, but we are still waiting for at least one new feature to land. Additionally, beta means that some new features may change before general release. Most importantly, this beta period is intended to solicit as much feedback as possible from everyone on breaking changes/oversights/bugs.
With this 1.0 release, we will have many breaking changes. The
0.* NEST releases have been around since 2010 and the internals were showing their age. This 1.0 release represents an almost complete refactoring of NEST.
Not one but two clients
NEST 0.12.0 introduced a separate, completely generated, client interface called
IRawElasticClient which allowed you to build your own requests and responses without having to worry about building endpoints.
Much of the work for this 1.0 beta release has been around refactoring this functionality out of the NEST assembly and making NEST’s strongly typed
Elasticsearch.Net’s client is now called
IElasticsearchClient and is almost completely generated from the client API specification, exposing all the Elasticsearch 1.1 endpoints. It brings a client that’s well aligned in spirit and architecture with the other official Elasticsearch client libraries.
Elasticsearch.Net leaves all the request and response mapping up to you although it comes with support for mapping responses into a dynamic container (a slightly modified DynamicDictionary from NancyFx) out of the box.
Elasticsearch.Net is very much intended to be used by other high level clients.
Another big new feature is built-in cluster failover/connection pooling support support. This allows the client to retry failed connections on different nodes and sniff the cluster for live nodes when a node fails or at certain configurable intervals.
Elasticsearch is elastic, and the client should be, too.
Elasticsearch 1.0.0-beta is a prerelease package so make sure you adjust your filter inside the package manager:
or pass the
-PreRelease flag to nuget from the console:
PM> Install-Package Elasticsearch.Net -PreRelease
Please read the new documentation for Elasticsearch.Net to find out more.
NEST’s client classes have completely been rewritten and internally use and still expose the Elasticsearch.Net client. It benefits from it’s cluster failover support and it no longer has complicated internal path builders.
NEST is highly opinionated on how you should build and consume Elasticsearch responses, and comes with a strongly typed Query DSL. NEST has really high coverage of mapped endpoints, and the unmapped API’s are all identified and documented on the roadmap.
PM> Install-Package NEST -PreRelease
This release consists of 300+ commits over a 4 month timespan so I will just list the highlights:
All of Elasticsearch’s (1.0 and 1.1!) aggregations have been mapped in NEST 1.0.
var results = this._client.Search<ElasticsearchProject>(s=>s .Size(0) .Aggregations(a=>a .Nested("contributors", n=>n .Path(p=>p.Contributors) .Aggregations(t=>t .Average("avg_age", m=>m .Field(p=>p.Contributors.First().Age) ) ) ) ) ); var bucket = results.Aggs.Nested("contributors"); var averageAge = bucket.Average("avg_age");
Snapshot and Restore
You can now script your create/delete/get repositories/snapshotting/restoring with NEST.
New query/filter types
The common terms query and
simple_query_string have all been mapped, along with other new query and filter types.
Lots of new API’s are mapped
Count percolations, clearing scrolls, external versioning, suggests and more are now all mapped in NEST. The goal is to get as close to a 100% API coverage as possible before the final 1.0 release goes out. Just to reiterate:
Elasticsearch.Net, the low level client, already has 100% API coverage but building and handling the responses is out of its scope.
Conditionless() query construct
NEST comes with a powerful feature called conditionless queries which greatly simplifies writing queries with optional parts.
Consider this example:
NEST will remove the term query and default to not sending a query at all in the body (equivalent to
But what if you want a different query to be run in instead of
match_all? This is where the
Conditionless() construct comes into play:
.Query(q=>q .Conditionless(qs=>qs .Query(qcq=>qcq.Term("this_term_is_conditionless", "")) .Fallback(qcf=>qcf.Term("name", "do_me_instead") ) )
Now we’ll fallback to a
term query on
name instead. Note that fallbacks themselves can be conditionless and
Conditionless() constructs can be nested inside fallbacks as well.
Also if you want to disable conditionless queries you can specify
.Strict() on individual parts in the query or globally on the search descriptor.
The previous example will throw an exception because the term misses a value. To enable conditionless only for certain parts in your query, you can also do:
.Strict() .Query(q=> q.Term("must", "exist") && q.Strict(false).Term("this_term_is_conditionless", "") )
The previous example will just send a
term query on the field
must, and NEST is smart enough to infer the bool is no longer needed.
Finally, if you really intend to send a conditionless query to Elasticsearch you can use
Verbatim() constructs are not new but worth repeating in this context.
As with every release the level of community engagement never ceases to amaze me. Whoever proclaimed open source in .Net is dead?
A huge thank you goes out to the folks who submitted the following pull requests.
- #412 Add
random_scoresupport for FunctionScore(). ty @vovikdrg!
- #419 Add script score support for FunctionScore(). ty @V1tOr!
- #419 Added support for index status API. ty @richclement!
- #430 Allow any object to be used for a partial update. ty @Plasma!
- #431 Add Type as constructor parameter to CustomAnalyzer. ty @sschlesier!
- #440 Support dictionary key evaluation in expressions. ty @azubanov!
- #444 Renamed QueryString() to Query() MatchQueryDescriptor. ty @gmarz!
- #445 Renamed QueryString() to Query() MultiMatchQueryDescriptor. ty @gmarz!
- #446 Renamed QueryString() to Query() TextQueryDescriptor. ty @gmarz!
- #447 Added Params() support to script on RangeFacetDescriptor. ty @angielamb!
- #448 Added filter option in
function_scorequeries. ty @gmarz!
- #479 Added detail response mapped to
_explain. ty @andreabalducci!
- #488 Added nested field attribute mapped back in. ty @lukapor!
- #490 Added size parameter to CompletionSuggestDescriptor. ty @bgiromini!
- #540 Added missing sort options. ty @Grastveit!
- #536 Worlds smallest pull request (typo). ty @ChrisMcKee!
- #541 FunctionScoreQueryDescriptor boost mode. ty @azubanov!
- #542 Allow fields boosting when using expressions in multimatch query. ty @azubanov!
- #545 Show requests/response as json string in debug mode when available. ty @azubanov!
- #548 Create cache directory for code generation if it does not exist. ty @gmarz!
- #550 Fixed typos, formatting, verbiage, naming conventions in the documentation. ty @paigecook!
- #563 Added support for new text query types on the
multi_matchquery. ty @azubanov!
- #565 Thrift connection does not clean up properly after going into faulted state. ty @danp60!
In addition a huge thank you goes out to all the folks who gave continuous feedback on the state of
master while it underwent major refactoring.
The goal is to come to a final Elasticsearch.Net and NEST 1.0 release as soon as possible and with your help we can!
Please don’t forget to post your issues and feedback in elasticsearch-net’s GitHub issues!