NEST and Elasticsearch.Net 7.0 now GA
After many months of work, two alphas and a beta, we are pleased to announce the GA release of the NEST and Elasticsearch.Net 7.0 clients.
The overall themes of this release have been based around faster serialization, performance improvements, codebase simplification, and ensuring parity with the many new features available in Elasticsearch 7.0.
Types removal
Specifying types within the .NET clients is now deprecated in 7.0, in line with the overall Elasticsearch type removal strategy.
In instances where your index contains type information and you need to preserve this information, one recommendation is to introduce a property to describe the document type (similar to a table per class with discriminator field in the ORM world) and then implement a custom serialization / deserialization implementation for that class.
This Elasticsearch page details some other approaches.
Faster serialization
After internalizing the serialization routines, and IL-merging the Newtonsoft.Json package in 6.x, we are pleased to announce that the next stage of serialization improvements have been completed in 7.0.
Both SimpleJson and Newtonsoft.Json have been completely removed and replaced with an implementation of Utf8Json, a fast serializer that works directly with UTF-8 binary. This has yielded a significant performance improvement, which we will be sharing in more detail in a later blog post.
With the move to Utf8Json, we have removed some features that were available in the previous JSON libraries that have proven too onerous to carry forward at this stage.
- JSON in the request is never indented, even if
SerializationFormatting.Indentedis specified. The serialization routines generated by Utf8Json never generate anIJsonFormatter<T>that will indent JSON, for performance reasons. We are considering options for exposing indented JSON for development and debugging purposes. - NEST types cannot be extended by inheritance. With NEST 6.x, additional properties can be included for a type by deriving from that type and annotating these new properties. With the current implementation of serialization with Utf8Json, this approach will not work.
- Serializer uses
Reflection.Emit. Utf8Json usesReflection.Emitto generate efficient formatters for serializing types that it sees.Reflection.Emitis not supported on all platforms, for example, UWP, Xamarin.iOS, and Xamarin.Android. Elasticsearch.Net.DynamicResponsedeserializes JSON arrays toList<object>. SimpleJson deserialized JSON arrays toobject[], but Utf8Json deserializes them toList<object>. This change is preferred for allocation and performance reasons.- Utf8Json is much stricter when deserializing JSON object field names to C# POCO properties. With the internal Json.NET serializer in 6.x, JSON object field names would attempt to be matched with C# POCO property names first by an exact match, falling back to a case insensitive match. With Utf8Json in 7.x however, JSON object field names must match exactly the name configured for the C# POCO property name.
We believe that the trade-off of features vs. GA release has been worthwhile at this stage. We hold a view to address some of these missing features in a later release.
High- to low-level client dispatch changes
In 6.x, the process of an API call within NEST looked roughly like this
client.Search()
=> Dispatch()
=> LowLevelDispatch.SearchDispatch()
=> lowlevelClient.Search()
=> lowlevelClient.DoRequest()
With 7.x, this process has been changed to remove dispatching to the low-level client methods. The new process looks like this
client.Search()
=> lowlevelClient.DoRequest()
This means that in the high-level client IRequest now builds its own URLs, with the upside that the call chain is shorter and allocates fewer closures. The downside is that there are now two URL building mechanisms, one in the low-level client and a new one in the high-level client. In practice, this area of the codebase is kept up to date via code generation, so it does not place any additional burden on development.
Given the simplified call chain and debugging experience, we believe this is an improvement worth making.
Namespaced API methods and Upgrade Assistant
As the API surface of Elasticsearch has grown to well over 200 endpoints, so has the number of client methods exposed, leading to an almost overwhelming number to navigate and explore through in an IDE. This is further exacerbated by the fact that the .NET client exposes both synchronous and asynchronous API methods for both the fluent API syntax as well as the object initializer syntax.
To address this, the APIs are now accessible through sub-properties on the client instance.
For example, in 6.x, to create a machine learning job
var putJobResponse = client.PutJob<Metric>("id", c => c
.Description("Lab 1 - Simple example")
.ResultsIndexName("server-metrics")
.AnalysisConfig(a => a
.BucketSpan("30m")
.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
)
.DataDescription(d => d.TimeField(r => r.Timestamp))
);
This has changed to the following in 7.0
var putJobResponse = client.MachineLearning.PutJob<Metric>("id", c => c
.Description("Lab 1 - Simple example")
.ResultsIndexName("server-metrics")
.AnalysisConfig(a => a
.BucketSpan("30m")
.Detectors(d => d.Sum(c => c.FieldName(r => r.Total)))
)
.DataDescription(d => d.TimeField(r => r.Timestamp))
);
Notice the client.MachineLearning.PutJob method call in 7.0, as opposed to client.PutJob in 6.x.
We believe this grouping of functionality leads to a better discoverability experience when writing your code, and improved readability when reviewing somebody else's.
The Upgrade Assistant
To assist developers in migrating from 6.x, we have published the Nest.7xUpgradeAssistant Nuget package. When included in your project and the using Nest.ElasticClientExtensions; directive is added, calls will be redirected from the old API method names to the new API method names in 7.0. The result is that your project will compile and you won't need to immediately update your code to use the namespaced methods; instead you'll see compiler warnings indicating the location of the new API methods in 7.0.
This package is to assist developers migrating from 6.x to 7.0 and is limited in scope to this purpose. It is recommended that you observe the compiler warnings and adjust your code as indicated.
Observability and DiagnosticSource
7.0 introduces emitting System.Diagnostics.DiagnosticSource information from the client, during a request. The goal is to enable rich information exchange with the Elastic APM .NET agent and other monitoring libraries.
We emit DiagnosticSource information for key parts of a client request via an Activity event, shipping support for Id, ParentId, RootId, as well as request Duration.
To facilitate wiring this up to DiagnosticListener.AllListeners, we ship both with static access to the publisher names and events through Elasticsearch.Net.Diagnostics.DiagnosticSources as well as strongly typed listeners, removing the need to cast the object passed to activity start/stop events.
An example listener implementation that writes events to the console is given below
private class ListenerObserver : IObserver<DiagnosticListener>
{
public void OnCompleted() => Console.WriteLine("Completed");
public void OnError(Exception error) => Console.Error.WriteLine(error.Message);
public void OnNext(DiagnosticListener value)
{
void WriteToConsole<T>(string eventName, T data)
{
var a = Activity.Current;
Console.WriteLine($"{eventName?.PadRight(30)} {a.Id?.PadRight(32)} {a.ParentId?.PadRight(32)} {data?.ToString().PadRight(10)}");
}
if (value.Name == DiagnosticSources.AuditTrailEvents.SourceName)
value.Subscribe(new AuditDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
if (value.Name == DiagnosticSources.RequestPipeline.SourceName)
value.Subscribe(new RequestPipelineDiagnosticObserver(
v => WriteToConsole(v.Key, v.Value),
v => WriteToConsole(v.Key, v.Value))
);
if (value.Name == DiagnosticSources.HttpConnection.SourceName)
value.Subscribe(new HttpConnectionDiagnosticObserver(
v => WriteToConsole(v.Key, v.Value),
v => WriteToConsole(v.Key, v.Value)
));
if (value.Name == DiagnosticSources.Serializer.SourceName)
value.Subscribe(new SerializerDiagnosticObserver(v => WriteToConsole(v.Key, v.Value)));
}
Using the following example
var pool = new SniffingConnectionPool(new[] { node.NodesUris().First() });
var settings = new ConnectionSettings(pool).SniffOnStartup();
var client = new ElasticClient(settings);
var searchResponse = client.Search<object>(s => s.AllIndices());
Console.WriteLine(new string('-', Console.WindowWidth - 1));
searchResponse = client.Search<object>(s => s.Index("does-not-exist"));
emits the following console output
SniffOnStartup.Start |59e275e-4f9c835d189eb14a. Event: SniffOnStartup
Sniff.Start |59e275e-4f9c835d189eb14a.1. |59e275e-4f9c835d189eb14a. GET _nodes/http,settings
Sniff.Start |59e275e-4f9c835d189eb14a.1.1. |59e275e-4f9c835d189eb14a.1. GET _nodes/http,settings
SendAndReceiveHeaders.Start |59e275e-4f9c835d189eb14a.1.1.1. |59e275e-4f9c835d189eb14a.1.1. GET _nodes/http,settings
SendAndReceiveHeaders.Stop |59e275e-4f9c835d189eb14a.1.1.1. |59e275e-4f9c835d189eb14a.1.1. 200
ReceiveBody.Start |59e275e-4f9c835d189eb14a.1.1.2. |59e275e-4f9c835d189eb14a.1.1. GET _nodes/http,settings
Deserialize.Start |59e275e-4f9c835d189eb14a.1.1.2.1. |59e275e-4f9c835d189eb14a.1.1.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop |59e275e-4f9c835d189eb14a.1.1.2.1. |59e275e-4f9c835d189eb14a.1.1.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop |59e275e-4f9c835d189eb14a.1.1.2. |59e275e-4f9c835d189eb14a.1.1. 200
Sniff.Stop |59e275e-4f9c835d189eb14a.1.1. |59e275e-4f9c835d189eb14a.1. GET _nodes/http,settings
Sniff.Stop |59e275e-4f9c835d189eb14a.1. |59e275e-4f9c835d189eb14a. Successful low level call on GET: /_nodes/http,settings?timeout=2s&flat_settings=true
SniffOnStartup.Stop |59e275e-4f9c835d189eb14a. Event: SniffOnStartup Took: 00:00:00.1872459
Ping.Start |59e275f-4f9c835d189eb14a. HEAD /
SendAndReceiveHeaders.Start |59e275f-4f9c835d189eb14a.1. |59e275f-4f9c835d189eb14a. HEAD /
SendAndReceiveHeaders.Stop |59e275f-4f9c835d189eb14a.1. |59e275f-4f9c835d189eb14a. 200
ReceiveBody.Start |59e275f-4f9c835d189eb14a.2. |59e275f-4f9c835d189eb14a. HEAD /
ReceiveBody.Stop |59e275f-4f9c835d189eb14a.2. |59e275f-4f9c835d189eb14a. 200
Ping.Stop |59e275f-4f9c835d189eb14a. Successful low level call on HEAD: /
CallElasticsearch.Start |59e2760-4f9c835d189eb14a. POST _all/_search
SendAndReceiveHeaders.Start |59e2760-4f9c835d189eb14a.1. |59e2760-4f9c835d189eb14a. POST _all/_search
Serialize.Start |59e2760-4f9c835d189eb14a.1.1. |59e2760-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
Serialize.Stop |59e2760-4f9c835d189eb14a.1.1. |59e2760-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
SendAndReceiveHeaders.Stop |59e2760-4f9c835d189eb14a.1. |59e2760-4f9c835d189eb14a. 200
ReceiveBody.Start |59e2760-4f9c835d189eb14a.2. |59e2760-4f9c835d189eb14a. POST _all/_search
Deserialize.Start |59e2760-4f9c835d189eb14a.2.1. |59e2760-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop |59e2760-4f9c835d189eb14a.2.1. |59e2760-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop |59e2760-4f9c835d189eb14a.2. |59e2760-4f9c835d189eb14a. 200
CallElasticsearch.Stop |59e2760-4f9c835d189eb14a. Successful low level call on POST: /_all/_search?typed_keys=true
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CallElasticsearch.Start |59e2761-4f9c835d189eb14a. POST does-not-exist/_search
SendAndReceiveHeaders.Start |59e2761-4f9c835d189eb14a.1. |59e2761-4f9c835d189eb14a. POST does-not-exist/_search
Serialize.Start |59e2761-4f9c835d189eb14a.1.1. |59e2761-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
Serialize.Stop |59e2761-4f9c835d189eb14a.1.1. |59e2761-4f9c835d189eb14a.1. request/response: Nest.DefaultHighLevelSerializer
SendAndReceiveHeaders.Stop |59e2761-4f9c835d189eb14a.1. |59e2761-4f9c835d189eb14a. 404
ReceiveBody.Start |59e2761-4f9c835d189eb14a.2. |59e2761-4f9c835d189eb14a. POST does-not-exist/_search
Deserialize.Start |59e2761-4f9c835d189eb14a.2.1. |59e2761-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
Deserialize.Stop |59e2761-4f9c835d189eb14a.2.1. |59e2761-4f9c835d189eb14a.2. request/response: Nest.DefaultHighLevelSerializer
ReceiveBody.Stop |59e2761-4f9c835d189eb14a.2. |59e2761-4f9c835d189eb14a. 404
CallElasticsearch.Stop |59e2761-4f9c835d189eb14a. Unsuccessful low level call on POST: /does-not-exist/_search?typed_keys=true
Response interfaces removed
Most API methods now return classes and not interfaces; for example, the client method client.Cat.Help now returns a CatResponse<CatAliasesRecord> as opposed to an interface named ICatResponse<CatAliasesRecord>.
In instances where methods can benefit from returning an interface, these have been left intact, for example, ISearchResponse<T>.
So why make the change?
Firstly, this significantly reduces the number of types in the library, reducing the overall download size, improving assembly load times and eventually the execution.
Secondly, it removes the need for us to manage the conversion of a Task<Response> to Task<IResponse>, a somewhat awkward part of the request pipeline.
The downside is that it does make it somewhat more difficult to create mocks / stubs of responses in the client.
After lengthy discussion we decided that users can achieve a similar result using a JSON string and the InMemoryConnection available in Elasticsearch.Net. We use this technique extensively in the Tests.Reproduce project.
Another alternative would be to introduce an intermediate layer in your application, and conceal the client calls and objects within that layer so they can be mocked.
Response.IsValid semantics
IApiCallDetails.Success and ResponseBase.IsValid have been simplified, making it easier to inspect if a request to Elasticsearch was indeed successful or not.
Low-level client
If the status code from Elasticsearch is 2xx then .Success will be true. In instances where a 404 status code is received, for example if a GET request results in a missing document, then .Success will be false. This is also the case for HEAD requests that result in a 404.
This is controlled via IConnectionConfiguration.StatusCodeToResponseSuccess, which currently has no public setter.
High-level client
The NEST high-level client overrides StatusCodeToResponseSuccess, whereby 404 status codes now sets .Success as true.
The reasoning here is that because NEST is in full control of URL and path building the only instances where a 404 is received is in the case of a missing document, never from a missing endpoint.
However, in the case of a 404 the ResponseBase.IsValid property will be false.
It has the nice side effect that if you set .ThrowExceptions() and perform an action on an entity that does not exist it won't throw as .ThrowExceptions() only inspects .Success on ApiCallDetails.
In closing
Give the new Elasticsearch.Net and NEST clients a try in your own cluster, or spin up a 14-day free trial of the Elasticsearch Service on Elastic Cloud. And if you run into any problems or have any questions, reach out on the Discuss forums.
For the full documentation of indexing using the NEST Elasticsearch.Net client, please refer to our docs.