In 7.11, we’re excited to announce schema on read in the Elastic Stack. We now offer the best of both worlds on a single platform. Schema on read for flexibility and schema on write for performance. We call our implementation runtime fields.

<p>In 7.11, we’re excited to announce support for schema on read in the Elastic Stack. We now offer the best of both worlds on a single platform — the performance and scale of the existing schema on write mechanism that our users love and depend on, coupled with a new level of flexibility for defining and executing queries with schema on read. We call our implementation of schema on read <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime.html">runtime fields</a>.
</p><p>Runtime fields enable you to create and query fields that are evaluated only at query time. Instead of indexing all the fields in your data as it’s ingested, you can pick and choose which fields are indexed and which ones are calculated only at runtime as you execute your queries. Runtime fields support new use cases and you won’t have to reindex any of your data. Adapt to changing log file formats, query new fields that were not indexed at ingest time, fix errors in index mappings, or iterate on the preparation and perfection of indices over time.
</p> <strong><h2>Schema on write or schema on read — why not both?</h2></strong><p>Schema on write remains the default way that Elasticsearch handles incoming data. All fields in a document are indexed as it’s ingested. This is what makes running searches in Elastic so fast, regardless of the volume of data returned or the number of queries executed. It’s also a big part of what our users love about Elastic.
</p><p>Schema on write works really well if you know your data before it is ingested. That way, the schema can be fully defined in the index mapping. It also requires sticking to that defined schema when queries are run against the index. However, in the real world data can often change. New data sources may arrive, for example. An added layer of flexibility to dynamically extract or query new fields after the data has been indexed adds tremendous value, even if it comes at a slight cost to performance.&nbsp;
</p><p>That’s where schema on read comes in. Data can be ingested in raw form without any indexing, except for certain necessary fields such as timestamp or response code. Other fields can be created on the fly when queries are run against the data. You don’t need to have intimate knowledge of your data ahead of time, nor do you have to predict all the possible ways that the data may eventually be queried. You can change the data structure at any time, even after the documents have been indexed.
</p><p>Here’s what’s unique about how we’ve done it. We’ve built runtime fields on the same Elasticsearch platform — the same architecture, the same tools, and the same interfaces you are already using. There are no new datastores, languages, or components, and there’s no additional procedural overhead. Schema on read and schema on write work together and seamlessly complement each other, so that you can decide which fields to calculate when a query requires them and which fields to index when the document is ingested into Elasticsearch.&nbsp;
</p><p>By offering you the best of both worlds on a single stack, we make it easy for you to decide which combination of schema on write and schema on read works best for your use cases.&nbsp;
</p> <strong><h2>More ways to get value out of your data</h2></strong><p>Define fields at runtime and use them right away. Some of the most useful scenarios are outlined below.
</p> <strong><h3>Add new fields to indexed documents<strong> </strong></h3></strong><p>The ability to add fields to documents is a common requirement for addressing a new need that did not exist when the documents were ingested and indexed. It could be the addition of an entirely new field, or just a field that needs to be calculated (or extracted and transformed) from other fields. For example, the date from the message field may need to be transformed to show the day of the week, or the IP address may need to be extracted from the client.address field so that it can be used in a map visualization. Either way, it’s significantly faster to just add a runtime field to address this than to update the index mapping and reindex the documents.
</p> <strong><h3>Use ephemeral fields</h3></strong><p>Analysts can define ephemeral fields that only exist within the context of a query or visualization, allowing them to work in their own confines without affecting the document index and without overriding the work of others on the same data. Empowering users to autonomously adjust the data for their needs leads to lower costs and higher productivity, cutting out the need to request admin users to make changes to the index or wait for the requests to be approved and completed.
</p> <strong><h3>Fix errors in the index</h3></strong><p>Indexed fields can be shadowed with runtime fields to immediately fix errors that were made when the documents were ingested. This capability makes those indexed documents more usable and eliminates the need for lengthy QA cycles, thus reducing costs and facilitating the rapid loading of documents.
</p> <strong><h3>Iterate during data preparation</h3></strong><p>As new users set up their data sources and begin ingesting data, they may not have a complete picture of how to define or perfect their index mappings. Runtime fields give them the leverage to not have to index all fields ahead of time, and runtime fields can be added gradually based on need. Over time, users can then observe the sizes of their indices, as well as the frequency and performance of different kinds of queries. During peak hours, if the performance of the most frequently run queries is falling behind optimal levels, some of the runtime fields that appear in those queries can be converted to indexed fields. This workflow makes introducing new data sources much faster, reduces the size of indices, increases ingestion speed, and lowers costs.
</p> <strong><h2>7.11: Runtime fields in beta</h2></strong><p>In 7.11, runtime fields are available in beta. You can define a runtime field in an index mapping alongside the indexed fields. The runtime field definition requires nothing more than a datatype and a painless script that specifies how to evaluate the field at query time. Regex can be used in the painless script. You can also define a runtime field without a script, and the system will use the value of the _source field that has the same name as its value. You can then use the Elasticsearch API to query those runtime fields in the same way that you query the indexed fields via Query DSL. You can further define a runtime field on the fly in the query itself to operate within the confines of that query.
</p><p>In Kibana, in 7.11 runtime fields are exposed through the Kibana Query Language (KQL) bar. You can run queries or filters in KQL on runtime fields as you would on indexed fields.
</p> <strong><h2>What’s next for runtime fields</h2></strong><p>While 7.11 provides the fundamentals to get you going with runtime fields, we’re building a number of features to provide you with a comprehensive experience. Going forward, runtime fields will be fully exposed in Kibana for search and exploration in both Discover and Lens. Runtime fields defined in index mappings will be visible alongside the indexed fields to create index patterns, making it easy for you to build your visualizations from any combination of the two.
</p><p>Kibana will also expose a UI editor to define and test runtime fields directly in your query interface, and this editor will be available in the various Kibana apps. This will make it easy for analysts to create runtime fields on the fly by themselves, operating only within the context of the queries and visualizations they are running at any point in time.&nbsp;
</p><p>Beyond that, we’ll add the ability to override indexed fields with runtime fields and convert runtime fields back to indexed fields. This will greatly help the use cases that focus on data preparation and perfecting indices over time. A Grok UI editor will be added, as will the ability to enrich your data at query time from an external index.&nbsp;
</p><p>Balancing performance and flexibility, this simple yet powerful implementation of schema on read opens up new ways of indexing and searching data while complementing the existing schema on write mechanism that Elastic is known for.&nbsp;
</p> <strong><h2>Get started today</h2></strong><p>To get started with runtime fields, spin up a cluster on the <a href="https://info.elastic.co/elasticsearch-service-trial-30d.html">Elasticsearch Service</a> or <a href="https://www.elastic.co/downloads/">install the latest version</a> of the Elastic Stack. Already have Elasticsearch running? Just upgrade your clusters to 7.11 and give it a try. If you’d like to know more about how it works, you can read the <a href="https://elastic.co/blog/getting-started-with-elasticsearch-runtime-fields">runtime fields detailed technical blog</a>&nbsp;or the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime.html">runtime fields documentation</a>.&nbsp;
</p>

blog-banner-search-results-orange.png

blog-thumb-search-results-orange.png

Runtime fields: Schema on read for Elastic

Introducing runtime fields, Elastic's implementation of schema on read. Now you can use either schema on write for performance or schema on read for flexibility. In this blog, we discuss how to get started with runtime fields.

<p>Historically, Elasticsearch has relied on a schema on write approach to make searching data fast. We are now adding schema on read capabilities to Elasticsearch so that users have the flexibility to alter a document's schema after ingest and also generate fields that exist only as part of the search query. Together, schema on read and schema on write provides users with the choice to balance performance and flexibility based on their needs.<br>
</p><p>Our solution for schema on read is runtime fields, which are evaluated only at query time. They are defined in the index mapping or in the query, and once defined they are immediately available for search requests, aggregations, filtering, and sorting. Because runtime fields aren’t indexed, adding a runtime field doesn’t increase the index size. They can, in fact, reduce storage costs and increase the speed of ingestion.
</p><p>However, there are tradeoffs. Queries against runtime fields can be expensive, so data that you commonly search or filter on should still be mapped to indexed fields. Runtime fields can also decrease search speed, even though your index size is smaller. We recommend using runtime fields in tandem with indexed fields to find the right balance between ingest speed, index size, flexibility, and search performance for your use cases.
</p> <strong><h2>It’s easy to add runtime fields</h2></strong><p>The easiest way to define a runtime field is in the query. For example, if we have the following index:
</p><pre class="prettyprint"> PUT my_index
 {
   "mappings": {
     "properties": {
       "address": {
         "type": "ip"},
       "port": {
         "type": "long"
       }
     }
   } 
 }
</pre><p>And load a few documents into it:
</p><pre class="prettyprint"> POST my_index/_bulk
 {"index":{"_id":"1"}}
 {"address":"1.2.3.4","port":"80"}
 {"index":{"_id":"2"}}
 {"address":"1.2.3.4","port":"8080"}
 {"index":{"_id":"3"}}
 {"address":"2.4.8.16","port":"80"}
</pre><p>We can create the concatenation of two fields with a static string as follows:
</p><pre class="prettyprint"> GET my_index/_search
 {
   "runtime_mappings": {
     "socket": {
       "type": "keyword",
       "script": {
         "source": "emit(doc['address'].value + ':' + doc['port'].value)"
       }
     }
   },
   "fields": [
     "socket"
   ],
   "query": {
     "match": {
       "socket": "1.2.3.4:8080"
     }
   }
 }
</pre><p>Yielding the following response:
</p><pre class="prettyprint">…
     "hits" : [
       {
         "_index" : "my_index",
         "_type" : "_doc",
         "_id" : "2",
         "_score" : 1.0,
         "_source" : {
           "address" : "1.2.3.4",
           "port" : "8080"
         },
         "fields" : {
           "socket" : [
             "1.2.3.4:8080"
           ]
         }
       }
     ]
</pre><p>We defined the field socket in the runtime_mappings section. We used a short Painless script that defines how the value of socket will be calculated per document (using + to indicate concatenation of the value of the address field with the static string ‘:’ and the value of the port field). We then used the field socket in the query. The field socket is an ephemeral runtime field that exists only for this query and is calculated when the query is run. When defining a Painless script to use with runtime fields, you must include emit to return calculated values.
</p><p>If we find that socket is a field that we want to use in multiple queries without having to define it per query, we can simply add it to the mapping by making the call:
</p><pre class="prettyprint"> PUT my_index/_mapping
 {
   "runtime": {
     "socket": {
       "type": "keyword",
       "script": {
         "source": "emit(doc['address'].value + ':' + doc['port'].value)"
       }
     } 
   } 
 }
</pre><p>And then the query does not have to include the definition of the field, for example:
</p><pre class="prettyprint"> GET my_index/_search
 {
   "fields": [
     "socket"
  ],
   "query": {
     "match": {
       "socket": "1.2.3.4:8080"
     }
   }
 }
</pre><p>The statement "fields": ["socket"] is only required if you want to display the value of the socket field. While the field socket is now available to any query, it does not exist in the index and does not increase the index’s size. Socket is calculated only when a query requires it and for the documents for which it is required.
</p> <strong><h2>Consumed like any field</h2></strong><p>Because runtime fields are exposed through the same API as indexed fields, a query can refer to some indices where the field is a runtime field, and other indices where the field is an indexed field. You have the flexibility to choose which fields to index and which ones to keep as runtime fields. This separation between field generation and field consumption facilitates more organized code that is easier to create and maintain.
</p><p>You define runtime fields in the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime-mapping-fields.html">index mapping</a> or in the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime-search-request.html">search request</a>. This inherent capability provides flexibility in how you use runtime fields in conjunction with indexed fields.&nbsp;
</p> <strong><h2>Override field values at query time</h2></strong><p>Oftentimes, you realize mistakes in your production data when it's too late.&nbsp; While it is easy to fix the ingest instructions for documents that you will ingest in the future, it’s much more challenging to fix the data that has already been ingested and indexed. Using runtime fields, you can fix errors in your indexed data by <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime-override-values.html" target="_blank">overriding values at query time</a>. Runtime fields can shadow indexed fields with the same name so that you can correct errors in your indexed data.&nbsp;&nbsp;
</p><p>Here’s a simple example to make this more concrete. Let’s say we have an index with a message field and an address field:
</p><pre class="prettyprint"> PUT my_raw_index 
{
  "mappings": {
    "properties": {
      "raw_message": {
        "type": "keyword"
      },
      "address": {
        "type": "ip"
      }
    }
  }
}
</pre><p>And let’s load a document into it:
</p><pre class="prettyprint"> POST my_raw_index/_doc/1
{
  "raw_message": "199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] GET /history/apollo/ HTTP/1.0 200 6245",
  "address": "1.2.3.4"
}
</pre><p>Alas, the document contains a wrong IP address in the address field. The correct IP address exists in the message but somehow the wrong address was parsed out in the document that was sent to be ingested into Elasticsearch and indexed. For a single document, that’s not a problem, but what if we discover after a month that 10% of our documents contain a wrong address? Fixing it for new documents is not a big deal, but reindexing the documents that were already ingested is frequently operationally complex. With runtime fields, it can be fixed immediately, by shadowing the indexed field with a runtime field. Here is how you would do it in a query:
</p><pre class="prettyprint">GET my_raw_index/_search
{
  "runtime_mappings": {
    "address": {
      "type": "ip",
      "script": "Matcher m = /\\d+\\.\\d+\\.\\d+\\.\\d+/.matcher(doc[\"raw_message\"].value);if (m.find()) emit(m.group());"
    }
  },
  "fields": [ 
    "address"
  ]
}
</pre><p>You can also do the change in the mapping so that it is made available for all queries. Note that the use of regex is now enabled by default through Painless script.
</p><strong><h2>Balance performance and flexibility</h2></strong><p>With indexed fields, you make all the preparations during ingest and maintain the sophisticated data structures to provide optimal performance. But querying runtime fields is slower than querying indexed fields. So what if your queries are slow after you start using runtime fields?
</p><p>We recommend using asynchronous search when retrieving a runtime field. The full result set is returned just like in a synchronous search, provided that the query completes within a given time threshold. However, even if the query doesn't finish in that time, you still get a partial result set and Elasticsearch will continue polling until the complete result set is returned. This mechanism is particularly useful when managing an index lifecycle because newer results typically return first and are also typically more important to users.
</p><p>To provide optimal performance, we rely on the indexed fields to do the heavy lifting of the query so that the values of runtime fields are only calculated for a subset of the documents.
</p><strong><h2>Changing a field from runtime to indexed</h2></strong><p>Runtime fields allow users to flexibly change their mapping and parsing while working on data in a live environment. Because a runtime field does not consume resources, and because the script that defines it can be changed, users can experiment until they reach the optimal mapping. When a runtime field is found to be useful for the long term, it is possible to precalculate its value at index time by simply defining that field in the&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-templates.html">template</a>&nbsp;as an indexed field and making sure that the ingested document includes it. The field will be indexed from the next&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/index-lifecycle-management.html">index rollover</a>&nbsp;and provide better performance. The queries that use the field do not need to change at all.&nbsp;
</p><p>This scenario is particularly useful with dynamic mapping. On the one hand, it is very helpful to allow new documents to generate new fields, because that way the data in them can be immediately used (the structure of entries frequently changes, e.g., due to a change in the software that generates the log). On the other hand, dynamic mapping comes with the risk of burdening the index and even creating a mapping explosion, because you never know if some document might surprise you with&nbsp;<a href="https://www.elastic.co/blog/found-crash-elasticsearch#mapping-explosion">2000 new fields</a>. Runtime fields can provide a solution to this scenario. The new fields can be automatically created as runtime fields so as not to burden the index (since they do not exist in the index), and they are not counted in the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/mapping.html#mapping-limit-settings">index.mapping.total_fields.limit.</a> These automatically created runtime fields are queryable, albeit with lower performance, so users can use them and, if needed, decide to change them to indexed fields in the next rollover.&nbsp;&nbsp;&nbsp;
</p><p>We recommend using runtime fields initially to experiment with your data structure. After working with your data, you might decide to index a runtime field for better search performance. You can create a new index and then add the field definition to the index mapping, add the field to&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/mapping-source-field.html">_source</a>&nbsp;and make sure the new field is included in the ingested documents. If you're using&nbsp;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/data-streams.html">data streams</a>, you can update your index template so that when indices are created from that template, Elasticsearch knows to index that field. In a future release, we plan to make the process of changing a runtime field to an indexed field as simple as moving the field from the runtime section of the mapping to the properties section.&nbsp;
</p><p>The following request creates a simple index mapping with a timestamp field. Including "dynamic": "runtime" instructs Elasticsearch to dynamically create additional fields in this index as runtime fields. If a runtime field includes a Painless script, the value of the field will be calculated based on the Painless script. If a runtime field is created <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.11/runtime-mapping-fields.html">without a script</a>, as shown in the following request, the system will look for a field in _source that has the same name as the runtime field and will use its value as the value of the runtime field.
</p><pre class="prettyprint">PUT my_index-1
{
  "mappings": {
    "dynamic": "runtime",
    "properties": {
      "timestamp": {
        "type": "date",
        "format": "yyyy-MM-dd"
      }
    }
  }
}
</pre><p>Let’s index a document to see the advantages of these settings:
</p><pre class="prettyprint">POST my_index-1/_doc/1
{
  "timestamp": "2021-01-01",
  "message": "my message",
  "voltage": "12"
}
</pre><p>Now that we have an indexed timestamp field and two runtime fields (message and voltage), we can view the index mapping:
</p><pre class="prettyprint">GET my_index-1/_mapping
</pre><p>The runtime section includes message and voltage. These fields are not indexed, but we can still query them exactly as if they were indexed fields.
</p><pre class="prettyprint">{
  "my_index-1" : {
    "mappings" : {
      "dynamic" : "runtime",
      "runtime" : {
        "message" : {
          "type" : "keyword"
        },
        "voltage" : {
          "type" : "keyword"
        }
      },
      "properties" : {
        "timestamp" : {
          "type" : "date",
          "format" : "yyyy-MM-dd"
        }
      }
    }
  }
}
</pre><p>We’ll create a simple search request that queries on the message field:
</p><pre class="prettyprint">GET my_index-1/_search
{
  "query": {
    "match": {
      "message": "my message"
    }
  }
}
</pre><p>The response includes the following hits:
</p><pre class="prettyprint">... 
"hits" : [
      {
        "_index" : "my_index-1", 
        "_type" : "_doc", 
        "_id" : "1", 
        "_score" : 1.0, 
        "_source" : { 
          "timestamp" : "2021-01-01", 
          "message" : "my message", 
          "voltage" : "12" 
        } 
      } 
    ]
…
</pre><p>Looking at this response, we notice a problem: we didn’t specify that voltage is a number! Because voltage is a runtime field, that’s easy to fix by updating the field definition in the runtime section of the mapping:
</p><pre class="prettyprint">PUT my_index-1/_mapping
{
  "runtime":{
    "voltage":{
      "type": "long"
    }
  }
}
</pre><p>The previous request changes voltage to a type of long, which immediately takes effect for documents that were already indexed. To test that behavior, we construct a simple query for all documents with a voltage between 11 and 13:
</p><pre class="prettyprint">GET my_index-1/_search
{
  "query": {
    "range": {
      "voltage": {
        "gt": 11,
        "lt": 13
      }
    }
  }
}
</pre><p>Because our voltage was 12, the query returns our document in my_index-1. If we view the mapping again, we’ll see that voltage is now a runtime field of type long, even for documents that were ingested into Elasticsearch before we updated the field type in the mapping:
</p><pre class="prettyprint">...
{
  "my_index-1" : {
    "mappings" : {
      "dynamic" : "runtime",
      "runtime" : {
        "message" : {
          "type" : "keyword"
        },
        "voltage" : {
          "type" : "long"
        }
      },
      "properties" : {
        "timestamp" : {
          "type" : "date",
          "format" : "yyyy-MM-dd"
        }
      }
    }
  }
}
…
</pre><p>Later, we might decide that voltage is useful in aggregations and we want to index it into the next index that's created in a data stream. We create a new index (my_index-2) that matches the index template for the data stream and define voltage as an integer, knowing which data type we want after experimenting with runtime fields.
</p><p>Ideally, we would update the index template itself so that changes take effect on the next rollover. You can run queries on the voltage field in any index matching the my_index* pattern, even though the field is a runtime field in one index and an indexed field in another.
</p><pre class="prettyprint">PUT my_index-2
{
  "mappings": {
    "dynamic": "runtime",
    "properties": {
      "timestamp": {
        "type": "date",
        "format": "yyyy-MM-dd"
      },
      "voltage":
      {
        "type": "integer"
      }
    }
  }
}
</pre><p>With runtime fields, we have therefore introduced a new field lifecycle workflow. In this workflow, a field can automatically be generated as a runtime field without impacting resource consumption and without risking mapping explosion, allowing users to immediately start working with the data. The field’s mapping can be refined on real data while it is still a runtime field, and due to the flexibility of runtime fields, the changes take effect on documents that were already ingested into Elasticsearch. When it is clear that the field is useful, the template can be changed so that in the indexes that will be created from that point on (after the next rollover), the field will be indexed for optimal performance.
</p><strong><h2>Summary</h2></strong><p>For the great majority of cases, and in particular, if you know your data and what you wish to do with it, indexed fields are the way to go due to their performance advantage. On the other hand, when there is a need for flexibility in document parsing and schema structure, runtime fields now provide the answer.
</p><p>Runtime fields and indexed fields are complementing features — they form a symbiosis. Runtime fields offer flexibility, but they would never be able to perform well in a high-scale environment without the support of the index. The powerful, rigid structure of the index provides a sheltered environment in which the flexibility of runtime fields can boast their true colors, in a way not too different from how algae find shelter in a coral. Everyone benefits from this symbiosis.
</p><strong><h2>Get started today</h2></strong><p>To get started with runtime fields, spin up a cluster on the <a href="https://info.elastic.co/elasticsearch-service-trial-30d.html">Elasticsearch Service</a> or <a href="https://www.elastic.co/downloads/">install the latest version</a> of the Elastic Stack. Already have Elasticsearch running? Just upgrade your clusters to 7.11 and give it a try. For a higher-level view of runtime fields and its benefits, please read the <a href="https://elastic.co/blog/introducing-elasticsearch-runtime-fields">Runtime fields: Schema on read for Elastic</a> blog post. In addition, we have also recorded 4 vidoes to help you get started using runtime fields.
</p><ul>
	<li><a href="https://youtu.be/s6BNXA5o0rY">Dynamically Created Runtime Fields</a>
	</li>
	<li><a href="https://youtu.be/T4bZ35E6LgE">Creating a Day of Week Runtime Field + Using It in Kibana Lens</a>
	</li>
	<li><a href="https://youtube.com/playlist?list=PLhLSfisesZIuiFhz5kJYXYLwQK5acm4Et">Shadow an Indexed Field W/ a Runtime Field to Fix Errors</a></li>
	<li><a href="https://youtu.be/Yvtus8Y0uJ8">Ephemeral Runtime Field Using Arithmetic Operations</a>
	</li>
</ul>

blog-banner-elasticsearch-gears-light-blue.png

blog-thumb-elasticsearch-gears-light-blue.png

Getting started with runtime fields, Elastic’s implementation of schema on read

Elasticsearch 7.2.0 is here! Learn about our new proximity ranking (geo and time), enhancements to SQL, and support for OpenID Connect.

<p>Today we are pleased to announce the release of Elasticsearch 7.2.0, based on Lucene 8.0.0. This is the latest stable release, and is already available for deployment via our <a href="/cloud/elasticsearch-service">Elasticsearch Service</a> on <a href="https://www.elastic.co/cloud">Elastic Cloud</a>.
</p><p>Latest stable release in 7.2.0:
</p><ul>
	<li><a href="https://www.elastic.co/downloads/elasticsearch">Download Elasticsearch 7.2.0</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/release-notes-7.2.0.html">Elasticsearch 7.2.0 release notes</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/breaking-changes.html">Elasticsearch 7.2 breaking changes</a></li>
</ul><p>You can read about all the changes in the release notes linked above, but there are a few changes which are worth highlighting:
</p><h2>Geo and Time Proximity as Ranking Consideration</h2><p>In many scenarios, things that are closer are more relevant, and now there is a simple and effective way to include proximity among the relevance ranking considerations. When searching for information, records that are nearer to a point in time (typically more recent) are typically more relevant. When searching for anything with physical presence, the geographical proximity should typically play a part in defining relevancy. In other words, including proximity (time or geo) in relevance ranking will improve ranking in most scenarios. With Elasticsearch 7.2.0 users can normalize the time or geographical distance using saturation normalization function and add them to the relevance ranking score, using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/query-dsl-distance-feature-query.html">distance feature query</a> to achieve better relevance ranking. This feature already leverages the ‘<a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">top-k faster queries</a>’ performance improvement we introduced in 7.0. Read more about this capability and how to configure it in our blog post <a href="https://www.elastic.co/blog/distance-feature-query-time-and-geo-in-elasticsearch-result-ranking">Distance Feature Query: Time and Geo in Elasticsearch Result Ranking</a>.
</p><h2>Typeahead Search</h2><p>Those of you that have worked with search systems, probably noticed that getting suggested search results while you type the query can save you a lot of time.
</p><p>The needs and benefits are thus clear and simple. The engineering challenge is, however, significant: there are typically many words in the index that start with the few letters that the user keyed in, yet the system has but tens of milliseconds to come up with a few top ranking suggestions before the user types in the next letter, potentially with a spelling mistake… This is a clear case in which there is no need to know the number of results or make some aggregation about them, but rather to bring the few top ranking results, typically the ones that are most frequently found in the index. Because only the top ranking results are of interest, we can leverage the ‘<a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">top-k faster queries</a>’ mechanism to achieve better performance with lower hardware requirements. The improved performance and reduced consumption of computing resources in turn allows us to do more: in 7.2 we support cases in which the word that the user keyed in is not necessarily the first word in the field, so a user may key in e.g. ‘hem’ and get the suggestion ‘Ernest Hemingway’ from the field.
</p><p>To achieve that we introduced a new field type -- <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-as-you-type.html">search_as_you_type</a> -- that is optimized for providing results from the field while the user is typing the query. The number of shingles used is configurable so that any user can balance index size and recall considerations according to their particular needs.
</p><h2>Resiliency Improvements<br></h2><p>Elasticsearch used to be, you know, just for search. Nowadays, it is also rapidly becoming the main data repository and analytical system for many users, or in other words, the source of truth. One of the reasons for this change is the system’s reliability and resilience. We at Elastic take this responsibility very seriously, and we keep improving the mechanisms that allow users to deal with malfunctions if they occur, and to do that in an economic manner. Features like replication, snapshot and backup allow our customers to maintain a more reliable service. In 7.2 we added two new mechanisms for that: replicated closed indices and snapshot UI.
</p><h3>Replicated Closed Indices</h3><p>In 7.2 we have added a layer on top of the closed indices mechanism, that allows replicating closed indices. We made this improvement in response to many users’ requests and as part of our commitment to resiliency and ease of use. We do expect, however, that the majority of our users will be using Frozen Indices instead of Closed Indices. <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/frozen-indices.html">Frozen Indices</a> have nearly no memory footprint, which is the main reason for which users use Closed Indices, but Frozen Indices can be searched almost as if they were open indices, without having to actively manage the logic and workflow of opening and closing the index. &nbsp;
</p><h3>Snapshot Repositories UI</h3><p>Elasticsearch’s snapshot mechanism allows keeping a backup of data at a point in time, using an efficient incremental snapshotting mechanism.
</p><p>With 7.2 we added a new <a href="https://www.elastic.co/guide/en/kibana/7.2/snapshot-repositories.html">Snapshot Repositories</a> app to Kibana. This app allows you to register repositories using all the cloud and on-prem storage plugins. It also allows you to browse the repositories and snapshots that you've created. We plan to enhance this app with functionality for taking, restoring, and deleting snapshots in future releases.
</p><p>Snapshot Repositories &nbsp;is released under Basic license.
</p><p><span></span><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blteeb08831c4456723/5d000b737e77466b173d9250/register_repo.png" data-sys-asset-uid="blteeb08831c4456723" alt="Register your snapshot repository" "="">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc54a919bb57eb2e1/5d000b83c06daee314d1f4fc/snapshot_details.png" data-sys-asset-uid="bltc54a919bb57eb2e1" alt="Snapshots tab in Snapshot Repositories" "="">
</p><h2>Enhancements to Elasticsearch SQL</h2><p>The ability to query Elasticsearch using SQL statements has just recently been made generally available as part of <a href="https://www.elastic.co/blog/elasticsearch-6-7-0-released">Elasticsearch 6.7</a> and is gaining popularity, in particular with those who are fluent in SQL but new to DSL and in scenarios of integration, including with BI tools, which frequently have built in integration through SQL. In line with its popularity, we keep expanding Elasticsearch’s SQL capabilities. Here are some of the things we added in 7.2:
</p><ul>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/sql-functions-geo.html">Geographical queries through SQL statements</a></li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/search-aggregations-metrics-median-absolute-deviation-aggregation.html">Median Absolute Deviation</a>, which calculates the median of each data point’s deviation from the median of the whole sample, i.e. for &nbsp;random variable x it is median(|median(x)-xi|)</li>
	<li><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/sql-functions-conditional.html">CASE/WHEN/ELSE/END</a> to define a set of if-else statements</li>
</ul><p>The above SQL functionality is released under Basic license.
</p><h2>Data Frames</h2><p>In 7.2 we have added <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/data-frame-apis.html">Data Frames</a>, a new plugin that allows you to pivot and aggregate your data from one or multiple indices to a single, separate index. This enables you to transform existing indexed data for specific use-cases. For example, you can now summarize users’ behaviors and gain new insights using a data frame pivot transform.
</p><p>Data frames are available under Basic license.
</p><h2>HTML Strip Processor</h2><p>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.2/htmlstrip-processor.html">HTML Strip Processor</a> simply replaces all HTML tags with “\n”. This operation means the _source field contains text that is more readable to humans. If you want to display the source data but not as an HTML, this new processor will probably save you some development time. It’s a fairly frequent case, so we decided to include it in the product.
</p><h2>OpenID Connect Realm</h2><p>In today’s software world security is a top priority. Unfortunately, security needs sometimes end up burdening users, e.g. by requesting users to authenticate separately to different systems. To avoid that, modern software systems use authentication layers to allow users to roam freely between various systems while automatically identifying themselves to every new system they access and retaining their permissions.
</p><p>So what is OpenID Connect? OpenID Connect is a commonly used authentication layer on top of oAuth2. oAuth2 is a protocol that allows third-party applications to grant limited access to an HTTP service. OpenID Connect is the authentication backbone used by Okta, Google and others, to allow users to authenticate in the background when moving between connected systems.
</p><p>OpenID Connect (OIDC) support is widely adopted by enterprise systems, so the newly added support in Elasticsearch and Kibana provides many of our customers with new (but commonly asked for) choices of how to secure their stack.
</p><p>OpenID support is provided under Platinum license.
</p><h2>Conclusion</h2><p>Please <a href="https://www.elastic.co/downloads/elasticsearch">download Elasticsearch 7.2.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our forum. You can report any problems on the <a href="https://github.com/elastic/elasticsearch/issues">GitHub issues page</a>.
</p>

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

Elasticsearch 7.2.0 released

Coming soon to Elasticsearch, you'll be able to implement the Distance Feature Query to use proximity (geographical and time) as part of your ranking score.

<p>In Elasticsearch, you will soon be able to use proximity (geographical or time) as part of your ranking score. The new way for combining proximity within the ranking score is easy to configure, performs well, and will likely yield superior ranking in many scenarios. This blog explains why and how.
</p><h2>Why use geographical distance and time for ranking?</h2><p>Consider a search for a restaurant. One typically uses a criteria, including type, price range, rating, as well as physical proximity. All other things kept equal, you would prefer the nearest restaurant. This is not unique — it would be the same in countless other relevance ranking scenarios e.g. when searching for a job, a date, a plumber, a dog-walker, a babysitter, a tourist attraction. In fact, relevance ranking for anything in which physical location matters — which is almost anything that exists in the physical world — would benefit from relating to geographical distance.
</p><p>A similar situation occurs with time related search, e.g. when searching for an academic article, a newspaper article, a blogpost or a product review, one will typically prefer recent records over less recent records (assuming all other aspects are equal). As with physical proximity in search for things that have physical location in the real world, recency would be the typical preference in searches for information. In other words, almost any relevance ranking criteria would benefit from the inclusion of proximity, either date proximity or geographic proximity.
</p><p>We have frequently seen date and geographical proximity used in either of the following ways:
</p><ol>
	<li>As a filter, i.e. dichotomic criteria, e.g. retrieve only records nearer than x or within the last y days, and rank these records by other search criteria (e.g. using <a href="https://www.elastic.co/blog/practical-bm25-part-2-the-bm25-algorithm-and-its-variables">BM25</a> on text)</li>
	<li>As the only ranking criteria, e.g. sort all the records that answer a dichotomic criteria by their geographical or date proximity</li>
</ol><p>In other words, proximity is either used as a retrieval criterion, rather than for relevance ranking, or is used as a single criterion for relevance ranking.
</p><p>The problem is that the user is often willing to walk a longer distance for a better ranked restaurant, wait a little longer for a better ranked babysitter or read the most cited article even if it is not the most recent. In other words our actual search criteria as searchers combines proximity (space and time) with other criteria (such as word frequency related criteria) into a single relevance score.
</p><h2>Can’t you do it with Function Score or Painless script?</h2><p>Picking our other hat, as the people designing, tuning and programming search algorithms, we frequently avoided such a combined proximity and word frequency (or various other numeric) ranking criteria because it was technically challenging. There are two main challenges:
</p><ol>
	<li><strong>Normalization</strong> - The proximity score has to be normalized so that it will have the right impact when summed together with other relevance scores, such as BM25. The default in Lucene is to simply sum the various scores into one number and in most cases it makes sense, as long as the different scores have been normalized first. The normalization itself, however, is not trivial.</li>
	<li><strong>Performance</strong> - Re-ranking with proximity took a significant performance toll. To deal with it, the more sophisticated Elastic users frequently formed a criteria that did not include proximity to select a very small subset of most relevant records, and then reranked only these records after taking proximity into account. This compromise means the algorithm will not consider results that could become the most relevant ones after proximity is taken into account.</li>
</ol><p>We want to assist our users with non-trivial algorithmic challenges, so we sprung into action. In an upcoming version,&nbsp;we will enable users to rank all records with a combination of the ‘normal’ ranking score and a normalized proximity (time or space) score.
</p><p>So can you add proximity to the ranking algorithm using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-function-score-query.html">Function Score</a> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.0/query-dsl-function-score-query.html#function-script-score">Script Score</a>? It depends. It is possible to relate to proximity and to normalize it with function score and Painless script currently, but only on a small subset of the result set, and with a significant performance toll. Sometimes these quantitative differences sum up to a qualitative difference: with distance_feature queries, ranking by proximity in combination with other relevance ranking algorithms, becomes the reasonable principle ranking mechanism in many scenarios.
</p><h2>Saturation Normalization Function</h2><p>For the normalization, we used the concept of the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-rank-feature-query.html#_saturation_2">Saturation</a> normalization function. The concept behind the Saturation normalization function is similar to the one used in BM25 to relate to the length of the text in the document in setting the ranking score for that document. A record’s score will be computed using the following formula: <math>
	<mi>Score</mi>
	<mo>=</mo>
	<mi>Boost</mi>
	<mo>*</mo>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	</math>. Assuming <math>
	<mi>Pivot</mi>
	<mo>&gt;</mo>
	<mi>0</mi>
	</math>, the function <math>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	</math> is a nonlinear strictly decreasing function starting from <math>
	<mi>1</mi>
	</math> when <math>
	<mi>Distance</mi>
	<mo>=</mo>
	<mi>0</mi>
	</math> (assuming <math>
	<mi>Distance</mi>
	<mo>&gt;=</mo>
	<mi>0</mi>
	</math>) with <math>
	<msub>
	<mi>lim</mi>
	<mrow>
	<mi>Distance</mi>
	<mo>→</mo>
	<mi>∞</mi>
	</mrow>
	</msub>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	<mo>=</mo>
	<mi>0</mi>
	</math>. One way to think of the <math>
	<mi>Pivot</mi>
	</math> parameter is that it defines the distance at which <math>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	<mo>=</mo>
	<mfrac>
	<mi>1</mi>
	<mi>2</mi>
	</mfrac>
	</math>but another way to think of it is that it defines how quickly the contribution of distance to the score diminishes. The boost parameter is an optional parameter that by default is set to <math>
	<mi>1</mi>
	</math>, and allows to set the influence of distance on the overall score in a way that is reduced linearly to the distance, i.e. it is a way to indicate how important will distance be in the overall score, regardless of the specific area of the scale a record is in.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9ac76ab76b19c976/5cd0ccf34c5fe89364167936/geo-distance-relevance.png" data-sys-asset-uid="blt9ac76ab76b19c976" alt="Linear relevance reduction" style="display: block; margin: auto;">
</p><p>The above is the function plot for the function&nbsp;<math style="background-color: initial;"><mi>Score</mi>
	<mo>=</mo>
	<mfrac>
	<mi>Pivot</mi>
	<mrow>
	<mi>Pivot</mi>
	<mo>+</mo>
	<mi>Distance</mi>
	</mrow>
	</mfrac>
	</math> with <math style="background-color: initial;">
	<mi>Pivot</mi>
	</math> set to <math style="background-color: initial;">
	<mi>2</mi>
	</math> and <math style="background-color: initial;">
	<mi>Distance</mi>
	</math> plotted on the X axis.
</p><p>The bottom line is that by providing one mandatory parameter (<math style="background-color: initial;"><mi>Pivot</mi>
	</math>) the user gets a normalized score, ranging from <math style="background-color: initial;">
	<mi>1</mi>
	</math> to <math style="background-color: initial;">
	<mi>0</mi>
	</math> (given that our computers are discrete and have a ‘highest number’ that functions as infinity in this case). This score allows distance to impact more when the distance is small and less when the distance is large, which is how people typically want distance to impact.
</p><h2>Better performance enabling better ranking</h2><p>In theory, all of this could be done currently using re-ranking through Painless script. The addition of <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-distance-feature-query.html">Distance Feature Query</a> simplifies it considerably and opens the door for those who are less proficient with Painless, which is important, but not the main benefit. The main issue is that in practice the performance toll frequently meant that the proximity impact on the ranking score could only be calculated for a tiny fraction of the records. The implication was that it was quite likely that the most relevant records after the introduction of the proximity score would not be in the set for which this impact would be calculated.
</p><p>The Distance Feature Query performs better than the Painless script, but that’s not the big news. The really big news is that the distance feature query was designed to leverage the performance improvements resulting from the faster top-k queries (see <a href="https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand">Magic WAND: Faster Retrieval of Top Hits in Elasticsearch</a> blog post). Those who can afford to get a lower boundary for number of results instead of an exact count can enjoy a potentially unbounded improvement in performance because the algorithm will disqualify at an early stage all records that cannot be one of the top K candidates.
</p><h2>Examples</h2><p>Let’s plan an application for hotel reviews. A core functionality is to allow searching for reviews about a hotel. The user will want to search for the specific hotel, and to get reviews about aspects that are of interest to the user in the hotel as well as being comprehensive and as recent as possible.
</p><p>Say our record structure is as follows:
</p><pre class="prettyprint">PUT reviews
{
  "mappings": {
    "properties": {
      "hotel": {
        "type": "text"
      },
      "review": {
        "type": "text"
      },
      "date": {
        "type": "date"
      }
    }
  }
}
PUT reviews/_doc/1
{
  "hotel": "Fawlty Towers",
  "review": "The hotel would have been managed in a perfect way, if it wasn't for the guests",
  "date" : "2017-12-27"
}
PUT reviews/_doc/2
{
  "hotel": "Fawlty Towers",
  "review": "The restaurant is, well, interesting",
  "date": "2018-07-07"
}
PUT reviews/_doc/3
{
  "hotel": "Fawlty Towers",
  "review": "We had an interesting vacation there. The service at the restaurant is a great chance to brush the dust off your Spanish. Mr. Fawlty himself had recently moved to The Life of Brian.",
  "date": "2019-03-25"
}
GET reviews/_search
{
    "query": {
        "bool" : {
            "must" : {
                "match" : {"hotel": "Fawlty Towers"}
            },
            "should" : [
                {"distance_feature" : {"field" : "date","pivot" : "30d","origin": "now"}},
                {"match" : {"review": "restaurant"}}    
            ]
        }
    }
}
Results:
{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.5571662,
    "hits": [
      {
        "_index": "reviews",
        "_type": "_doc",
        "_id": "3",
        "_score": 1.5571662,
        "_source": {
          "hotel": "Fawlty Towers",
          "review": "We had an interesting vacation there. The service at the restaurant is a great chance to brush the dust off your Spanish. Mr. Fawlty himself had recently moved to The Life of Brian.",
          "date": "2019-03-25"
        }
      },
      {
        "_index": "reviews",
        "_type": "_doc",
        "_id": "2",
        "_score": 1.0365787,
        "_source": {
          "hotel": "Fawlty Towers",
          "review": "The restaurant is, well, interesting",
          "date": "2018-07-07"
        }
      },
      {
        "_index": "reviews",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.32892755,
        "_source": {
          "hotel": "Fawlty Towers",
          "review": "The hotel would have been managed in a perfect way, if it wasn't for the guests",
          "date": "2017-12-27"
        }
      }
    ]
  }
}
</pre><p>The query filters to show only records in which the hotel is “Fawlty Towers”. Within these records, the ranking is based on the word restaurant, which two records contain. By BM25 only, record 2 would be ranked higher than record 1, just because both records include the word “restaurant” once and the review in the second record is longer than in the 3rd one (the first record does not include the word restaurant and so would be ranked lowest). However, due to the time distance query, the 3rd record would show up first, and the user will get the latest update about Fawlty Towers.
</p><p>The distance feature query skips non-competitive records and through that achieved better performance, which allows running the combined relevance criteria that includes both time proximity and BM25 on natural language.
</p><h2>Conclusion</h2><p>Proximity in either geographical space or time is an important aspect of almost every relevance ranking use case. Adding distance ranking to the relevance ranking, together with the normalization capability, will for sure prove useful in real life scenarios, and represents our long term strategic investment in relevance ranking.
</p><p>Just as important is the full story of how this capability came about. It’s all started with a user from the community that came up with <a href="https://issues.apache.org/jira/browse/LUCENE-4100">a suggestion to incorporate the MAXSCORE</a> algorithm into Lucene. That original suggestion, although it was not accepted to Lucene in its original form, later led to our contribution of the block-max WAND algorithm to Lucene, and more generally introduced APIs that help compute top hits without evaluating all matches. This in-turn led to the development of Lucene's FeatureField for static features — exposed in Elasticsearch via the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rank-feature.html#rank-feature">rank_feature</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rank-features.html">rank_features</a> fields — and now feature queries for dynamically computed distances. This cycle of contributing and exposing the code we produce so that the community can come up with feedback and ideas, helps steer us towards new, useful and innovative directions.
</p><p>We’re excited about this new feature, and we hope you are too. So for now, read more about the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.x/query-dsl-distance-feature-query.html"><code>distance_feature</code> query</a>, and once it&nbsp;is available, download it and take it for a spin. Once you’ve had a chance to test this new geo-ranking feature out, <a href="https://discuss.elastic.co">give us your feedback and comments</a> about how you want to use distance feature query. Your input will help direct us in our future development.
</p>

AUTHOR

Articles by Gilad Gal

Runtime fields: Schema on read for Elastic

Getting started with runtime fields, Elastic’s implementation of schema on read

Elasticsearch 7.2.0 released

Distance Feature Query: Time and Geo in Elasticsearch Result Ranking

Sign up for Elastic Cloud free trial

Products & Solutions

Company

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Gilad Gal

Runtime fields: Schema on read for Elastic

Getting started with runtime fields, Elastic’s implementation of schema on read

Elasticsearch 7.2.0 released

Distance Feature Query: Time and Geo in Elasticsearch Result Ranking

Sign up for Elastic Cloud free trial