To improve the simplicity of retrieving fields, Discover now uses the fields API by default. Learn more about how you can benefit from the new search parameter.

With Elastic 7.12, Discover now uses the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param"><code>fields</code> API</a> by default. Reading from <code>_source</code> is still supported through a switch in the Advanced Settings. This change stems from updates made to Elasticsearch in 7.11 with the extension of the Search API to include the new <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param"><code>fields</code></a> parameter.
When using the new search parameter, both a document’s raw source and the index mappings to load and return values are used. Because it makes use of the mappings, <code>fields</code> has some advantages over referencing the <code>_source</code> directly: it accepts multifields and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/field-alias.html">field aliases</a>, and also formats field values like dates in a consistent way.
In short, the fields option simplifies all the nuances of <code>_source</code> and other places&nbsp;by dealing with multi-mapping and aliases, type coercion, docvalues, and all the other edge cases.
<h2>What are the benefits of the new search parameter?</h2>More and more use cases arise where Elasticsearch is treated as a data store. In order to improve the simplicity of retrieving fields without the expectation of expertise in areas like mappings, doc_values, stored fields, etc., we decided to extend the existing API.&nbsp;
<h3>Some benefits of the new parameter:</h3><ul>
	<li aria-level="1">If a non-standard field like a field alias, multi-field, or constant_keyword is specified in <code>fields</code>, mappings will find and return the right value.</li>
	<li aria-level="1">The fields are returned in a flat list, as opposed to structured JSON.</li>
	<li aria-level="1">Each value would be returned in a 'canonical' format — for example if a field is mapped as an integer, it will be returned as an integer even if it was specified as a string in the <code>_source</code>.</li>
</ul><h2>What should I expect?</h2>The most meaningful change is the support for runtime fields. When a runtime field is defined in a mapping, it will show up in Discover like any other field type. The next sections expand on all of the improvements and changes.
<h3>Default column is now called Document</h3>If there are no columns configured, the default column changes from <code>_source</code> to <code>Document</code>.
Existing saved searches with <code>_source</code> column will be changed to <code>Document</code>. Unlike source, the <code>Document</code> column cannot be combined with other columns in the grid.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta9496688ad999005/60639a1c9a1d7d1050bd3839/discover-fields-api-document-columns.png" data-sys-asset-uid="blta9496688ad999005" alt="discover-fields-api-document-columns.png">
Although it is formatted similar to the <code>_source</code> column, it is a collection of fields as returned from an Elasticsearch response and therefore, combining it with other columns would not make much sense.
<h3>Multi-fields are grouped together</h3>Multi-field is the field mapped differently for different purposes. For instance, a string field could be mapped as a text field for full-text search, and as a keyword field for sorting or aggregations.
Due to the changes in the API, multi-sub fields will now have values where before they didn’t. In order to avoid suddenly increasing the number of fields that are shown, we’ve decided to group multi-sub fields under the root field so they are still usable as columns, but not as prominent.&nbsp;&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt99224476ee765750/60639aaebc3ffc1037a403fa/discover-fields-api-multi-fields.png" data-sys-asset-uid="blt99224476ee765750" alt="discover-fields-api-multi-fields.png">
<h3>Differences in how object fields are displayed</h3>Object fields usually consist of one or more leaf fields. Example of an object field:
<pre class="prettyprint">"manager": { 
 "age": 30, 
 "name": { 
 "first": "John", 
 "last": "Smith" 
 } 
 }
</pre>Before the introduction of the fields API, Discover would show the object roots, together with the leaf fields.
With the introduction of the fields API, object roots won't be shown in the field list anymore (unless there is a previously saved search containing one as a column). Leaf fields will now be correctly detected as having data, where previously they were not. This will be the case even when the document has an array of objects. Leaf fields will work for all objects, and the values of all objects in that array will be shown flattened.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc7782faad0e4eff0/60639ad325c15e102fb9a032/discover-fields-api-object-fields.png" data-sys-asset-uid="bltc7782faad0e4eff0" alt="discover-fields-api-object-fields.png">
<h3>Differences in how nested fields are displayed</h3>The nested data type is used for indexing arrays of objects when there is a need to maintain the independence of each object in the array. They need to be specified as a type in the mapping, otherwise they would be treated as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html"><code>object</code></a> data type.
Before the introduction of the fields API, Discover would show a root field if at least one document loaded has an array of objects for that field. Otherwise, the root field wouldn’t be shown. The leaf fields will be shown if they appear in at least one document. Leaf fields that do not appear in any document or only in documents that have arrays of objects for this object field, will not appear in the field list unless "Hide missing fields" is switched off.
In the table, root fields as columns will show the JSON of the fields’ content ONLY if that document contains an array of objects and will be empty otherwise.&nbsp;
Let’s say we have a mapping as follows:
<pre class="prettyprint">"products": {
 "type": "nested",
 "properties": {
 "name": { "type": "keyword" },
 "price": { "type": "double" }
 }
 },
</pre>And the following documents:
<pre class="prettyprint">POST /discover_test/_doc
{
"products": [
 { "name": "Kibana", "price": 42.23 },
 { "name": "Faber-Castell Polychromos", "price": 29.95 }
 ],
}
POST /discover_test/_doc
{
 "products": { "name": "Product name", "price": 123.21 },
}
</pre>In this case, when using <code>_source</code>, Discover would only correctly display the JSON of the products object in the first document since it is an array. The second document, however, is a single object, meaning it would display as empty:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt98ab2b26c766472d/60639b475a03370f176cfebc/discover-fields-api-document-1.png" data-sys-asset-uid="blt98ab2b26c766472d" alt="discover-fields-api-document-1.png">
Leaf fields as columns will only show values if the document did not contain an array of objects and stay empty otherwise:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb7f04a8dbc41e1f9/60639bbf389f150ff9aa4539/discover-fields-api-document-2.png" data-sys-asset-uid="bltb7f04a8dbc41e1f9" alt="discover-fields-api-document-2.png">
With the introduction of the fields API, the root of a nested field can be added from the field list and will show the structured JSON of that field in the document. Nested field columns now show the JSON correctly for all documents:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc5bd634b92a55c98/60639bca4f68540feec10164/discover-fields-api-document-3.png" data-sys-asset-uid="bltc5bd634b92a55c98" alt="discover-fields-api-document-3.png">
In the case of an existing saved search with a nested leaf field as a column, this will show as empty now:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2547ad02430a0445/60639bd74f68540feec10168/discover-fields-api-document-4.png" data-sys-asset-uid="blt2547ad02430a0445" alt="discover-fields-api-document-4.png">
We want to have this behavior going forward since we assume the nested mapping type is only used when an array of objects is indexed.
<h3>Unmapped fields won’t show up if field filters are configured</h3>When a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping (unless <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/dynamic.html">dynamic</a> parameter is set to false). Discover will continue showing these fields as it did previously, except in the case when field filters are configured. In that case, unmapped fields will not show up.
<h2>Limitations of _source</h2>Source contains the raw information from the document, exactly as it was ingested. This means users lose the benefit of features built on top of <code>_source</code>, like runtime fields, multi-fields, date formatting, and alias fields. Also, the whole <code>_source</code> object must be loaded and parsed even if only a small number of fields are requested.
With the introduction of the <code>fields</code> parameter, developers can retrieve the field values after the logic from these features is applied. This abstraction removes the need for developers to know implementation details about a field. It doesn’t matter if the field is runtime or not, it will always be accessed in the same way.
Here is an example. Let’s say we have a mapping defined as follows:
<pre class="prettyprint">PUT /my-index
{
 "mappings": {
 "properties": {
 "xlong": {
 "type": "long"
 },
 "xfloat": {
 "type": "float"
 }
 }
 }
}
POST my-index/_doc/1
{
 "xlong": "1.5",
 "Xfloat": "1.5"
}
POST my-index/_doc/2
{
 "Xlong": "2",
 "Xfloat": "2"
}
</pre>The field xlong is mapped as long, which means that internally its value will be rounded towards zero. In that case, 1.5 becomes 1. When executing the following search in Discover,
<pre class="prettyprint">xlong &gt; 1 and xlong &lt; 2
</pre>the search will not return any values, which is expected. However, when viewing documents in Discover, when reading fields from <code>_source</code>, the value will be displayed exactly as it was ingested, ie. 1.5:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0f94d4d4de89e7bf/60639c95dfb49b0f38b4cc14/discover-fields-api-document-ingest-1.png" data-sys-asset-uid="blt0f94d4d4de89e7bf" alt="discover-fields-api-document-ingest-1.png">
This used to create a lot of confusion. With the fields option enabled, the actual mapped value (1)&nbsp; is displayed:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbbab80ab6123c318/60639cb3573d281045d35d5b/discover-fields-api-document-ingest-2.png" data-sys-asset-uid="bltbbab80ab6123c318" alt="discover-fields-api-document-ingest-2.png">
<h2>Wrap up</h2>We are excited to bring the power and flexibility of runtime fields to Discover, including the switch to using the fields API. If you are currently not on Elastic 7.12, but would still like to test out these updates, starting a free <a href="https://www.elastic.co/elasticsearch/service">Elasticsearch Service trial on Elastic Cloud</a> is a fast and easy way to do so.&nbsp;
To learn more about other recent product updates, be sure to read our <a href="https://www.elastic.co/blog/whats-new-elastic-7-12-0-schema-on-read-frozen-tier-autoscaling">Elastic 7.12 release blog</a>.&nbsp;

blog-banner-release-kibana.png

blog-thumb-release-kibana.png

Discover in Kibana uses the fields API in 7.12

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Senior Software Engineer

Maja Grubic

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/elastic-cloud-account-terms" target="_self">服务条款</a>，并且您同意通过您提供的上述联系方式接收<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">所选的通信</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/serverless-preview-terms" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

提交即表示您同意 <a href="/cn/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud 服务条款</a>。我们将会按照 <a href="/cn/legal/privacy-statement">Elastic 的隐私声明</a>处理您的个人数据。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

提交即表示您确认您已阅读并同意我们的<a href="/cn/legal/terms-of-use" target="_self">服务条款</a>，并且 Elastic 可以使用您提供的上述详细信息与您联系，以向您介绍我们相关的<a href="/cn/legal/privacy-statement#how-we-use-the-information" target="_self">产品和服务</a>。如需了解更多信息，或想随时取消订阅，请参阅 <a href="/cn/legal/privacy-statement/" target="_self">Elastic 的隐私声明</a>。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

本页内容尚不支持所选语言。Elastic 正在不断努力，以实现对多种语言内容的支持。感谢您在此期间给予的耐心与陪伴！

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

查看本次网络研讨会在我的时区的开始时间

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

无法实时参加？注册后我们会给您发送录像。您还将收到相关内容的电子邮件。

Author

Articles by Maja Grubic

Senior Software Engineer, Elastic

Discover in Kibana uses the fields API in 7.12