To improve the simplicity of retrieving fields, Discover now uses the fields API by default. Learn more about how you can benefit from the new search parameter.

With Elastic 7.12, Discover now uses the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param"><code>fields</code> API</a> by default. Reading from <code>_source</code> is still supported through a switch in the Advanced Settings. This change stems from updates made to Elasticsearch in 7.11 with the extension of the Search API to include the new <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param"><code>fields</code></a> parameter.
When using the new search parameter, both a document’s raw source and the index mappings to load and return values are used. Because it makes use of the mappings, <code>fields</code> has some advantages over referencing the <code>_source</code> directly: it accepts multifields and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/field-alias.html">field aliases</a>, and also formats field values like dates in a consistent way.
In short, the fields option simplifies all the nuances of <code>_source</code> and other places&nbsp;by dealing with multi-mapping and aliases, type coercion, docvalues, and all the other edge cases.
<h2>What are the benefits of the new search parameter?</h2>More and more use cases arise where Elasticsearch is treated as a data store. In order to improve the simplicity of retrieving fields without the expectation of expertise in areas like mappings, doc_values, stored fields, etc., we decided to extend the existing API.&nbsp;
<h3>Some benefits of the new parameter:</h3><ul>
	<li aria-level="1">If a non-standard field like a field alias, multi-field, or constant_keyword is specified in <code>fields</code>, mappings will find and return the right value.</li>
	<li aria-level="1">The fields are returned in a flat list, as opposed to structured JSON.</li>
	<li aria-level="1">Each value would be returned in a 'canonical' format — for example if a field is mapped as an integer, it will be returned as an integer even if it was specified as a string in the <code>_source</code>.</li>
</ul><h2>What should I expect?</h2>The most meaningful change is the support for runtime fields. When a runtime field is defined in a mapping, it will show up in Discover like any other field type. The next sections expand on all of the improvements and changes.
<h3>Default column is now called Document</h3>If there are no columns configured, the default column changes from <code>_source</code> to <code>Document</code>.
Existing saved searches with <code>_source</code> column will be changed to <code>Document</code>. Unlike source, the <code>Document</code> column cannot be combined with other columns in the grid.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta9496688ad999005/60639a1c9a1d7d1050bd3839/discover-fields-api-document-columns.png" data-sys-asset-uid="blta9496688ad999005" alt="discover-fields-api-document-columns.png">
Although it is formatted similar to the <code>_source</code> column, it is a collection of fields as returned from an Elasticsearch response and therefore, combining it with other columns would not make much sense.
<h3>Multi-fields are grouped together</h3>Multi-field is the field mapped differently for different purposes. For instance, a string field could be mapped as a text field for full-text search, and as a keyword field for sorting or aggregations.
Due to the changes in the API, multi-sub fields will now have values where before they didn’t. In order to avoid suddenly increasing the number of fields that are shown, we’ve decided to group multi-sub fields under the root field so they are still usable as columns, but not as prominent.&nbsp;&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt99224476ee765750/60639aaebc3ffc1037a403fa/discover-fields-api-multi-fields.png" data-sys-asset-uid="blt99224476ee765750" alt="discover-fields-api-multi-fields.png">
<h3>Differences in how object fields are displayed</h3>Object fields usually consist of one or more leaf fields. Example of an object field:
<pre class="prettyprint">"manager": { 
 "age": 30, 
 "name": { 
 "first": "John", 
 "last": "Smith" 
 } 
 }
</pre>Before the introduction of the fields API, Discover would show the object roots, together with the leaf fields.
With the introduction of the fields API, object roots won't be shown in the field list anymore (unless there is a previously saved search containing one as a column). Leaf fields will now be correctly detected as having data, where previously they were not. This will be the case even when the document has an array of objects. Leaf fields will work for all objects, and the values of all objects in that array will be shown flattened.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc7782faad0e4eff0/60639ad325c15e102fb9a032/discover-fields-api-object-fields.png" data-sys-asset-uid="bltc7782faad0e4eff0" alt="discover-fields-api-object-fields.png">
<h3>Differences in how nested fields are displayed</h3>The nested data type is used for indexing arrays of objects when there is a need to maintain the independence of each object in the array. They need to be specified as a type in the mapping, otherwise they would be treated as <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/object.html"><code>object</code></a> data type.
Before the introduction of the fields API, Discover would show a root field if at least one document loaded has an array of objects for that field. Otherwise, the root field wouldn’t be shown. The leaf fields will be shown if they appear in at least one document. Leaf fields that do not appear in any document or only in documents that have arrays of objects for this object field, will not appear in the field list unless "Hide missing fields" is switched off.
In the table, root fields as columns will show the JSON of the fields’ content ONLY if that document contains an array of objects and will be empty otherwise.&nbsp;
Let’s say we have a mapping as follows:
<pre class="prettyprint">"products": {
 "type": "nested",
 "properties": {
 "name": { "type": "keyword" },
 "price": { "type": "double" }
 }
 },
</pre>And the following documents:
<pre class="prettyprint">POST /discover_test/_doc
{
"products": [
 { "name": "Kibana", "price": 42.23 },
 { "name": "Faber-Castell Polychromos", "price": 29.95 }
 ],
}
POST /discover_test/_doc
{
 "products": { "name": "Product name", "price": 123.21 },
}
</pre>In this case, when using <code>_source</code>, Discover would only correctly display the JSON of the products object in the first document since it is an array. The second document, however, is a single object, meaning it would display as empty:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt98ab2b26c766472d/60639b475a03370f176cfebc/discover-fields-api-document-1.png" data-sys-asset-uid="blt98ab2b26c766472d" alt="discover-fields-api-document-1.png">
Leaf fields as columns will only show values if the document did not contain an array of objects and stay empty otherwise:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb7f04a8dbc41e1f9/60639bbf389f150ff9aa4539/discover-fields-api-document-2.png" data-sys-asset-uid="bltb7f04a8dbc41e1f9" alt="discover-fields-api-document-2.png">
With the introduction of the fields API, the root of a nested field can be added from the field list and will show the structured JSON of that field in the document. Nested field columns now show the JSON correctly for all documents:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltc5bd634b92a55c98/60639bca4f68540feec10164/discover-fields-api-document-3.png" data-sys-asset-uid="bltc5bd634b92a55c98" alt="discover-fields-api-document-3.png">
In the case of an existing saved search with a nested leaf field as a column, this will show as empty now:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2547ad02430a0445/60639bd74f68540feec10168/discover-fields-api-document-4.png" data-sys-asset-uid="blt2547ad02430a0445" alt="discover-fields-api-document-4.png">
We want to have this behavior going forward since we assume the nested mapping type is only used when an array of objects is indexed.
<h3>Unmapped fields won’t show up if field filters are configured</h3>When a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping (unless <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/dynamic.html">dynamic</a> parameter is set to false). Discover will continue showing these fields as it did previously, except in the case when field filters are configured. In that case, unmapped fields will not show up.
<h2>Limitations of _source</h2>Source contains the raw information from the document, exactly as it was ingested. This means users lose the benefit of features built on top of <code>_source</code>, like runtime fields, multi-fields, date formatting, and alias fields. Also, the whole <code>_source</code> object must be loaded and parsed even if only a small number of fields are requested.
With the introduction of the <code>fields</code> parameter, developers can retrieve the field values after the logic from these features is applied. This abstraction removes the need for developers to know implementation details about a field. It doesn’t matter if the field is runtime or not, it will always be accessed in the same way.
Here is an example. Let’s say we have a mapping defined as follows:
<pre class="prettyprint">PUT /my-index
{
 "mappings": {
 "properties": {
 "xlong": {
 "type": "long"
 },
 "xfloat": {
 "type": "float"
 }
 }
 }
}
POST my-index/_doc/1
{
 "xlong": "1.5",
 "Xfloat": "1.5"
}
POST my-index/_doc/2
{
 "Xlong": "2",
 "Xfloat": "2"
}
</pre>The field xlong is mapped as long, which means that internally its value will be rounded towards zero. In that case, 1.5 becomes 1. When executing the following search in Discover,
<pre class="prettyprint">xlong &gt; 1 and xlong &lt; 2
</pre>the search will not return any values, which is expected. However, when viewing documents in Discover, when reading fields from <code>_source</code>, the value will be displayed exactly as it was ingested, ie. 1.5:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0f94d4d4de89e7bf/60639c95dfb49b0f38b4cc14/discover-fields-api-document-ingest-1.png" data-sys-asset-uid="blt0f94d4d4de89e7bf" alt="discover-fields-api-document-ingest-1.png">
This used to create a lot of confusion. With the fields option enabled, the actual mapped value (1)&nbsp; is displayed:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbbab80ab6123c318/60639cb3573d281045d35d5b/discover-fields-api-document-ingest-2.png" data-sys-asset-uid="bltbbab80ab6123c318" alt="discover-fields-api-document-ingest-2.png">
<h2>Wrap up</h2>We are excited to bring the power and flexibility of runtime fields to Discover, including the switch to using the fields API. If you are currently not on Elastic 7.12, but would still like to test out these updates, starting a free <a href="https://www.elastic.co/elasticsearch/service">Elasticsearch Service trial on Elastic Cloud</a> is a fast and easy way to do so.&nbsp;
To learn more about other recent product updates, be sure to read our <a href="https://www.elastic.co/blog/whats-new-elastic-7-12-0-schema-on-read-frozen-tier-autoscaling">Elastic 7.12 release blog</a>.&nbsp;

blog-banner-release-kibana.png

blog-thumb-release-kibana.png

Discover in Kibana uses the fields API in 7.12

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Senior Software Engineer

Maja Grubic

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、上記で提供された連絡先に<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">特定の通知</a>が送信されることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシーポリシー</a>をご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。このフォームを送信することにより、Elasticが私の連絡先情報をGoogle Cloudに共有することに同意します。また、Elasticが私に食材パッケージを発送する目的で私の情報をベンダーに提供することにも同意します。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

送信することにより、Elasticの<a href="/jp/legal/serverless-preview-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

送信することにより、<a href="/jp/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud利用規約</a>に同意したものとみなされます。個人情報は<a href="/jp/legal/privacy-statement">Elasticのプライバシーポリシー</a>に基づいて取り扱われます。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

送信することにより、Elasticの<a href="/jp/legal/terms-of-use" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

このページのコンテンツは、選択された言語ではご利用いただけません。Elasticでは現在、コンテンツをさまざまな言語でご利用いただけるよう取り組んでいます。お使いの言語が準備されるまでお待ちくださるようお願いいたします。

Author

Articles by

Watch now (no PT)

Watch now

See all top stories

すべてのトップストーリーを見る

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Search Integrations

All Solutions

Video for

See when this webinar starts in my time zone

現在のタイムゾーンでこのウェビナー開始時間を見る

Register to Watch

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

他の予定と重なってしまった…という場合も、問題ありません。参加登録されると、録画配信をご利用いただけます。関連コンテンツのご案内もメールでお届けします。

Author

Articles by Maja Grubic

Senior Software Engineer, Elastic

Discover in Kibana uses the fields API in 7.12