<html><head></head><body>Elasticsearch allows for many types of document mappings. An interesting mapping is the <code>_ttl</code> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html" target="_blank">Time to Live</a> mapping.&#xA0;

This mapping allows us to set a expiry time for a document. Once the time has past, the expired documents are deleted.&#xA0;
Note that in the current version of Elasticsearch,<code>_ttl</code> is deprecated. This post will look at the current <code>_ttl</code> mapping and how you can try to avoid using them.
<h3>Let&apos;s define the mapping</h3>Let&#x2019;s say we are creating an index structure for a store called crazy shop that has regular specials that end at various time periods.
Here is an example mapping with a <code>_ttl</code> for a regular document, in this case called special
<pre class="prettyprint">POST /crazy-shop
{
 &quot;mappings&quot;: {
 &quot;special&quot;: {
 &quot;properties&quot;: {
 &quot;message&quot;: {
 &quot;type&quot;: &quot;string&quot;
 }
 },
 &quot;_ttl&quot;: {
 &quot;enabled&quot;: true,
 &quot;default&quot;: &quot;10s&quot;
 }
 }
 }
}
</pre>The <code>_ttl</code> is set to enabled and defaults to 10 seconds.
We can test this out by indexing the following document:
<pre class="prettyprint">PUT /crazy-shop/special/1
{
 &quot;message&quot; : &quot;An awesome new special&quot;
}
</pre>We can get this document with the following request:
<pre class="prettyprint">GET /crazy-shop/special/1
</pre>which will return:
<pre class="prettyprint">{
 &quot;_index&quot;: &quot;crazy-shop&quot;,
 &quot;_type&quot;: &quot;special&quot;,
 &quot;_id&quot;: &quot;1&quot;,
 &quot;_version&quot;: 1,
 &quot;found&quot;: true,
 &quot;_source&quot;: {
 &quot;message&quot;: &quot;An awesome new special&quot;
 }
}
</pre>So after 10 seconds, this document should be deleted, right? Not exactly.
<h3>How does expiring a document actually work?</h3>Elasticsearch has a background system process that will check for all <code>_ttl</code> expired documents and add them to a bulk delete request. This process by default runs every 60 seconds.
This means that setting the <code>_ttl</code> for a document to anything under 60 seconds won&#x2019;t actually delete&#xA0;the document until 60 seconds has past. This interval can be adjusted though, through the setting <code>indices.ttl.interval</code>, so you can adjust this if needed.
<h3>Delete operations are really expensive</h3>Deleting documents doesn&#x2019;t remove them from the index, but rather marks&#xA0;them as deleted and filters&#xA0;them out at query time.
Large amounts of bulk deletions can result in large Lucene index segments and in turn a large amount of data to handle until segment merges occur and deleted documents are finally removed from the index.
Refer to <a href="https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#index-segments" target="_blank">https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#index-segments</a> for more information around Elasticsearch and Lucene indexes.
With this in mind, our advice is to try and avoid using TTL. The expense that TTL causes Elasticsearch can in most circumstances be avoided using time based indices.
The suggested re-implementation is to use a combination of two things:
<ul>
	<li>Time-based indices that contain a expiration field for highly-granular query filtering and index aliases. Moving to time-based indices significantly reduces the amount of work that&#x2019;s required to remove documents and reduces memory CPU and overhead associated with records laying around but marked deleted.</li>
	<li>Filtered queries which are extremely&#xA0;fast in Elasticsearch, as&#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-filters.html" target="_blank">filters are cached</a>. As time-based indices are now being used,&#xA0;there will be significantly less Lucene&#xA0;segments, due to documents not being bulk deleted. This will result in increased query performance.</li>
</ul>You may think your data model doesn&#x2019;t allow for time-based indices or there&#x2019;s some overhead around managing time-based indices. We&#x2019;ll demonstrate a modified data model below and there are already <a href="https://github.com/elastic/curator" target="_blank">great tools</a> available for handling time-based indices.
<h3>Example time based indices</h3>Continuing our crazy shop example, below is an example <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html" target="_blank">index template</a> that allows for:
<ul>
	<li>A wildcard to specify the date of the index, if we decide to have daily indexes.</li>
	<li>Mapping of fields including the expiry field that is a date type. This will be used to filter documents.</li>
</ul>An alias is also included to use a constant index name.
<pre class="prettyprint">PUT /_template/crazy-shop-template
{
 &quot;order&quot;: 0,
 &quot;template&quot;: &quot;crazy-shop-*&quot;,
 &quot;mappings&quot;: {
 &quot;special&quot;: {
 &quot;properties&quot;: {
 &quot;message&quot;: {
 &quot;type&quot;: &quot;string&quot;
 },
 &quot;expiry&quot;: {
 &quot;type&quot;: &quot;date&quot;,
 &quot;format&quot;: &quot;yyyy-MM-dd&apos;T&apos;HHmmssZ&quot;
 }
 }
 }
 },
 &quot;aliases&quot;: {
 &quot;crazy-shop&quot;: {}
 }
}
</pre>We can add some documents to an index with a date in the index name which will create a new index with the document added to it:
<pre class="prettyprint">PUT /crazy-shop-2015-08-24/special/1
{
 &quot;message&quot; : &quot;An awesome new special&quot;,
 &quot;expiry&quot; : &quot;2015-08-24T202000+1000&quot;
}
</pre><pre class="prettyprint">PUT /crazy-shop-2015-08-24/special/2
{
 &quot;message&quot; : &quot;Another awesome new special&quot;,
 &quot;expiry&quot; : &quot;2015-08-24T203000+1000&quot;
}
</pre>Next, we can query the index using a filter for the expiry field with a timestamp, which will filter out documents we don&#x2019;t need anymore.
<pre class="prettyprint">POST /crazy-shop/_search
{
 &quot;query&quot;: {
 &quot;filtered&quot;: {
 &quot;query&quot;: {
 &quot;match_all&quot;: {}
 },
 &quot;filter&quot;: {
 &quot;range&quot;: {
 &quot;expiry&quot;: {
 &quot;gte&quot;: &quot;2015-08-24T202500+1000&quot;
 }
 }
 }
 }
 }
}
</pre>We can delete the daily index manually with the following request:
<pre class="prettyprint">DELETE /crazy-shop-2015-08-24
</pre>Or use tools like <a href="https://github.com/elastic/curator" target="_blank">curator</a> to automatically delete indices.
Again, deleting the entire index is much cheaper than having lots of deleted documents in an index.
<h3>When time based indices may not work</h3>There may be times when the use of time based indices may not work for your data model.
You might require documents with a TTL to be in a parent / child relationship which requires all documents to be in the same index.
For example, hotel deals that have a parent / child relationship and the deals expire at&#xA0;various intervals.
Deleting the index in this case would remove not only the deals but the hotel documents as well.
In these situations, <code>_ttl</code> mapped documents are valid but it needs to be reminded that care should be taken of the cluster when large amounts of deleted documents occur.
Also remember that <code>_ttl</code> is deprecated and that future versions of Elasticsearch may have new implementations of <code>_ttl</code>. It would be recommended to avoid using <code>_ttl</code> and have your application handle TTL of documents if time-based indicies doesn&apos;t work for you.
<h3>How does Shield affect TTL documents?</h3>In the current version of Elasticsearch,&#xA0;if Shield is installed, <a href="https://www.elastic.co/guide/en/shield/current/limitations.html#_document_expiration__ttl" target="_blank">there is a limitation</a> that&#xA0;will affect documents with <code>_ttl</code>.
The issue here is that Shield adds privileges to documents. When Elasticsearch&#x2019;s background system action is executed to add documents with an expired <code>_ttl</code>, the bulk delete operation fails as it doesn&#x2019;t have the appropriate privileges to delete these documents. The documents therefore are not deleted and stay available in the index.
Note: This does not affect the use of time based indices as Elasticsearch is not handling a background system level deletion of any documents or indices.
</body></html>

shield-ssl-security-for-elasticsearch-full-bleed.jpg

shield-ssl-security-for-elasticsearch-thumbnail.jpeg

TTL Documents, Shield and Found

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Senior Manager, Engineering

Christian Strzadala

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、上記で提供された連絡先に<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">特定の通知</a>が送信されることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシーポリシー</a>をご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

送信することにより、Elasticの<a href="/jp/legal/elastic-cloud-account-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。このフォームを送信することにより、Elasticが私の連絡先情報をGoogle Cloudに共有することに同意します。また、Elasticが私に食材パッケージを発送する目的で私の情報をベンダーに提供することにも同意します。

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

送信することにより、Elasticの<a href="/jp/legal/serverless-preview-terms" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

送信することにより、<a href="/jp/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud利用規約</a>に同意したものとみなされます。個人情報は<a href="/jp/legal/privacy-statement">Elasticのプライバシーポリシー</a>に基づいて取り扱われます。

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

送信することにより、Elasticの<a href="/jp/legal/terms-of-use" target="_self">利用規約</a>を読み、同意したものと見なされます。また、関連する製品やサービスについて、上記で提供された情報を使用してElasticから<a href="/jp/legal/privacy-statement#how-we-use-the-information" target="_self">連絡する</a>場合があることを認めたことになります。詳細については、<a href="/jp/legal/privacy-statement/" target="_self">Elasticのプライバシー</a>ポリシーをご覧ください。これらの同意についてはいつでもオプトアウトすることが可能です。

Explore similar demos

Overview

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

このページのコンテンツは、選択された言語ではご利用いただけません。Elasticでは現在、コンテンツをさまざまな言語でご利用いただけるよう取り組んでいます。お使いの言語が準備されるまでお待ちくださるようお願いいたします。

Author

Articles by

Watch now (no PT)

Watch now

See all top stories

すべてのトップストーリーを見る

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Search Integrations

All Solutions

Video for

See when this webinar starts in my time zone

現在のタイムゾーンでこのウェビナー開始時間を見る

Register to Watch

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

他の予定と重なってしまった…という場合も、問題ありません。参加登録されると、録画配信をご利用いただけます。関連コンテンツのご案内もメールでお届けします。

Author

Articles by Christian Strzadala

Senior Manager, Engineering, Elastic

TTL Documents, Shield and Found