<html><head></head><body>Elasticsearch allows for many types of document mappings. An interesting mapping is the <code>_ttl</code> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html" target="_blank">Time to Live</a> mapping.&#xA0;

This mapping allows us to set a expiry time for a document. Once the time has past, the expired documents are deleted.&#xA0;
Note that in the current version of Elasticsearch,<code>_ttl</code> is deprecated. This post will look at the current <code>_ttl</code> mapping and how you can try to avoid using them.
<h3>Let&apos;s define the mapping</h3>Let&#x2019;s say we are creating an index structure for a store called crazy shop that has regular specials that end at various time periods.
Here is an example mapping with a <code>_ttl</code> for a regular document, in this case called special
<pre class="prettyprint">POST /crazy-shop
{
 &quot;mappings&quot;: {
 &quot;special&quot;: {
 &quot;properties&quot;: {
 &quot;message&quot;: {
 &quot;type&quot;: &quot;string&quot;
 }
 },
 &quot;_ttl&quot;: {
 &quot;enabled&quot;: true,
 &quot;default&quot;: &quot;10s&quot;
 }
 }
 }
}
</pre>The <code>_ttl</code> is set to enabled and defaults to 10 seconds.
We can test this out by indexing the following document:
<pre class="prettyprint">PUT /crazy-shop/special/1
{
 &quot;message&quot; : &quot;An awesome new special&quot;
}
</pre>We can get this document with the following request:
<pre class="prettyprint">GET /crazy-shop/special/1
</pre>which will return:
<pre class="prettyprint">{
 &quot;_index&quot;: &quot;crazy-shop&quot;,
 &quot;_type&quot;: &quot;special&quot;,
 &quot;_id&quot;: &quot;1&quot;,
 &quot;_version&quot;: 1,
 &quot;found&quot;: true,
 &quot;_source&quot;: {
 &quot;message&quot;: &quot;An awesome new special&quot;
 }
}
</pre>So after 10 seconds, this document should be deleted, right? Not exactly.
<h3>How does expiring a document actually work?</h3>Elasticsearch has a background system process that will check for all <code>_ttl</code> expired documents and add them to a bulk delete request. This process by default runs every 60 seconds.
This means that setting the <code>_ttl</code> for a document to anything under 60 seconds won&#x2019;t actually delete&#xA0;the document until 60 seconds has past. This interval can be adjusted though, through the setting <code>indices.ttl.interval</code>, so you can adjust this if needed.
<h3>Delete operations are really expensive</h3>Deleting documents doesn&#x2019;t remove them from the index, but rather marks&#xA0;them as deleted and filters&#xA0;them out at query time.
Large amounts of bulk deletions can result in large Lucene index segments and in turn a large amount of data to handle until segment merges occur and deleted documents are finally removed from the index.
Refer to <a href="https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#index-segments" target="_blank">https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#index-segments</a> for more information around Elasticsearch and Lucene indexes.
With this in mind, our advice is to try and avoid using TTL. The expense that TTL causes Elasticsearch can in most circumstances be avoided using time based indices.
The suggested re-implementation is to use a combination of two things:
<ul>
	<li>Time-based indices that contain a expiration field for highly-granular query filtering and index aliases. Moving to time-based indices significantly reduces the amount of work that&#x2019;s required to remove documents and reduces memory CPU and overhead associated with records laying around but marked deleted.</li>
	<li>Filtered queries which are extremely&#xA0;fast in Elasticsearch, as&#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-filters.html" target="_blank">filters are cached</a>. As time-based indices are now being used,&#xA0;there will be significantly less Lucene&#xA0;segments, due to documents not being bulk deleted. This will result in increased query performance.</li>
</ul>You may think your data model doesn&#x2019;t allow for time-based indices or there&#x2019;s some overhead around managing time-based indices. We&#x2019;ll demonstrate a modified data model below and there are already <a href="https://github.com/elastic/curator" target="_blank">great tools</a> available for handling time-based indices.
<h3>Example time based indices</h3>Continuing our crazy shop example, below is an example <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html" target="_blank">index template</a> that allows for:
<ul>
	<li>A wildcard to specify the date of the index, if we decide to have daily indexes.</li>
	<li>Mapping of fields including the expiry field that is a date type. This will be used to filter documents.</li>
</ul>An alias is also included to use a constant index name.
<pre class="prettyprint">PUT /_template/crazy-shop-template
{
 &quot;order&quot;: 0,
 &quot;template&quot;: &quot;crazy-shop-*&quot;,
 &quot;mappings&quot;: {
 &quot;special&quot;: {
 &quot;properties&quot;: {
 &quot;message&quot;: {
 &quot;type&quot;: &quot;string&quot;
 },
 &quot;expiry&quot;: {
 &quot;type&quot;: &quot;date&quot;,
 &quot;format&quot;: &quot;yyyy-MM-dd&apos;T&apos;HHmmssZ&quot;
 }
 }
 }
 },
 &quot;aliases&quot;: {
 &quot;crazy-shop&quot;: {}
 }
}
</pre>We can add some documents to an index with a date in the index name which will create a new index with the document added to it:
<pre class="prettyprint">PUT /crazy-shop-2015-08-24/special/1
{
 &quot;message&quot; : &quot;An awesome new special&quot;,
 &quot;expiry&quot; : &quot;2015-08-24T202000+1000&quot;
}
</pre><pre class="prettyprint">PUT /crazy-shop-2015-08-24/special/2
{
 &quot;message&quot; : &quot;Another awesome new special&quot;,
 &quot;expiry&quot; : &quot;2015-08-24T203000+1000&quot;
}
</pre>Next, we can query the index using a filter for the expiry field with a timestamp, which will filter out documents we don&#x2019;t need anymore.
<pre class="prettyprint">POST /crazy-shop/_search
{
 &quot;query&quot;: {
 &quot;filtered&quot;: {
 &quot;query&quot;: {
 &quot;match_all&quot;: {}
 },
 &quot;filter&quot;: {
 &quot;range&quot;: {
 &quot;expiry&quot;: {
 &quot;gte&quot;: &quot;2015-08-24T202500+1000&quot;
 }
 }
 }
 }
 }
}
</pre>We can delete the daily index manually with the following request:
<pre class="prettyprint">DELETE /crazy-shop-2015-08-24
</pre>Or use tools like <a href="https://github.com/elastic/curator" target="_blank">curator</a> to automatically delete indices.
Again, deleting the entire index is much cheaper than having lots of deleted documents in an index.
<h3>When time based indices may not work</h3>There may be times when the use of time based indices may not work for your data model.
You might require documents with a TTL to be in a parent / child relationship which requires all documents to be in the same index.
For example, hotel deals that have a parent / child relationship and the deals expire at&#xA0;various intervals.
Deleting the index in this case would remove not only the deals but the hotel documents as well.
In these situations, <code>_ttl</code> mapped documents are valid but it needs to be reminded that care should be taken of the cluster when large amounts of deleted documents occur.
Also remember that <code>_ttl</code> is deprecated and that future versions of Elasticsearch may have new implementations of <code>_ttl</code>. It would be recommended to avoid using <code>_ttl</code> and have your application handle TTL of documents if time-based indicies doesn&apos;t work for you.
<h3>How does Shield affect TTL documents?</h3>In the current version of Elasticsearch,&#xA0;if Shield is installed, <a href="https://www.elastic.co/guide/en/shield/current/limitations.html#_document_expiration__ttl" target="_blank">there is a limitation</a> that&#xA0;will affect documents with <code>_ttl</code>.
The issue here is that Shield adds privileges to documents. When Elasticsearch&#x2019;s background system action is executed to add documents with an expired <code>_ttl</code>, the bulk delete operation fails as it doesn&#x2019;t have the appropriate privileges to delete these documents. The documents therefore are not deleted and stay available in the index.
Note: This does not affect the use of time based indices as Elasticsearch is not handling a background system level deletion of any documents or indices.
</body></html>

shield-ssl-security-for-elasticsearch-full-bleed.jpg

shield-ssl-security-for-elasticsearch-thumbnail.jpeg

TTL Documents, Shield and Found

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Senior Manager, Engineering

Christian Strzadala

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/elastic-cloud-account-terms" target="_self">Conditions générales d'utilisation</a>, et vous acceptez de recevoir les <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">types de notification sélectionnés</a> à l'adresse que vous avez fournie plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/serverless-preview-terms" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

En envoyant votre demande d'inscription, vous acceptez les <a href="/fr/agreements/global/cloud-services-monthly" rel="nofollow">conditions générales d'utilisation d'Elastic Cloud</a>. Vos données personnelles seront traitées conformément à la <a href="/fr/legal/privacy-statement">déclaration de confidentialité d'Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

En confirmant votre envoi, vous reconnaissez que vous avez lu et que vous acceptez nos <a href="/fr/legal/terms-of-use" target="_self">Conditions générales d'utilisation</a>, et vous permettez à Elastic de <a href="/fr/legal/privacy-statement#how-we-use-the-information" target="_self">vous contacter</a> à propos de produits et services similaires en utilisant les informations que vous avez fournies plus haut. Pour en savoir plus ou pour revenir sur votre décision (possible à tout moment), consultez la <a href="/fr/legal/privacy-statement/" target="_self">Déclaration de confidentialité d'Elastic</a>.

Explore similar demos

Overview

You'll also receive an email with related content.

You'll also receive an email with related content

Nous vous enverrons plus de contenu susceptible de vous intéresser.

Speakers

Close

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

Le contenu de cette page n'est pas disponible dans la langue sélectionnée. Chez Elastic, nous mettons tout en œuvre pour vous proposer du contenu dans différentes langues. En attendant, nous vous remercions de votre patience !

See more insights

Articles by

Author

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Load More

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Voir à quelle heure débute le webinar dans mon fuseau horaire

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Si vous ne pouvez pas vous libérer le jour J, inscrivez-vous et nous vous enverrons l'enregistrement. Vous recevrez aussi un e-mail contenant des ressources connexes.

Small image for

On-demand webinar

Featured webinar

Register to Attend

Share on Facebook

Reset all

Upcoming webinar

View more posts

Print

Hosted by

Sign In to Attend

View next

Follow us on Facebook

Share

Highlights

Read less

Thank you for registering. We will send you a confirmation email soon.

Nous vous remercions de votre inscription. Vous allez recevoir un e-mail de confirmation.

Author

Articles by Christian Strzadala

Senior Manager, Engineering, Elastic

TTL Documents, Shield and Found