<html><head></head><body>Elasticsearch allows for many types of document mappings. An interesting mapping is the <code>_ttl</code> or <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-ttl-field.html" target="_blank">Time to Live</a> mapping.&#xA0;

This mapping allows us to set a expiry time for a document. Once the time has past, the expired documents are deleted.&#xA0;
Note that in the current version of Elasticsearch,<code>_ttl</code> is deprecated. This post will look at the current <code>_ttl</code> mapping and how you can try to avoid using them.
<h3>Let&apos;s define the mapping</h3>Let&#x2019;s say we are creating an index structure for a store called crazy shop that has regular specials that end at various time periods.
Here is an example mapping with a <code>_ttl</code> for a regular document, in this case called special
<pre class="prettyprint">POST /crazy-shop
{
 &quot;mappings&quot;: {
 &quot;special&quot;: {
 &quot;properties&quot;: {
 &quot;message&quot;: {
 &quot;type&quot;: &quot;string&quot;
 }
 },
 &quot;_ttl&quot;: {
 &quot;enabled&quot;: true,
 &quot;default&quot;: &quot;10s&quot;
 }
 }
 }
}
</pre>The <code>_ttl</code> is set to enabled and defaults to 10 seconds.
We can test this out by indexing the following document:
<pre class="prettyprint">PUT /crazy-shop/special/1
{
 &quot;message&quot; : &quot;An awesome new special&quot;
}
</pre>We can get this document with the following request:
<pre class="prettyprint">GET /crazy-shop/special/1
</pre>which will return:
<pre class="prettyprint">{
 &quot;_index&quot;: &quot;crazy-shop&quot;,
 &quot;_type&quot;: &quot;special&quot;,
 &quot;_id&quot;: &quot;1&quot;,
 &quot;_version&quot;: 1,
 &quot;found&quot;: true,
 &quot;_source&quot;: {
 &quot;message&quot;: &quot;An awesome new special&quot;
 }
}
</pre>So after 10 seconds, this document should be deleted, right? Not exactly.
<h3>How does expiring a document actually work?</h3>Elasticsearch has a background system process that will check for all <code>_ttl</code> expired documents and add them to a bulk delete request. This process by default runs every 60 seconds.
This means that setting the <code>_ttl</code> for a document to anything under 60 seconds won&#x2019;t actually delete&#xA0;the document until 60 seconds has past. This interval can be adjusted though, through the setting <code>indices.ttl.interval</code>, so you can adjust this if needed.
<h3>Delete operations are really expensive</h3>Deleting documents doesn&#x2019;t remove them from the index, but rather marks&#xA0;them as deleted and filters&#xA0;them out at query time.
Large amounts of bulk deletions can result in large Lucene index segments and in turn a large amount of data to handle until segment merges occur and deleted documents are finally removed from the index.
Refer to <a href="https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#index-segments" target="_blank">https://www.elastic.co/blog/found-elasticsearch-from-the-bottom-up#index-segments</a> for more information around Elasticsearch and Lucene indexes.
With this in mind, our advice is to try and avoid using TTL. The expense that TTL causes Elasticsearch can in most circumstances be avoided using time based indices.
The suggested re-implementation is to use a combination of two things:
<ul>
	<li>Time-based indices that contain a expiration field for highly-granular query filtering and index aliases. Moving to time-based indices significantly reduces the amount of work that&#x2019;s required to remove documents and reduces memory CPU and overhead associated with records laying around but marked deleted.</li>
	<li>Filtered queries which are extremely&#xA0;fast in Elasticsearch, as&#xA0;<a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-filters.html" target="_blank">filters are cached</a>. As time-based indices are now being used,&#xA0;there will be significantly less Lucene&#xA0;segments, due to documents not being bulk deleted. This will result in increased query performance.</li>
</ul>You may think your data model doesn&#x2019;t allow for time-based indices or there&#x2019;s some overhead around managing time-based indices. We&#x2019;ll demonstrate a modified data model below and there are already <a href="https://github.com/elastic/curator" target="_blank">great tools</a> available for handling time-based indices.
<h3>Example time based indices</h3>Continuing our crazy shop example, below is an example <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html" target="_blank">index template</a> that allows for:
<ul>
	<li>A wildcard to specify the date of the index, if we decide to have daily indexes.</li>
	<li>Mapping of fields including the expiry field that is a date type. This will be used to filter documents.</li>
</ul>An alias is also included to use a constant index name.
<pre class="prettyprint">PUT /_template/crazy-shop-template
{
 &quot;order&quot;: 0,
 &quot;template&quot;: &quot;crazy-shop-*&quot;,
 &quot;mappings&quot;: {
 &quot;special&quot;: {
 &quot;properties&quot;: {
 &quot;message&quot;: {
 &quot;type&quot;: &quot;string&quot;
 },
 &quot;expiry&quot;: {
 &quot;type&quot;: &quot;date&quot;,
 &quot;format&quot;: &quot;yyyy-MM-dd&apos;T&apos;HHmmssZ&quot;
 }
 }
 }
 },
 &quot;aliases&quot;: {
 &quot;crazy-shop&quot;: {}
 }
}
</pre>We can add some documents to an index with a date in the index name which will create a new index with the document added to it:
<pre class="prettyprint">PUT /crazy-shop-2015-08-24/special/1
{
 &quot;message&quot; : &quot;An awesome new special&quot;,
 &quot;expiry&quot; : &quot;2015-08-24T202000+1000&quot;
}
</pre><pre class="prettyprint">PUT /crazy-shop-2015-08-24/special/2
{
 &quot;message&quot; : &quot;Another awesome new special&quot;,
 &quot;expiry&quot; : &quot;2015-08-24T203000+1000&quot;
}
</pre>Next, we can query the index using a filter for the expiry field with a timestamp, which will filter out documents we don&#x2019;t need anymore.
<pre class="prettyprint">POST /crazy-shop/_search
{
 &quot;query&quot;: {
 &quot;filtered&quot;: {
 &quot;query&quot;: {
 &quot;match_all&quot;: {}
 },
 &quot;filter&quot;: {
 &quot;range&quot;: {
 &quot;expiry&quot;: {
 &quot;gte&quot;: &quot;2015-08-24T202500+1000&quot;
 }
 }
 }
 }
 }
}
</pre>We can delete the daily index manually with the following request:
<pre class="prettyprint">DELETE /crazy-shop-2015-08-24
</pre>Or use tools like <a href="https://github.com/elastic/curator" target="_blank">curator</a> to automatically delete indices.
Again, deleting the entire index is much cheaper than having lots of deleted documents in an index.
<h3>When time based indices may not work</h3>There may be times when the use of time based indices may not work for your data model.
You might require documents with a TTL to be in a parent / child relationship which requires all documents to be in the same index.
For example, hotel deals that have a parent / child relationship and the deals expire at&#xA0;various intervals.
Deleting the index in this case would remove not only the deals but the hotel documents as well.
In these situations, <code>_ttl</code> mapped documents are valid but it needs to be reminded that care should be taken of the cluster when large amounts of deleted documents occur.
Also remember that <code>_ttl</code> is deprecated and that future versions of Elasticsearch may have new implementations of <code>_ttl</code>. It would be recommended to avoid using <code>_ttl</code> and have your application handle TTL of documents if time-based indicies doesn&apos;t work for you.
<h3>How does Shield affect TTL documents?</h3>In the current version of Elasticsearch,&#xA0;if Shield is installed, <a href="https://www.elastic.co/guide/en/shield/current/limitations.html#_document_expiration__ttl" target="_blank">there is a limitation</a> that&#xA0;will affect documents with <code>_ttl</code>.
The issue here is that Shield adds privileges to documents. When Elasticsearch&#x2019;s background system action is executed to add documents with an expired <code>_ttl</code>, the bulk delete operation fails as it doesn&#x2019;t have the appropriate privileges to delete these documents. The documents therefore are not deleted and stay available in the index.
Note: This does not affect the use of time based indices as Elasticsearch is not handling a background system level deletion of any documents or indices.
</body></html>

shield-ssl-security-for-elasticsearch-full-bleed.jpg

shield-ssl-security-for-elasticsearch-thumbnail.jpeg

TTL Documents, Shield and Found

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Senior Manager, Engineering

Christian Strzadala

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Christian Strzadala

Senior Manager, Engineering, Elastic

TTL Documents, Shield and Found