Set up a time series data stream (TSDS) | Elasticsearch Guide [master]

You are looking at preliminary documentation for a future release. Not what you want? See the current release documentation.

› › ›

« Time series data stream (TSDS) Time series index settings »

Set up a time series data stream (TSDS)edit

To set up a time series data stream (TSDS), follow these steps:

Prerequisitesedit

Before you create a TSDS, you should be familiar with data streams and TSDS concepts.
To follow this tutorial, you must have the following permissions:
- Cluster privileges: manage_ilm and manage_index_templates.
- Index privileges: create_doc and create_index for any TSDS you create or convert. To roll over a TSDS, you must have the manage privilege.

Create an index lifecycle policyedit

While optional, we recommend using ILM to automate the management of your TSDS’s backing indices. ILM requires an index lifecycle policy.

We recommend you specify a max_age criteria for the rollover action in the policy. This ensures the @timestamp ranges for the TSDS’s backing indices are consistent. For example, setting a max_age of 1d for the rollover action ensures your backing indices consistently contain one day’s worth of data.

PUT _ilm/policy/my-weather-sensor-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Create a mappings component templateedit

A TSDS requires a matching index template. In most cases, you compose this index template using one or more component templates. You typically use separate component templates for mappings and index settings. This lets you reuse the component templates in multiple index templates.

For a TSDS, the mappings component template must include mappings for:

One or more dimension fields with a time_series_dimension value of true. At least one of these dimensions must be a plain keyword field.

Optionally, the template can also include mappings for:

One or more metric fields, marked using the time_series_metric mapping parameter.
A date or date_nanos mapping for the @timestamp field. If you don’t specify a mapping, Elasticsearch maps @timestamp as a date field with default options.

response = client.cluster.put_component_template(
  name: 'my-weather-sensor-mappings',
  body: {
    template: {
      mappings: {
        properties: {
          sensor_id: {
            type: 'keyword',
            time_series_dimension: true
          },
          location: {
            type: 'keyword',
            time_series_dimension: true
          },
          temperature: {
            type: 'half_float',
            time_series_metric: 'gauge'
          },
          humidity: {
            type: 'half_float',
            time_series_metric: 'gauge'
          },
          "@timestamp": {
            type: 'date',
            format: 'strict_date_optional_time'
          }
        }
      }
    },
    _meta: {
      description: 'Mappings for weather sensor data'
    }
  }
)
puts response

PUT _component_template/my-weather-sensor-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "sensor_id": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "location": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "temperature": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "humidity": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "@timestamp": {
          "type": "date",
          "format": "strict_date_optional_time"
        }
      }
    }
  },
  "_meta": {
    "description": "Mappings for weather sensor data"
  }
}

Create an index settings component templateedit

Optionally, the index settings component template for a TSDS can include:

Your lifecycle policy in the index.lifecycle.name index setting.
Other index settings, such as index.number_of_replicas, for your TSDS’s backing indices.

Don’t specify the index.routing_path index setting in a component template. If you choose, you can configure index.routing_path directly in the index template, as shown in the following step.

response = client.cluster.put_component_template(
  name: 'my-weather-sensor-settings',
  body: {
    template: {
      settings: {
        'index.lifecycle.name' => 'my-lifecycle-policy'
      }
    },
    _meta: {
      description: 'Index settings for weather sensor data'
    }
  }
)
puts response

PUT _component_template/my-weather-sensor-settings
{
  "template": {
    "settings": {
      "index.lifecycle.name": "my-lifecycle-policy"
    }
  },
  "_meta": {
    "description": "Index settings for weather sensor data"
  }
}

Create an index templateedit

Use your component templates to create an index template. In the index template, specify:

One or more index patterns that match the TSDS’s name. We recommend using our data stream naming scheme.
That the template is data stream enabled.
An index.mode object set to time_series.
Optional: The index.routing_path index setting. The setting value should only match plain keyword dimension fields and should be set directly in the index template. When not defined explicitly, the index.routing_path setting is generated from the mapping using all mappings that are set with time_series_dimension set to true.
The component templates containing your mappings and other index settings.
A priority higher than 200 to avoid collisions with built-in templates. See Avoid index pattern collisions.

response = client.indices.put_index_template(
  name: 'my-weather-sensor-index-template',
  body: {
    index_patterns: [
      'metrics-weather_sensors-*'
    ],
    data_stream: {},
    template: {
      settings: {
        'index.mode' => 'time_series',
        'index.routing_path' => [
          'sensor_id',
          'location'
        ]
      }
    },
    composed_of: [
      'my-weather-sensor-mappings',
      'my-weather-sensor-settings'
    ],
    priority: 500,
    _meta: {
      description: 'Template for my weather sensor data'
    }
  }
)
puts response

PUT _index_template/my-weather-sensor-index-template
{
  "index_patterns": ["metrics-weather_sensors-*"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.routing_path": [ "sensor_id", "location" ]
    }
  },
  "composed_of": [ "my-weather-sensor-mappings", "my-weather-sensor-settings" ],
  "priority": 500,
  "_meta": {
    "description": "Template for my weather sensor data"
  }
}

Create the TSDSedit

Indexing requests add documents to a TSDS. Documents in a TSDS must include:

A @timestamp field
One or more dimension fields. At least one dimension must be a keyword field that matches the index.routing_path index setting, if specified. If not specified explicitly, index.routing_path is set automatically to whichever mappings have time_series_dimension set to true.

To automatically create your TSDS, submit an indexing request that targets the TSDS’s name. This name must match one of your index template’s index patterns.

To test the following example, update the timestamps to within three hours of your current time. Data added to a TSDS must always fall within an accepted time range.

PUT metrics-weather_sensors-dev/_bulk
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "HAL-000001", "location": "plains", "temperature": 26.7,"humidity": 49.9 }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 }

POST metrics-weather_sensors-dev/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "sensor_id": "SYKENET-000001",
  "location": "swamp",
  "temperature": 32.4,
  "humidity": 88.9
}

You can also manually create the TSDS using the create data stream API. The TSDS’s name must still match one of your template’s index patterns.

response = client.indices.create_data_stream(
  name: 'metrics-weather_sensors-dev'
)
puts response

PUT _data_stream/metrics-weather_sensors-dev

Secure the TSDSedit

Use index privileges to control access to a TSDS. Granting privileges on a TSDS grants the same privileges on its backing indices.

For an example, refer to Data stream privileges.

Convert an existing data stream to a TSDSedit

You can also use the above steps to convert an existing regular data stream to a TSDS. In this case, you’ll want to:

Edit your existing index lifecycle policy, component templates, and index templates instead of creating new ones.
Instead of creating the TSDS, manually roll over its write index. This ensures the current write index and any new backing indices have an index.mode of time_series.

You can manually roll over the write index using the rollover API.
```
response = client.indices.rollover(
  alias: 'metrics-weather_sensors-dev'
)
puts response
```
```
POST metrics-weather_sensors-dev/_rollover
```

What’s next?edit

Now that you’ve set up your TSDS, you can manage and use it like a regular data stream. For more information, refer to:

« Time series data stream (TSDS) Time series index settings »