Set up a time series data stream (TSDS) | Elasticsearch Guide [8.13]

› › ›

« Time series data stream (TSDS) Time series index settings »

Set up a time series data stream (TSDS)edit

To set up a time series data stream (TSDS), follow these steps:

Prerequisitesedit

Before you create a TSDS, you should be familiar with data streams and TSDS concepts.
To follow this tutorial, you must have the following permissions:
- Cluster privileges: manage_ilm and manage_index_templates.
- Index privileges: create_doc and create_index for any TSDS you create or convert. To roll over a TSDS, you must have the manage privilege.

Create an index lifecycle policyedit

While optional, we recommend using ILM to automate the management of your TSDS’s backing indices. ILM requires an index lifecycle policy.

We recommend you specify a max_age criteria for the rollover action in the policy. This ensures the @timestamp ranges for the TSDS’s backing indices are consistent. For example, setting a max_age of 1d for the rollover action ensures your backing indices consistently contain one day’s worth of data.

PUT _ilm/policy/my-weather-sensor-lifecycle-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1d",
            "max_primary_shard_size": "50gb"
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "frozen": {
        "min_age": "90d",
        "actions": {
          "searchable_snapshot": {
            "snapshot_repository": "found-snapshots"
          }
        }
      },
      "delete": {
        "min_age": "735d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Create an index templateedit

To setup a TSDS create an index template with the following details:

One or more index patterns that match the TSDS’s name. We recommend using our data stream naming scheme.
Enable data streams.
Specify a mapping that defines your dimensions and metrics:
- One or more dimension fields with a time_series_dimension value of true. At least one of these dimensions must be a plain keyword field.
- One or more metric fields, marked using the time_series_metric mapping parameter.
- Optional: A date or date_nanos mapping for the @timestamp field. If you don’t specify a mapping, Elasticsearch maps @timestamp as a date field with default options.
Define index settings:
- Set index.mode setting to time_series.
- Your lifecycle policy in the index.lifecycle.name index setting.
- Optional: Other index settings, such as index.number_of_replicas, for your TSDS’s backing indices.
A priority higher than 200 to avoid collisions with built-in templates. See Avoid index pattern collisions.
Optional: Component templates containing your mappings and other index settings.

PUT _index_template/my-weather-sensor-index-template
{
  "index_patterns": ["metrics-weather_sensors-*"],
  "data_stream": { },
  "template": {
    "settings": {
      "index.mode": "time_series",
      "index.lifecycle.name": "my-lifecycle-policy"
    },
    "mappings": {
      "properties": {
        "sensor_id": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "location": {
          "type": "keyword",
          "time_series_dimension": true
        },
        "temperature": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "humidity": {
          "type": "half_float",
          "time_series_metric": "gauge"
        },
        "@timestamp": {
          "type": "date"
        }
      }
    }
  },
  "priority": 500,
  "_meta": {
    "description": "Template for my weather sensor data"
  }
}

Create the TSDSedit

Indexing requests add documents to a TSDS. Documents in a TSDS must include:

A @timestamp field
One or more dimension fields. At least one dimension must be a keyword field that matches the index.routing_path index setting, if specified. If not specified explicitly, index.routing_path is set automatically to whichever mappings have time_series_dimension set to true.

To automatically create your TSDS, submit an indexing request that targets the TSDS’s name. This name must match one of your index template’s index patterns.

To test the following example, update the timestamps to within three hours of your current time. Data added to a TSDS must always fall within an accepted time range.

PUT metrics-weather_sensors-dev/_bulk
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:21:15.000Z", "sensor_id": "HAL-000001", "location": "plains", "temperature": 26.7,"humidity": 49.9 }
{ "create":{ } }
{ "@timestamp": "2099-05-06T16:25:42.000Z", "sensor_id": "SYKENET-000001", "location": "swamp", "temperature": 32.4, "humidity": 88.9 }

POST metrics-weather_sensors-dev/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "sensor_id": "SYKENET-000001",
  "location": "swamp",
  "temperature": 32.4,
  "humidity": 88.9
}

You can also manually create the TSDS using the create data stream API. The TSDS’s name must still match one of your template’s index patterns.

response = client.indices.create_data_stream(
  name: 'metrics-weather_sensors-dev'
)
puts response

PUT _data_stream/metrics-weather_sensors-dev

Secure the TSDSedit

Use index privileges to control access to a TSDS. Granting privileges on a TSDS grants the same privileges on its backing indices.

For an example, refer to Data stream privileges.

Convert an existing data stream to a TSDSedit

You can also use the above steps to convert an existing regular data stream to a TSDS. In this case, you’ll want to:

Edit your existing index lifecycle policy, component templates, and index templates instead of creating new ones.
Instead of creating the TSDS, manually roll over its write index. This ensures the current write index and any new backing indices have an index.mode of time_series.

You can manually roll over the write index using the rollover API.
```
response = client.indices.rollover(
  alias: 'metrics-weather_sensors-dev'
)
puts response
```
```
POST metrics-weather_sensors-dev/_rollover
```

What’s next?edit

Now that you’ve set up your TSDS, you can manage and use it like a regular data stream. For more information, refer to:

« Time series data stream (TSDS) Time series index settings »