Transform limitations

This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

The following limitations and known problems apply to the 7.2 release of the Elastic data frame feature:

Data frame data type limitation

Data frames do not (yet) support fields containing arrays – in the UI or the API. If you try to create one, the UI will fail to show the source index table.

Cross-cluster search limitation

Cross-cluster search is not supported in 7.2 for transforms.

Kibana only displays up to 100 transforms

The Kibana Machine Learning > Data Frames page lists up to 100 transforms. You can work-around this limitation by calling the GET transforms API with the size parameter.

Date histogram limitation

If you use a date histogram in the group_by object in the create or preview transform APIs, the defined interval and time format must have the same time fidelity. Otherwise, it might cause issues in the data frame.

For example, if you set the calendar_interval of the date histogram to one minute (1m), then make sure that the format is yyyy-MM-dd HH:mm instead of yyyy-MM-dd HH:00.

Date format limitation in transform destination index

When you create a transform that uses a date_histogram as a group-by and set the interval to 1y, the date could be interpreted incorrectly in the generated date field of the destination index. The reason is that the yyyy value can be identified incorrectly as epoch_millis. As a workaround, using the API, you may define a custom destination index data format mapping prior to starting the transform. For example:

"mappings" : {
        "properties" : {
            "custom_date" : {
            "type" : "date",
            "format": "yyyy"
            }
          }
        }

Aggregation responses may be incompatible with destination index mappings

Data frames use composite aggregations to transform data. In some cases, composite aggregations may return responses which are not compatible with the mappings set for the destination index. For example "NaN", "Infinity" or possibly a numeric overflow. Where possible, a null response has been substituted. Please, check Elasticsearch logs if you think this may have occurred. As a workaround, using the API, you may define custom destination index mappings prior to starting the transform.