This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.

The following limitations and known problems apply to the 7.3 release of the Elastic data frame feature:

Beta data frame transforms do not have guaranteed backwards or forwards compatibilityedit

Whilst data frame transforms are beta, it is not guaranteed that a data frame transform created in a previous version of the Elastic Stack will be able to start and operate in a future version. Neither can support be provided for data frame transform tasks to be able to operate in a cluster with mixed node versions. Please note that the output of a data frame transform is persisted to a destination index. This is a normal Elasticsearch index and is not affected by the beta status.

Data frame UI will not work during a rolling upgrade from 7.2 to 7.3edit

If your cluster contains mixed version nodes, for example during a rolling upgrade from 7.2 to 7.3, and data frame transforms have been created in 7.2, the data frame UI will not work. Please wait until all nodes have been upgraded to 7.3 before using the data frame UI.

Data frame data type limitationedit

Data frames do not (yet) support fields containing arrays – in the UI or the API. If you try to create one, the UI will fail to show the source index table.

Cross-cluster search is not supportededit

Cross-cluster search is not supported in 7.3 for data frame transforms.

Up to 1,000 data frame transforms are supportededit

A single cluster will support up to 1,000 data frame transforms. When using the GET data frame transforms API a total count of transforms is returned. Use the size and from parameters to enumerate through the full list.

Aggregation responses may be incompatible with destination index mappingsedit

When a data frame transform is first started, it will deduce the mappings required for the destination index. This process is based on the field types of the source index and the aggregations used. If the fields are derived from scripted_metrics or bucket_scripts, dynamic mappings will be used. In some instances the deduced mappings may be incompatible with the actual data. For example, numeric overflows might occur or dynamically mapped fields might contain both numbers and strings. Please check Elasticsearch logs if you think this may have occurred. As a workaround, you may define custom mappings prior to starting the data frame transform. For example, create a custom destination index or define an index template.

Batch data frame transforms may not account for changed documentsedit

A batch data frame transform uses a composite aggregation which allows efficient pagination through all buckets. Composite aggregations do not yet support a search context, therefore if the source data is changed (deleted, updated, added) while the batch data frame is in progress, then the results may not include these changes.

Continuous data frame consistency does not account for deleted or updated documentsedit

While the process for continuous data frame transforms allows the continual recalculation of the data frame transform as new data is being ingested, it does also have some limitations.

Changed entities will only be identified if their time field has also been updated and falls within the range of the action to check for changes. This has been designed in principle for, and is suited to, the use case where new data is given a timestamp for the time of ingest.

If the indices that fall within the scope of the source index pattern are removed, for example when deleting historical time-based indices, then the composite aggregation performed in consecutive checkpoint processing will search over different source data, and entities that only existed in the deleted index will not be removed from the data frame destination index.

Depending on your use case, you may wish to recreate the data frame transform entirely after deletions. Alternatively, if your use case is tolerant to historical archiving, you may wish to include a max ingest timestamp in your aggregation. This will allow you to exclude results that have not been recently updated when viewing the data frame destination index.

Deleting a data frame transform does not delete the data frame destination index or Kibana index patternedit

When deleting a data frame transform using DELETE _data_frame/transforms/index neither the data frame destination index nor the Kibana index pattern, should one have been created, are deleted. These objects must be deleted separately.

Handling dynamic adjustment of aggregation page sizeedit

During the development of data frame transforms, control was favoured over performance. In the design considerations, it is preferred for the data frame transform to take longer to complete quietly in the background rather than to finish quickly and take precedence in resource consumption.

Composite aggregations are well suited for high cardinality data enabling pagination through results. If a circuit breaker memory exception occurs when performing the composite aggregated search then we try again reducing the number of buckets requested. This circuit breaker is calculated based upon all activity within the cluster, not just activity from data frame transforms, so it therefore may only be a temporary resource availability issue.

For a batch data frame transform, the number of buckets requested is only ever adjusted downwards. The lowering of value may result in a longer duration for the transform checkpoint to complete. For continuous data frames, the number of buckets requested is reset back to its default at the start of every checkpoint and it is possible for circuit breaker exceptions to occur repeatedly in the Elasticsearch logs.

The data frame transform retrieves data in batches which means it calculates several buckets at once. Per default this is 500 buckets per search/index operation. The default can be changed using max_page_search_size and the minimum value is 10. If failures still occur once the number of buckets requested has been reduced to its minimum, then the data frame transform will be set to a failed state.

Handling dynamic adjustments for many termsedit

For each checkpoint, entities are identified that have changed since the last time the check was performed. This list of changed entities is supplied as a terms query to the data frame transform composite aggregation, one page at a time. Then updates are applied to the destination index for each page of entities.

The page size is defined by max_page_search_size which is also used to define the number of buckets returned by the composite aggregation search. The default value is 500, the minimum is 10.

The index setting index.max_terms_count defines the maximum number of terms that can be used in a terms query. The default value is 65536. If max_page_search_size exceeds index.max_terms_count the transform will fail.

Using smaller values for max_page_search_size may result in a longer duration for the transform checkpoint to complete.

Cannot update a data frame transformedit

Data frame transform configurations cannot be updated. Please delete and then create a new transform instead.

Continuous data frame scheduling limitationsedit

A continuous data frame periodically checks for changes to source data. The functionality of the scheduler is currently limited to a basic periodic timer which can be within the frequency range from 1s to 1h. The default is 1m. This is designed to run little and often. When choosing a frequency for this timer consider your ingest rate along with the impact that the data frame transform search/index operations has other users in your cluster. Also note that retries occur at frequency interval.

Handling of failed data frame transformsedit

Failed data frame transforms remain as a persistent task and should be handled appropriately, either by deleting it or by resolving the root cause of the failure and re-starting.

When using the API to delete a failed data frame transform, first stop it using _stop?force=true, then delete it.

If starting a failed data frame transform, after the root cause has been resolved, the _start?force=true parameter must be specified.

Continuous data frames may give incorrect results if documents are not yet available to searchedit

After a document is indexed, there is a very small delay until it is available to search.

A continuous data frame transform periodically checks for changed entities between the time since it last checked and now minus sync.time.delay. This time window moves without overlapping. If the timestamp of a recently indexed document falls within this time window but this document is not yet available to search then this entity will not be updated.

If using a sync.time.field that represents the data ingest time and using a zero second or very small sync.time.delay, then it is more likely that this issue will occur.