Create a transform
Added in 7.2.0
Creates a transform.
A transform copies data from source indices, transforms it, and persists it into an entity-centric destination index. You can also think of the destination index as a two-dimensional tabular data structure (known as a data frame). The ID for each document in the data frame is generated from a hash of the entity, so there is a unique row per entity.
You must choose either the latest or pivot method for your transform; you cannot use both in a single transform. If
you choose to use the pivot method for your transform, the entities are defined by the set of group_by fields in
the pivot object. If you choose to use the latest method, the entities are defined by the unique_key field values
in the latest object.
You must have create_index, index, and read privileges on the destination index and read and
view_index_metadata privileges on the source indices. When Elasticsearch security features are enabled, the
transform remembers which roles the user that created it had at the time of creation and uses those same roles. If
those roles do not have the required privileges on the source and destination indices, the transform fails when it
attempts unauthorized operations.
NOTE: You must use Kibana or this API to create a transform. Do not add a transform directly into any
.transform-internal* indices using the Elasticsearch index API. If Elasticsearch security features are enabled, do
not give users any privileges on .transform-internal* indices. If you used transforms prior to 7.5, also do not
give users any privileges on .data-frame-internal* indices.
Path parameters
-
transform_id
string Required Identifier for the transform. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It has a 64 character limit and must start and end with alphanumeric characters.
Query parameters
-
defer_validation
boolean When the transform is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use this parameter to skip the checks, for example when the source index does not exist until after the transform is created. The validations are always run when you start the transform, however, with the exception of privilege checks.
-
timeout
string Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Body
Required
-
dest
object Required Additional properties are allowed.
-
description
string Free text description of the transform.
-
frequency
string A duration. Units can be
nanos,micros,ms(milliseconds),s(seconds),m(minutes),h(hours) andd(days). Also accepts "0" without a unit and "-1" to indicate an unspecified value. -
latest
object Additional properties are allowed.
-
_meta
object -
pivot
object Additional properties are allowed.
-
retention_policy
object Additional properties are allowed.
-
settings
object Additional properties are allowed.
-
source
object Required Additional properties are allowed.
-
sync
object Additional properties are allowed.
curl \
--request PUT http://api.example.com/_transform/{transform_id} \
--header "Authorization: $API_KEY" \
--header "Content-Type: application/json" \
--data '{"dest":{"index":"kibana_sample_data_ecommerce_transform1","pipeline":"add_timestamp_pipeline"},"sync":{"time":{"delay":"60s","field":"order_date"}},"pivot":{"group_by":{"customer_id":{"terms":{"field":"customer_id","missing_bucket":true}}},"aggregations":{"max_price":{"max":{"field":"taxful_total_price"}}}},"source":{"index":"kibana_sample_data_ecommerce","query":{"term":{"geoip.continent_name":{"value":"Asia"}}}},"frequency":"5m","description":"Maximum priced ecommerce data by customer_id in Asia","retention_policy":{"time":{"field":"order_date","max_age":"30d"}}}'{
"dest": {
"index": "kibana_sample_data_ecommerce_transform1",
"pipeline": "add_timestamp_pipeline"
},
"sync": {
"time": {
"delay": "60s",
"field": "order_date"
}
},
"pivot": {
"group_by": {
"customer_id": {
"terms": {
"field": "customer_id",
"missing_bucket": true
}
}
},
"aggregations": {
"max_price": {
"max": {
"field": "taxful_total_price"
}
}
}
},
"source": {
"index": "kibana_sample_data_ecommerce",
"query": {
"term": {
"geoip.continent_name": {
"value": "Asia"
}
}
}
},
"frequency": "5m",
"description": "Maximum priced ecommerce data by customer_id in Asia",
"retention_policy": {
"time": {
"field": "order_date",
"max_age": "30d"
}
}
}{
"dest": {
"index": "kibana_sample_data_ecommerce_transform2"
},
"sync": {
"time": {
"delay": "60s",
"field": "order_date"
}
},
"latest": {
"sort": "order_date",
"unique_key": [
"customer_id"
]
},
"source": {
"index": "kibana_sample_data_ecommerce"
},
"frequency": "5m",
"description": "Latest order for each customer"
}{
"acknowledged": true
}