POST /_transform/{transform_id}/_update

Updates certain properties of a transform.

All updated properties except description do not take effect until after the transform starts the next checkpoint, thus there is data consistency in each checkpoint. To use this API, you must have read and view_index_metadata privileges for the source indices. You must also have index and read privileges for the destination index. When Elasticsearch security features are enabled, the transform remembers which roles the user who updated it had at the time of update and runs with those privileges.

Required authorization

  • Index privileges: read,index,view_index_metadata
  • Cluster privileges: manage_transform

Path parameters

  • transform_id string Required

    Identifier for the transform.

Query parameters

  • defer_validation boolean

    When true, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the transform is created.

  • timeout string

    Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.

    Values are -1 or 0.

    External documentation
application/json

Body Required

  • dest object

    The destination for the transform.

    Hide dest attributes Show dest attributes object
    • index string

      The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the create index API prior to starting the transform.

    • pipeline string

      The unique identifier for an ingest pipeline.

  • description string

    Free text description of the transform.

  • frequency string

    The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is 1s and the maximum is 1h.

    External documentation
  • _meta object

    Defines optional transform metadata.

    Hide _meta attribute Show _meta attribute object
    • * object Additional properties
  • source object

    The source of the data for the transform.

    Hide source attributes Show source attributes object
    • index string | array[string] Required

      The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*""), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"]. For remote indices use the syntax "remote_name:index_name". If any indices are in remote clusters then the master node and at least one transform node must have the remote_cluster_client node role.

    • query object

      A query clause that retrieves a subset of data from the source index.

      External documentation
      Hide query attributes Show query attributes object
      • bool object
      • boosting object
      • common object Deprecated
      • combined_fields object
      • constant_score object
      • dis_max object
      • distance_feature
      • exists object
      • function_score object
      • fuzzy object

        Returns documents that contain terms similar to the search term, as measured by a Levenshtein edit distance.

        External documentation
      • geo_bounding_box object
      • geo_distance object
      • geo_grid object

        Matches geo_point and geo_shape values that intersect a grid cell from a GeoGrid aggregation.

      • geo_polygon object
      • geo_shape object
      • has_child object
      • has_parent object
      • ids object
      • intervals object

        Returns documents based on the order and proximity of matching terms.

        External documentation
      • knn object
      • match object

        Returns documents that match a provided text, number, date or boolean value. The provided text is analyzed before matching.

        External documentation
      • match_all object
      • match_bool_prefix object

        Analyzes its input and constructs a bool query from the terms. Each term except the last is used in a term query. The last term is used in a prefix query.

        External documentation
      • match_none object
      • match_phrase object

        Analyzes the text and creates a phrase query out of the analyzed text.

        External documentation
      • match_phrase_prefix object

        Returns documents that contain the words of a provided text, in the same order as provided. The last term of the provided text is treated as a prefix, matching any words that begin with that term.

        External documentation
      • more_like_this object
      • multi_match object
      • nested object
      • parent_id object
      • percolate object
      • prefix object

        Returns documents that contain a specific prefix in a provided field.

        External documentation
      • query_string object
      • range object

        Returns documents that contain terms within a provided range.

        External documentation
      • rank_feature object
      • regexp object

        Returns documents that contain terms matching a regular expression.

        External documentation
      • rule object
      • script object
      • script_score object
      • semantic object
      • shape object
      • simple_query_string object
      • span_containing object
      • span_field_masking object
      • span_first object
      • span_multi object
      • span_near object
      • span_not object
      • span_or object
      • span_term object

        Matches spans containing a term.

        External documentation
      • span_within object
      • term object

        Returns documents that contain an exact term in a provided field. To return a document, the query term must exactly match the queried field's value, including whitespace and capitalization.

        External documentation
      • terms object
      • terms_set object

        Returns documents that contain a minimum number of exact terms in a provided field. To return a document, a required number of terms must exactly match the field values, including whitespace and capitalization.

        External documentation
      • text_expansion object Deprecated Generally available

        Uses a natural language processing model to convert the query text into a list of token-weight pairs which are then used in a query against a sparse vector or rank features field.

        External documentation
      • weighted_tokens object Deprecated Generally available

        Supports returning text_expansion query results by sending in precomputed tokens with the query.

        External documentation
      • wildcard object

        Returns documents that contain terms matching a wildcard pattern.

        External documentation
      • wrapper object
      • type object
    • runtime_mappings object Generally available

      Definitions of search-time runtime fields that can be used by the transform. For search runtime fields all data nodes, including remote nodes, must be 7.12 or later.

      Hide runtime_mappings attribute Show runtime_mappings attribute object
      • * object Additional properties
        Hide * attributes Show * attributes object
        • fields object

          For type composite

          Hide fields attribute Show fields attribute object
          • * object Additional properties
        • fetch_fields array[object]

          For type lookup

        • format string

          A custom format for date type runtime fields.

        • input_field string

          For type lookup

        • target_field string

          For type lookup

        • target_index string

          For type lookup

        • script object

          Painless script executed at query time.

        • type string Required

          Field type, which can be: boolean, composite, date, double, geo_point, ip,keyword, long, or lookup.

          Values are boolean, composite, date, double, geo_point, geo_shape, ip, keyword, long, or lookup.

  • settings object

    Defines optional transform settings.

    Hide settings attributes Show settings attributes object
    • align_checkpoints boolean

      Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

      Default value is true.

    • dates_as_epoch_millis boolean

      Defines if dates in the ouput should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

      Default value is false.

    • deduce_mappings boolean

      Specifies whether the transform should deduce the destination index mappings from the transform configuration.

      Default value is true.

    • docs_per_second number

      Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

    • max_page_search_size number

      Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536.

      Default value is 500.0.

    • use_point_in_time boolean

      Specifies whether the transform checkpoint will use the Point In Time API while searching over the source index. In general, Point In Time is an optimization that will reduce pressure on the source index by reducing the amount of refreshes and merges, but it can be expensive if a large number of Point In Times are opened and closed for a given index. The benefits and impact depend on the data being searched, the ingest rate into the source index, and the amount of other consumers searching the same source index.

      Default value is true.

      External documentation
    • num_failure_retries number Generally available

      Defines the number of retries on a recoverable failure before the transform task is marked as failed. The minimum value is 0 and the maximum is 100, where -1 indicates that the transform retries indefinitely. If unset, the cluster-level setting num_transform_failure_retries is used.

      This setting cannot be specified when unattended is true, because unattended transforms always retry indefinitely.

    • unattended boolean Generally available

      If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

      Default value is false.

  • sync object

    Defines the properties transforms require to run continuously.

    Hide sync attribute Show sync attribute object
    • time object

      Specifies that the transform uses a time field to synchronize the source and destination indices.

      Hide time attributes Show time attributes object
      • delay string

        The time delay between the current time and the latest input data time.

        External documentation
      • field string Required

        The date field that is used to identify new documents in the source. In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

  • retention_policy object | string | null

    Defines a retention policy for the transform. Data that meets the defined criteria is deleted from the destination index.

    One of:

Responses

  • 200 application/json
    Hide response attributes Show response attributes object
    • authorization object
      Hide authorization attributes Show authorization attributes object
      • api_key object

        If an API key was used for the most recent update to the transform, its name and identifier are listed in the response.

        Hide api_key attributes Show api_key attributes object
        • id string Required

          The identifier for the API key.

        • name string Required

          The name of the API key.

      • roles array[string]

        If a user ID was used for the most recent update to the transform, its roles at the time of the update are listed in the response.

      • service_account string

        If a service account was used for the most recent update to the transform, the account name is listed in the response.

    • create_time number Required
    • description string Required
    • dest object Required
      Hide dest attributes Show dest attributes object
      • index string Required

        The name of the data stream, index, or index alias you are copying to.

      • op_type string

        If it is create, the operation will only index documents that do not already exist (also known as "put if absent").

        IMPORTANT: To reindex to a data stream destination, this argument must be create.

        Values are index or create.

      • pipeline string

        The name of the pipeline to use.

      • routing string

        By default, a document's routing is preserved unless it's changed by the script. If it is keep, the routing on the bulk request sent for each match is set to the routing on the match. If it is discard, the routing on the bulk request sent for each match is set to null. If it is =value, the routing on the bulk request sent for each match is set to all value specified after the equals sign (=).

        Default value is keep.

      • version_type string

        The versioning to use for the indexing operation.

        Values are internal, external, or external_gte.

    • frequency string

      A duration. Units can be nanos, micros, ms (milliseconds), s (seconds), m (minutes), h (hours) and d (days). Also accepts "0" without a unit and "-1" to indicate an unspecified value.

      External documentation
    • id string Required
    • latest object
      Hide latest attributes Show latest attributes object
      • sort string Required

        Specifies the date field that is used to identify the latest documents.

      • unique_key array[string] Required

        Specifies an array of one or more fields that are used to group the data.

    • pivot object
      Hide pivot attributes Show pivot attributes object
      • aggregations object

        Defines how to aggregate the grouped data. The following aggregations are currently supported: average, bucket script, bucket selector, cardinality, filter, geo bounds, geo centroid, geo line, max, median absolute deviation, min, missing, percentiles, rare terms, scripted metric, stats, sum, terms, top metrics, value count, weighted average.

      • group_by object

        Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported: date histogram, geotile grid, histogram, terms.

        Hide group_by attribute Show group_by attribute object
        • * object Additional properties
    • retention_policy object
      Hide retention_policy attribute Show retention_policy attribute object
      • time object

        Specifies that the transform uses a time field to set the retention policy.

        Hide time attributes Show time attributes object
        • field string Required

          The date field that is used to calculate the age of the document.

        • max_age string Required

          Specifies the maximum age of a document in the destination index. Documents that are older than the configured value are removed from the destination index.

    • settings object Required

      The source of the data for the transform.

      Hide settings attributes Show settings attributes object
      • align_checkpoints boolean

        Specifies whether the transform checkpoint ranges should be optimized for performance. Such optimization can align checkpoint ranges with the date histogram interval when date histogram is specified as a group source in the transform config. As a result, less document updates in the destination index will be performed thus improving overall performance.

        Default value is true.

      • dates_as_epoch_millis boolean

        Defines if dates in the ouput should be written as ISO formatted string or as millis since epoch. epoch_millis was the default for transforms created before version 7.11. For compatible output set this value to true.

        Default value is false.

      • deduce_mappings boolean

        Specifies whether the transform should deduce the destination index mappings from the transform configuration.

        Default value is true.

      • docs_per_second number

        Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.

      • max_page_search_size number

        Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 65,536.

        Default value is 500.0.

      • use_point_in_time boolean

        Specifies whether the transform checkpoint will use the Point In Time API while searching over the source index. In general, Point In Time is an optimization that will reduce pressure on the source index by reducing the amount of refreshes and merges, but it can be expensive if a large number of Point In Times are opened and closed for a given index. The benefits and impact depend on the data being searched, the ingest rate into the source index, and the amount of other consumers searching the same source index.

        Default value is true.

        External documentation
      • num_failure_retries number Generally available

        Defines the number of retries on a recoverable failure before the transform task is marked as failed. The minimum value is 0 and the maximum is 100, where -1 indicates that the transform retries indefinitely. If unset, the cluster-level setting num_transform_failure_retries is used.

        This setting cannot be specified when unattended is true, because unattended transforms always retry indefinitely.

      • unattended boolean Generally available

        If true, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails. Setting the number of retries other than infinite fails in validation.

        Default value is false.

    • source object Required
      Hide source attributes Show source attributes object
      • index string | array[string] Required

        The name of the data stream, index, or alias you are copying from. It accepts a comma-separated list to reindex from multiple sources.

      • query object

        The documents to reindex, which is defined with Query DSL.

        External documentation
        Hide query attributes Show query attributes object
        • common object Deprecated
        • distance_feature
        • fuzzy object

          Returns documents that contain terms similar to the search term, as measured by a Levenshtein edit distance.

          External documentation
        • geo_grid object

          Matches geo_point and geo_shape values that intersect a grid cell from a GeoGrid aggregation.

        • intervals object

          Returns documents based on the order and proximity of matching terms.

          External documentation
        • match object

          Returns documents that match a provided text, number, date or boolean value. The provided text is analyzed before matching.

          External documentation
        • match_bool_prefix object

          Analyzes its input and constructs a bool query from the terms. Each term except the last is used in a term query. The last term is used in a prefix query.

          External documentation
        • match_phrase object

          Analyzes the text and creates a phrase query out of the analyzed text.

          External documentation
        • match_phrase_prefix object

          Returns documents that contain the words of a provided text, in the same order as provided. The last term of the provided text is treated as a prefix, matching any words that begin with that term.

          External documentation
        • prefix object

          Returns documents that contain a specific prefix in a provided field.

          External documentation
        • range object

          Returns documents that contain terms within a provided range.

          External documentation
        • regexp object

          Returns documents that contain terms matching a regular expression.

          External documentation
        • span_term object

          Matches spans containing a term.

          External documentation
        • term object

          Returns documents that contain an exact term in a provided field. To return a document, the query term must exactly match the queried field's value, including whitespace and capitalization.

          External documentation
        • terms_set object

          Returns documents that contain a minimum number of exact terms in a provided field. To return a document, a required number of terms must exactly match the field values, including whitespace and capitalization.

          External documentation
        • text_expansion object Deprecated Generally available

          Uses a natural language processing model to convert the query text into a list of token-weight pairs which are then used in a query against a sparse vector or rank features field.

          External documentation
        • weighted_tokens object Deprecated Generally available

          Supports returning text_expansion query results by sending in precomputed tokens with the query.

          External documentation
        • wildcard object

          Returns documents that contain terms matching a wildcard pattern.

          External documentation
      • remote object Technical preview

        A remote instance of Elasticsearch that you want to index from.

        Hide remote attributes Show remote attributes object
        • connect_timeout string

          The remote connection timeout.

        • headers object

          An object containing the headers of the request.

          Hide headers attribute Show headers attribute object
          • * string Additional properties
        • host string Required

          The URL for the remote instance of Elasticsearch that you want to index from. This information is required when you're indexing from remote.

        • username string

          The username to use for authentication with the remote host (required when using basic auth).

        • password string

          The password to use for authentication with the remote host (required when using basic auth).

        • api_key string Generally available

          The API key to use for authentication with the remote host (as an alternative to basic auth when the remote cluster is in Elastic Cloud). (It is not permitted to set this and also to set an Authorization header via headers.)

        • socket_timeout string

          The remote socket read timeout.

      • size number

        The number of documents to index per batch. Use it when you are indexing from remote to ensure that the batches fit within the on-heap buffer, which defaults to a maximum size of 100 MB.

        Default value is 1000.0.

      • slice object

        Slice the reindex request manually using the provided slice ID and total number of slices.

        Hide slice attributes Show slice attributes object
        • field string

          Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

        • id string Required
        • max number Required
      • sort string | object | array[string | object]

        A comma-separated list of <field>:<direction> pairs to sort by before indexing. Use it in conjunction with max_docs to control what documents are reindexed.

        WARNING: Sort in reindex is deprecated. Sorting in reindex was never guaranteed to index documents in order and prevents further development of reindex such as resilience and performance improvements. If used in combination with max_docs, consider using a query filter instead.

        One of:

        Path to field or array of paths. Some API's support wildcards in the path to select multiple fields.

      • _source boolean | object

        If true, reindex all source fields. Set it to a list to reindex select fields.

        One of:
      • runtime_mappings object
        Hide runtime_mappings attribute Show runtime_mappings attribute object
        • * object Additional properties
          Hide * attributes Show * attributes object
          • fields object

            For type composite

          • fetch_fields array[object]

            For type lookup

          • format string

            A custom format for date type runtime fields.

    • sync object
      Hide sync attribute Show sync attribute object
      • time object

        Specifies that the transform uses a time field to synchronize the source and destination indices.

        Hide time attributes Show time attributes object
        • delay string

          The time delay between the current time and the latest input data time.

        • field string Required

          The date field that is used to identify new documents in the source. In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

    • version string Required
    • _meta object
      Hide _meta attribute Show _meta attribute object
      • * object Additional properties
POST /_transform/{transform_id}/_update
curl \
 --request POST 'http://api.example.com/_transform/{transform_id}/_update' \
 --header "Content-Type: application/json" \
 --data '"{\n  \"source\": {\n    \"index\": \"kibana_sample_data_ecommerce\",\n    \"query\": {\n      \"term\": {\n        \"geoip.continent_name\": {\n          \"value\": \"Asia\"\n        }\n      }\n    }\n  },\n  \"description\": \"Maximum priced ecommerce data by customer_id in Asia\",\n  \"dest\": {\n    \"index\": \"kibana_sample_data_ecommerce_transform_v2\",\n    \"pipeline\": \"add_timestamp_pipeline\"\n  },\n  \"frequency\": \"15m\",\n  \"sync\": {\n    \"time\": {\n      \"field\": \"order_date\",\n      \"delay\": \"120s\"\n    }\n  }\n}"'
Request example
Run `POST _transform/simple-kibana-ecomm-pivot/_update` to update a transform that uses the pivot method.
{
  "source": {
    "index": "kibana_sample_data_ecommerce",
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform_v2",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "15m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "120s"
    }
  }
}
Response examples (200)
A successful response when creating a transform.
{
  "id": "simple-kibana-ecomm-pivot",
  "authorization": {
    "roles": [
      "superuser"
    ]
  },
  "version": "10.0.0",
  "create_time": 1712951576767,
  "source": {
    "index": [
      "kibana_sample_data_ecommerce"
    ],
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform_v2",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "15m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "120s"
    }
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id",
          "missing_bucket": true
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "settings": {},
  "retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "30d"
    }
  }
}