Changes made using this API take effect immediately.
Period to wait for a connection to the master node. If no response is received before the timeout expires, the request fails and returns an error.
Period to wait for a response. If no response is received before the timeout expires, the request fails and returns an error.
Required version for optimistic concurrency control for pipeline updates
Optional metadata about the ingest pipeline. May have any contents. This map is not automatically generated by Elasticsearch.
Description of the ingest pipeline.
Processors to run immediately after a processor failure. Each processor supports a processor-level on_failure value. If a processor without an on_failure value fails, Elasticsearch uses this pipeline-level parameter as a fallback. The processors in this parameter run sequentially in the order specified. Elasticsearch will not attempt to run the pipeline's remaining processors.
Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be appended to. Supports template snippets.
The origin field which will be appended to field, cannot set value simultaneously.
If false, the processor does not append values already present in the field.
Default value is true.
The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to get the base64 encoded field from.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The number of chars being used for extraction to prevent huge fields.
Use -1 for no limit.
Default value is 100000.
Field name from which you can overwrite the number of chars being used for extraction.
Array of properties to select to be stored.
Can be content, title, name, author, keywords, date, content_type, content_length, language.
The field that will hold the attachment information.
If true, the binary field will be removed from the document
Default value is false.
Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.
Converts a human readable byte value (for example 1kb) to its value in bytes (for example 1024).
If the field is an array of strings, all members of the array will be converted.
Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive.
An error will occur if the field is not a supported format or resultant value exceeds 263.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to convert.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
Converts circle definitions of shapes to regular polygons which approximate them.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).
The field to interpret as a circle. Either a string in WKT format or a map for GeoJSON.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Which field mapping type is to be used when processing the circle: geo_shape or shape.
Values are geo_shape or shape.
The field to assign the polygon shape to By default, the field is updated in-place.
Computes the Community ID for network flow data as defined in the Community ID Specification. You can use a community ID to correlate network events related to a single flow.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the source IP address.
Field containing the source port.
Field containing the destination IP address.
Field containing the destination port.
Field containing the IANA number.
Field containing the ICMP type.
Field containing the ICMP code.
Field containing the transport protocol name or number. Used only when the iana_number field is not present. The following protocol names are currently supported: eigrp, gre, icmp, icmpv6, igmp, ipv6-icmp, ospf, pim, sctp, tcp, udp
Output field for the community ID.
Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.
Default value is 0.
If true and any required fields are missing, the processor quietly exits without modifying the document.
Default value is true.
Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field whose value is to be converted.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to.
By default, the field is updated in-place.
The type to convert the existing value to.
Values are integer, long, double, float, boolean, ip, string, or auto.
Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Value used to fill empty fields.
Empty fields are skipped if this is not provided.
An empty field is one with no value (2 consecutive separators) or empty quotes ("").
The field to extract data from.
If true and field does not exist, the processor quietly exits without modifying the document.
Quote used in CSV, has to be single character string.
Default value is ".
Separator used in CSV, has to be single character string.
Default value is ,.
The array of fields to assign extracted values to.
Trim whitespaces in unquoted fields.
Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to get the date from.
An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.
Default value is ENGLISH.
The field that will hold the parsed date.
The timezone to use when parsing the date. Supports template snippets.
Default value is UTC.
The format to use when writing the date to target_field. Must be a valid java time pattern.
Default value is yyyy-MM-dd'T'HH:mm:ss.SSSXXX.
The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
How to round the date when formatting the date into the index name. Valid values are:
y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second).
Supports template snippets.
The field to get the date or timestamp from.
The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.
Default value is yyyy-MM-dd.
A prefix of the index name to be prepended before the printed date. Supports template snippets.
The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
Default value is ENGLISH.
The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
Default value is UTC.
Extracts structured fields out of a single text field by matching the text field against a delimiter-based pattern.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The character(s) that separate the appended fields.
Default value is "".
The field to dissect.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The pattern to apply to the field.
Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to expand into an object field.
If set to *, all top-level fields will be expanded.
Controls the behavior when there is already an existing nested object that conflicts with the expanded field.
When false, the processor will merge conflicts by combining the old and the new values into an array.
When true, the value from the expanded field will overwrite the existing value.
Default value is false.
The field that contains the field to expand.
Only required if the field to expand is part another object field, because the field option can only understand leaf fields.
Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The enrich processor can enrich documents with data from another index.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field in the input document that matches the policies match_field used to retrieve the enrichment data. Supports template snippets.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The maximum number of matched documents to include under the configured target field.
The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object.
In order to avoid documents getting too large, the maximum allowed value is 128.
Default value is 1.
If processor will update fields with pre-existing non-null-valued field.
When set to false, such fields will not be touched.
Default value is true.
The name of the enrich policy to use.
A spatial relation operator used to match the geoshape of incoming documents to documents in the enrich index.
This option is only used for geo_match enrich policy types.
Supported values include:
intersects: Return all documents whose geo_shape or geo_point field intersects the query geometry.disjoint: Return all documents whose geo_shape or geo_point field has nothing in common with the query geometry.within: Return all documents whose geo_shape or geo_point field is within the query geometry.
Line geometries are not supported.contains: Return all documents whose geo_shape or geo_point field contains the query geometry.Values are intersects, disjoint, within, or contains.
Field added to incoming documents to contain enrich data. This field contains both the match_field and enrich_fields specified in the enrich policy.
Supports template snippets.
Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The error message thrown by the processor. Supports template snippets.
Computes a hash of the document’s content. You can use this hash for content fingerprinting.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Array of fields to include in the fingerprint. For objects, the processor hashes both the field key and value. For other fields, the processor hashes only the field value.
Output field for the fingerprint.
Salt value for the hash function.
The hash method used to compute the fingerprint. Must be one of MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.
Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.
If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.
Default value is false.
Runs an ingest processor on each element of an array or object.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing array or object values.
If true, the processor silently exits without changing the document if the field is null or missing.
Default value is false.
Ingest processor to run on each element.
Currently an undocumented alias for GeoIP Processor.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.
Default value is GeoLite2-City.mmdb.
The field to get the ip address from for the geographical lookup.
If true, only the first found IP location data will be returned, even if the field contains an array.
Default value is true.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Controls what properties are added to the target_field based on the IP location lookup.
The field that will hold the geographical information looked up from the MaxMind database.
If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created.
Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.
Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape. This is useful if there is a need to interact with the tile shapes as spatially indexable fields.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to interpret as a geo-tile.=
The field format is determined by the tile_type.
Three tile formats are understood: geohash, geotile and geohex.
Values are geotile, geohex, or geohash.
The field to assign the polygon shape to, by default, the field is updated in-place.
If specified and a parent tile exists, save that tile address to this field.
If specified and children tiles exist, save those tile addresses to this field as an array of strings.
If specified and intersecting non-child tiles exist, save their addresses to this field as an array of strings.
If specified, save the tile precision (zoom) as an integer to this field.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Which format to save the generated polygon in.
Values are geojson or wkt.
The geoip processor adds information about the geographical location of an IPv4 or IPv6 address.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.
Default value is GeoLite2-City.mmdb.
The field to get the ip address from for the geographical lookup.
If true, only the first found geoip data will be returned, even if the field contains an array.
Default value is true.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Controls what properties are added to the target_field based on the geoip lookup.
The field that will hold the geographical information looked up from the MaxMind database.
If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created.
Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.
Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.
Default value is disabled.
The field to use for grok expression parsing.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.
An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.
When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.
Default value is false.
Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to apply the replacement to.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The pattern to be replaced.
The string to replace the matching patterns with.
The field to assign the converted value to
By default, the field is updated in-place.
Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The string-valued field to remove HTML tags from.
If true and field does not exist or is null, the processor quietly exits without modifying the document,
Default value is false.
The field to assign the converted value to
By default, the field is updated in-place.
Uses a pre-trained data frame analytics model or a model deployed for natural language processing tasks to infer against the data that is being ingested in the pipeline.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The ID or alias for the trained model, or the ID of the deployment.
Field added to incoming documents to contain results objects.
Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.
Contains the inference type and its options.
If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.
Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing array values to join.
The separator character.
The field to assign the joined value to. By default, the field is updated in-place.
Converts a JSON string into a structured JSON object.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Flag that forces the parsed JSON to be added at the top level of the document.
target_field must not be set when this option is chosen.
Default value is false.
When set to replace, root fields that conflict with fields from the parsed JSON will be overridden.
When set to merge, conflicting fields will be merged.
Only applicable if add_to_root is set to true.
Supported values include:
replace: Root fields that conflict with fields from the parsed JSON will be overridden.merge: Conflicting fields will be merged.Values are replace or merge.
When set to true, the JSON parser will not fail if the JSON contains duplicate keys.
Instead, the last encountered value for any duplicate key wins.
Default value is false.
The field to be parsed.
The field that the converted structured object will be written into. Any existing content in this field will be overwritten.
This processor helps automatically parse messages (or specific event fields) which are of the foo=bar variety.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
List of keys to exclude from document.
The field to be parsed. Supports template snippets.
Regex pattern to use for splitting key-value pairs.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
List of keys to filter and insert into document. Defaults to including all keys.
Prefix to be added to extracted keys.
Default value is null.
If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.
Default value is false.
The field to insert the extracted keys into. Defaults to the root of the document. Supports template snippets.
String of characters to trim from extracted keys.
String of characters to trim from extracted values.
Regex pattern to use for splitting the key from the value within a key-value pair.
Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to make lowercase.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the source IP address.
Field containing the destination IP address.
Output field for the network direction.
List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.
A field on the given document to read the internal_networks configuration from.
If true and any required fields are missing, the processor quietly exits without modifying the document.
Default value is true.
Executes another pipeline.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The name of the pipeline to execute. Supports template snippets.
Whether to ignore missing pipelines instead of failing.
Default value is false.
The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns.
The processor can be used to obscure Personal Identifying Information (PII) by configuring it to detect known patterns such as email or IP addresses.
Text that matches a Grok pattern is replaced with a configurable string such as <EMAIL> where an email address is matched or simply replace all matches with the text <REDACTED> if preferred.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be redacted
A list of grok expressions to match and redact named captures with
Start a redacted section with this token
Default value is <.
End a redacted section with this token
Default value is >.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document
Default value is false.
If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted
Default value is false.
Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN). Uses the registered domains defined in the Mozilla Public Suffix List.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the source FQDN.
Object field containing extracted domain components. If an empty string, the processor adds components to the document’s root.
If true and any required fields are missing, the processor quietly exits without modifying the document.
Default value is true.
Removes existing fields. If one field doesn’t exist, an exception will be thrown.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Fields to be removed. Supports template snippets.
Fields to be kept. When set, all fields other than those specified are removed.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be renamed. Supports template snippets.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The new name of the field. Supports template snippets.
Routes a document to another target index or data stream.
When setting the destination option, the target is explicitly specified and the dataset and namespace options can’t be set.
When the destination option is not set, this processor is in a data stream mode. Note that in this mode, the reroute processor can only be used on data streams that follow the data stream naming scheme.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
A static value for the target. Can’t be set when the dataset or namespace option is set.
Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.
Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.
default {{data_stream.dataset}}
Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.
Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.
default {{data_stream.namespace}}
Runs an inline or stored script on incoming documents.
The script runs in the ingest context.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
ID of a stored script.
If no source is specified, this parameter is required.
Object containing parameters for the script.
Adds a field with the specified value. If the field already exists, its value will be replaced with the provided one.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The origin field which will be copied to field, cannot set value simultaneously.
Supported data types are boolean, number, array, object, string, date, etc.
The field to insert, upsert, or update. Supports template snippets.
If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.
Default value is false.
The media type for encoding value.
Applies only when value is a template snippet.
Must be one of application/json, text/plain, or application/x-www-form-urlencoded.
If true processor will update fields with pre-existing non-null-valued field.
When set to false, such fields will not be touched.
Default value is true.
The value to be set for the field.
Supports template snippets.
May specify only one of value or copy_from.
Sets user-related details (such as username, roles, email, full_name, metadata, api_key, realm and authentication_type) from the current authenticated user to the current document by pre-processing the ingest.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to store the user information into.
Controls what user related properties are added to the field.
Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be sorted.
The sort order to use.
Accepts "asc" or "desc".
Supported values include:
asc: Ascending (smallest to largest)desc: Descending (largest to smallest)Values are asc or desc.
The field to assign the sorted value to. By default, the field is updated in-place.
Splits a field into an array using a separator character. Only works on string fields.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to split.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Preserves empty trailing fields, if any.
Default value is false.
A regex which matches the separator, for example, , or \s+.
The field to assign the split value to. By default, the field is updated in-place.
Terminates the current ingest pipeline, causing no further processors to be run.
This will normally be executed conditionally, using the if option.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Trims whitespace from a field. If the field is an array of strings, all members of the array will be trimmed. This only works on leading and trailing whitespace.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The string-valued field to trim whitespace from.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the trimmed value to. By default, the field is updated in-place.
Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to make uppercase.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to decode.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
Parses a Uniform Resource Identifier (URI) string and extracts its components as an object. This URI object includes properties for the URI’s domain, path, fragment, port, query, scheme, user info, username, and password.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the URI string.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
If true, the processor copies the unparsed URI to <target_field>.original.
Default value is true.
If true, the processor removes the field after parsing the URI string.
If parsing fails, the processor does not remove the field.
Default value is false.
Output field for the URI object.
The user_agent processor extracts details from the user agent string a browser sends with its web requests.
This processor adds this information by default under the user_agent field.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field containing the user agent string.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.
The field that will be filled with the user agent details.
Controls what properties are added to target_field.
Values are name, os, device, original, or version. Default value is ["name", "major", "minor", "patch", "build", "os", "os_name", "os_major", "os_minor", "device"].
Extracts device type from the user agent string on a best-effort basis.
Default value is false.
Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.
Appends one or more values to an existing array if the field already exists and it is an array. Converts a scalar to an array and appends one or more values to it if the field exists and it is a scalar. Creates an array containing the provided values if the field doesn’t exist. Accepts a single value or an array of values.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be appended to. Supports template snippets.
The origin field which will be appended to field, cannot set value simultaneously.
If false, the processor does not append values already present in the field.
Default value is true.
The attachment processor lets Elasticsearch extract file attachments in common formats (such as PPT, XLS, and PDF) by using the Apache text extraction library Tika.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to get the base64 encoded field from.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The number of chars being used for extraction to prevent huge fields.
Use -1 for no limit.
Default value is 100000.
Field name from which you can overwrite the number of chars being used for extraction.
Array of properties to select to be stored.
Can be content, title, name, author, keywords, date, content_type, content_length, language.
The field that will hold the attachment information.
If true, the binary field will be removed from the document
Default value is false.
Field containing the name of the resource to decode. If specified, the processor passes this resource name to the underlying Tika library to enable Resource Name Based Detection.
Converts a human readable byte value (for example 1kb) to its value in bytes (for example 1024).
If the field is an array of strings, all members of the array will be converted.
Supported human readable units are "b", "kb", "mb", "gb", "tb", "pb" case insensitive.
An error will occur if the field is not a supported format or resultant value exceeds 263.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to convert.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
Converts circle definitions of shapes to regular polygons which approximate them.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The difference between the resulting inscribed distance from center to side and the circle’s radius (measured in meters for geo_shape, unit-less for shape).
The field to interpret as a circle. Either a string in WKT format or a map for GeoJSON.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Which field mapping type is to be used when processing the circle: geo_shape or shape.
Values are geo_shape or shape.
The field to assign the polygon shape to By default, the field is updated in-place.
Computes the Community ID for network flow data as defined in the Community ID Specification. You can use a community ID to correlate network events related to a single flow.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the source IP address.
Field containing the source port.
Field containing the destination IP address.
Field containing the destination port.
Field containing the IANA number.
Field containing the ICMP type.
Field containing the ICMP code.
Field containing the transport protocol name or number. Used only when the iana_number field is not present. The following protocol names are currently supported: eigrp, gre, icmp, icmpv6, igmp, ipv6-icmp, ospf, pim, sctp, tcp, udp
Output field for the community ID.
Seed for the community ID hash. Must be between 0 and 65535 (inclusive). The seed can prevent hash collisions between network domains, such as a staging and production network that use the same addressing scheme.
Default value is 0.
If true and any required fields are missing, the processor quietly exits without modifying the document.
Default value is true.
Converts a field in the currently ingested document to a different type, such as converting a string to an integer. If the field value is an array, all members will be converted.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field whose value is to be converted.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to.
By default, the field is updated in-place.
The type to convert the existing value to.
Values are integer, long, double, float, boolean, ip, string, or auto.
Extracts fields from CSV line out of a single text field within a document. Any empty field in CSV will be skipped.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Value used to fill empty fields.
Empty fields are skipped if this is not provided.
An empty field is one with no value (2 consecutive separators) or empty quotes ("").
The field to extract data from.
If true and field does not exist, the processor quietly exits without modifying the document.
Quote used in CSV, has to be single character string.
Default value is ".
Separator used in CSV, has to be single character string.
Default value is ,.
The array of fields to assign extracted values to.
Trim whitespaces in unquoted fields.
Parses dates from fields, and then uses the date or timestamp as the timestamp for the document.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to get the date from.
An array of the expected date formats. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
The locale to use when parsing the date, relevant when parsing month names or week days. Supports template snippets.
Default value is ENGLISH.
The field that will hold the parsed date.
The timezone to use when parsing the date. Supports template snippets.
Default value is UTC.
The format to use when writing the date to target_field. Must be a valid java time pattern.
Default value is yyyy-MM-dd'T'HH:mm:ss.SSSXXX.
The purpose of this processor is to point documents to the right time based index based on a date or timestamp field in a document by using the date math index name support.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
How to round the date when formatting the date into the index name. Valid values are:
y (year), M (month), w (week), d (day), h (hour), m (minute) and s (second).
Supports template snippets.
The field to get the date or timestamp from.
The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports template snippets.
Default value is yyyy-MM-dd.
A prefix of the index name to be prepended before the printed date. Supports template snippets.
The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
Default value is ENGLISH.
The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
Default value is UTC.
Extracts structured fields out of a single text field by matching the text field against a delimiter-based pattern.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The character(s) that separate the appended fields.
Default value is "".
The field to dissect.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The pattern to apply to the field.
Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline. Otherwise these fields can’t be accessed by any processor.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to expand into an object field.
If set to *, all top-level fields will be expanded.
Controls the behavior when there is already an existing nested object that conflicts with the expanded field.
When false, the processor will merge conflicts by combining the old and the new values into an array.
When true, the value from the expanded field will overwrite the existing value.
Default value is false.
The field that contains the field to expand.
Only required if the field to expand is part another object field, because the field option can only understand leaf fields.
Drops the document without raising any errors. This is useful to prevent the document from getting indexed based on some condition.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The enrich processor can enrich documents with data from another index.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field in the input document that matches the policies match_field used to retrieve the enrichment data. Supports template snippets.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The maximum number of matched documents to include under the configured target field.
The target_field will be turned into a json array if max_matches is higher than 1, otherwise target_field will become a json object.
In order to avoid documents getting too large, the maximum allowed value is 128.
Default value is 1.
If processor will update fields with pre-existing non-null-valued field.
When set to false, such fields will not be touched.
Default value is true.
The name of the enrich policy to use.
A spatial relation operator used to match the geoshape of incoming documents to documents in the enrich index.
This option is only used for geo_match enrich policy types.
Supported values include:
intersects: Return all documents whose geo_shape or geo_point field intersects the query geometry.disjoint: Return all documents whose geo_shape or geo_point field has nothing in common with the query geometry.within: Return all documents whose geo_shape or geo_point field is within the query geometry.
Line geometries are not supported.contains: Return all documents whose geo_shape or geo_point field contains the query geometry.Values are intersects, disjoint, within, or contains.
Field added to incoming documents to contain enrich data. This field contains both the match_field and enrich_fields specified in the enrich policy.
Supports template snippets.
Raises an exception. This is useful for when you expect a pipeline to fail and want to relay a specific message to the requester.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The error message thrown by the processor. Supports template snippets.
Computes a hash of the document’s content. You can use this hash for content fingerprinting.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Array of fields to include in the fingerprint. For objects, the processor hashes both the field key and value. For other fields, the processor hashes only the field value.
Output field for the fingerprint.
Salt value for the hash function.
The hash method used to compute the fingerprint. Must be one of MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.
Values are MD5, SHA-1, SHA-256, SHA-512, or MurmurHash3.
If true, the processor ignores any missing fields. If all fields are missing, the processor silently exits without modifying the document.
Default value is false.
Runs an ingest processor on each element of an array or object.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing array or object values.
If true, the processor silently exits without changing the document if the field is null or missing.
Default value is false.
Ingest processor to run on each element.
Currently an undocumented alias for GeoIP Processor.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.
Default value is GeoLite2-City.mmdb.
The field to get the ip address from for the geographical lookup.
If true, only the first found IP location data will be returned, even if the field contains an array.
Default value is true.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Controls what properties are added to the target_field based on the IP location lookup.
The field that will hold the geographical information looked up from the MaxMind database.
If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created.
Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.
Converts geo-grid definitions of grid tiles or cells to regular bounding boxes or polygons which describe their shape. This is useful if there is a need to interact with the tile shapes as spatially indexable fields.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to interpret as a geo-tile.=
The field format is determined by the tile_type.
Three tile formats are understood: geohash, geotile and geohex.
Values are geotile, geohex, or geohash.
The field to assign the polygon shape to, by default, the field is updated in-place.
If specified and a parent tile exists, save that tile address to this field.
If specified and children tiles exist, save those tile addresses to this field as an array of strings.
If specified and intersecting non-child tiles exist, save their addresses to this field as an array of strings.
If specified, save the tile precision (zoom) as an integer to this field.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Which format to save the generated polygon in.
Values are geojson or wkt.
The geoip processor adds information about the geographical location of an IPv4 or IPv6 address.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The database filename referring to a database the module ships with (GeoLite2-City.mmdb, GeoLite2-Country.mmdb, or GeoLite2-ASN.mmdb) or a custom database in the ingest-geoip config directory.
Default value is GeoLite2-City.mmdb.
The field to get the ip address from for the geographical lookup.
If true, only the first found geoip data will be returned, even if the field contains an array.
Default value is true.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Controls what properties are added to the target_field based on the geoip lookup.
The field that will hold the geographical information looked up from the MaxMind database.
If true (and if ingest.geoip.downloader.eager.download is false), the missing database is downloaded when the pipeline is created.
Else, the download is triggered by when the pipeline is used as the default_pipeline or final_pipeline in an index.
Extracts structured fields out of a single text field within a document. You choose which field to extract matched fields from, as well as the grok pattern you expect will match. A grok pattern is like a regular expression that supports aliased expressions that can be reused.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Must be disabled or v1. If v1, the processor uses patterns with Elastic Common Schema (ECS) field names.
Default value is disabled.
The field to use for grok expression parsing.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
A map of pattern-name and pattern tuples defining custom patterns to be used by the current processor. Patterns matching existing names will override the pre-existing definition.
An ordered list of grok expression to match and extract named captures with. Returns on the first expression in the list that matches.
When true, _ingest._grok_match_index will be inserted into your matched document’s metadata with the index into the pattern found in patterns that matched.
Default value is false.
Converts a string field by applying a regular expression and a replacement. If the field is an array of string, all members of the array will be converted. If any non-string values are encountered, the processor will throw an exception.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to apply the replacement to.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The pattern to be replaced.
The string to replace the matching patterns with.
The field to assign the converted value to
By default, the field is updated in-place.
Removes HTML tags from the field. If the field is an array of strings, HTML tags will be removed from all members of the array.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The string-valued field to remove HTML tags from.
If true and field does not exist or is null, the processor quietly exits without modifying the document,
Default value is false.
The field to assign the converted value to
By default, the field is updated in-place.
Uses a pre-trained data frame analytics model or a model deployed for natural language processing tasks to infer against the data that is being ingested in the pipeline.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The ID or alias for the trained model, or the ID of the deployment.
Field added to incoming documents to contain results objects.
Maps the document field names to the known field names of the model. This mapping takes precedence over any default mappings provided in the model configuration.
Contains the inference type and its options.
If true and any of the input fields defined in input_ouput are missing then those missing fields are quietly ignored, otherwise a missing field causes a failure. Only applies when using input_output configurations to explicitly list the input fields.
Joins each element of an array into a single string using a separator character between each element. Throws an error when the field is not an array.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing array values to join.
The separator character.
The field to assign the joined value to. By default, the field is updated in-place.
Converts a JSON string into a structured JSON object.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Flag that forces the parsed JSON to be added at the top level of the document.
target_field must not be set when this option is chosen.
Default value is false.
When set to replace, root fields that conflict with fields from the parsed JSON will be overridden.
When set to merge, conflicting fields will be merged.
Only applicable if add_to_root is set to true.
Supported values include:
replace: Root fields that conflict with fields from the parsed JSON will be overridden.merge: Conflicting fields will be merged.Values are replace or merge.
When set to true, the JSON parser will not fail if the JSON contains duplicate keys.
Instead, the last encountered value for any duplicate key wins.
Default value is false.
The field to be parsed.
The field that the converted structured object will be written into. Any existing content in this field will be overwritten.
This processor helps automatically parse messages (or specific event fields) which are of the foo=bar variety.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
List of keys to exclude from document.
The field to be parsed. Supports template snippets.
Regex pattern to use for splitting key-value pairs.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
List of keys to filter and insert into document. Defaults to including all keys.
Prefix to be added to extracted keys.
Default value is null.
If true. strip brackets (), <>, [] as well as quotes ' and " from extracted values.
Default value is false.
The field to insert the extracted keys into. Defaults to the root of the document. Supports template snippets.
String of characters to trim from extracted keys.
String of characters to trim from extracted values.
Regex pattern to use for splitting the key from the value within a key-value pair.
Converts a string to its lowercase equivalent. If the field is an array of strings, all members of the array will be converted.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to make lowercase.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
Calculates the network direction given a source IP address, destination IP address, and a list of internal networks.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the source IP address.
Field containing the destination IP address.
Output field for the network direction.
List of internal networks. Supports IPv4 and IPv6 addresses and ranges in CIDR notation. Also supports the named ranges listed below. These may be constructed with template snippets. Must specify only one of internal_networks or internal_networks_field.
A field on the given document to read the internal_networks configuration from.
If true and any required fields are missing, the processor quietly exits without modifying the document.
Default value is true.
Executes another pipeline.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The name of the pipeline to execute. Supports template snippets.
Whether to ignore missing pipelines instead of failing.
Default value is false.
The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns.
The processor can be used to obscure Personal Identifying Information (PII) by configuring it to detect known patterns such as email or IP addresses.
Text that matches a Grok pattern is replaced with a configurable string such as <EMAIL> where an email address is matched or simply replace all matches with the text <REDACTED> if preferred.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be redacted
A list of grok expressions to match and redact named captures with
Start a redacted section with this token
Default value is <.
End a redacted section with this token
Default value is >.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
If true and the current license does not support running redact processors, then the processor quietly exits without modifying the document
Default value is false.
If true then ingest metadata _ingest._redact._is_redacted is set to true if the document has been redacted
Default value is false.
Extracts the registered domain (also known as the effective top-level domain or eTLD), sub-domain, and top-level domain from a fully qualified domain name (FQDN). Uses the registered domains defined in the Mozilla Public Suffix List.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the source FQDN.
Object field containing extracted domain components. If an empty string, the processor adds components to the document’s root.
If true and any required fields are missing, the processor quietly exits without modifying the document.
Default value is true.
Removes existing fields. If one field doesn’t exist, an exception will be thrown.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Fields to be removed. Supports template snippets.
Fields to be kept. When set, all fields other than those specified are removed.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
Renames an existing field. If the field doesn’t exist or the new name is already used, an exception will be thrown.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be renamed. Supports template snippets.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The new name of the field. Supports template snippets.
Routes a document to another target index or data stream.
When setting the destination option, the target is explicitly specified and the dataset and namespace options can’t be set.
When the destination option is not set, this processor is in a data stream mode. Note that in this mode, the reroute processor can only be used on data streams that follow the data stream naming scheme.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
A static value for the target. Can’t be set when the dataset or namespace option is set.
Field references or a static value for the dataset part of the data stream name. In addition to the criteria for index names, cannot contain - and must be no longer than 100 characters. Example values are nginx.access and nginx.error.
Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.
default {{data_stream.dataset}}
Field references or a static value for the namespace part of the data stream name. See the criteria for index names for allowed characters. Must be no longer than 100 characters.
Supports field references with a mustache-like syntax (denoted as {{double}} or {{{triple}}} curly braces). When resolving field references, the processor replaces invalid characters with _. Uses the part of the index name as a fallback if all field references resolve to a null, missing, or non-string value.
default {{data_stream.namespace}}
Runs an inline or stored script on incoming documents.
The script runs in the ingest context.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
ID of a stored script.
If no source is specified, this parameter is required.
Object containing parameters for the script.
Adds a field with the specified value. If the field already exists, its value will be replaced with the provided one.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The origin field which will be copied to field, cannot set value simultaneously.
Supported data types are boolean, number, array, object, string, date, etc.
The field to insert, upsert, or update. Supports template snippets.
If true and value is a template snippet that evaluates to null or the empty string, the processor quietly exits without modifying the document.
Default value is false.
The media type for encoding value.
Applies only when value is a template snippet.
Must be one of application/json, text/plain, or application/x-www-form-urlencoded.
If true processor will update fields with pre-existing non-null-valued field.
When set to false, such fields will not be touched.
Default value is true.
The value to be set for the field.
Supports template snippets.
May specify only one of value or copy_from.
Sets user-related details (such as username, roles, email, full_name, metadata, api_key, realm and authentication_type) from the current authenticated user to the current document by pre-processing the ingest.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to store the user information into.
Controls what user related properties are added to the field.
Sorts the elements of an array ascending or descending. Homogeneous arrays of numbers will be sorted numerically, while arrays of strings or heterogeneous arrays of strings + numbers will be sorted lexicographically. Throws an error when the field is not an array.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to be sorted.
The sort order to use.
Accepts "asc" or "desc".
Supported values include:
asc: Ascending (smallest to largest)desc: Descending (largest to smallest)Values are asc or desc.
The field to assign the sorted value to. By default, the field is updated in-place.
Splits a field into an array using a separator character. Only works on string fields.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to split.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
Preserves empty trailing fields, if any.
Default value is false.
A regex which matches the separator, for example, , or \s+.
The field to assign the split value to. By default, the field is updated in-place.
Terminates the current ingest pipeline, causing no further processors to be run.
This will normally be executed conditionally, using the if option.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Trims whitespace from a field. If the field is an array of strings, all members of the array will be trimmed. This only works on leading and trailing whitespace.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The string-valued field to trim whitespace from.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the trimmed value to. By default, the field is updated in-place.
Converts a string to its uppercase equivalent. If the field is an array of strings, all members of the array will be converted.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to make uppercase.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
URL-decodes a string. If the field is an array of strings, all members of the array will be decoded.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field to decode.
If true and field does not exist or is null, the processor quietly exits without modifying the document.
Default value is false.
The field to assign the converted value to. By default, the field is updated in-place.
Parses a Uniform Resource Identifier (URI) string and extracts its components as an object. This URI object includes properties for the URI’s domain, path, fragment, port, query, scheme, user info, username, and password.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
Field containing the URI string.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
If true, the processor copies the unparsed URI to <target_field>.original.
Default value is true.
If true, the processor removes the field after parsing the URI string.
If parsing fails, the processor does not remove the field.
Default value is false.
Output field for the URI object.
The user_agent processor extracts details from the user agent string a browser sends with its web requests.
This processor adds this information by default under the user_agent field.
Description of the processor. Useful for describing the purpose of the processor or its configuration.
Conditionally execute the processor.
Ignore failures for the processor.
Handle failures for the processor.
Identifier for the processor. Useful for debugging and metrics.
The field containing the user agent string.
If true and field does not exist, the processor quietly exits without modifying the document.
Default value is false.
The name of the file in the config/ingest-user-agent directory containing the regular expressions for parsing the user agent string. Both the directory and the file have to be created before starting Elasticsearch. If not specified, ingest-user-agent will use the regexes.yaml from uap-core it ships with.
The field that will be filled with the user agent details.
Controls what properties are added to target_field.
Values are name, os, device, original, or version. Default value is ["name", "major", "minor", "patch", "build", "os", "os_name", "os_major", "os_minor", "device"].
Extracts device type from the user agent string on a best-effort basis.
Default value is false.
Version number used by external systems to track ingest pipelines. This parameter is intended for external systems only. Elasticsearch does not use or validate pipeline version numbers.
Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.
Default value is false.
Controls how processors in this pipeline should read and write data on a document's source.
Values are classic or flexible.
PUT _ingest/pipeline/my-pipeline-id
{
"description" : "My optional pipeline description",
"processors" : [
{
"set" : {
"description" : "My optional processor description",
"field": "my-keyword-field",
"value": "foo"
}
}
]
}resp = client.ingest.put_pipeline(
id="my-pipeline-id",
description="My optional pipeline description",
processors=[
{
"set": {
"description": "My optional processor description",
"field": "my-keyword-field",
"value": "foo"
}
}
],
)const response = await client.ingest.putPipeline({
id: "my-pipeline-id",
description: "My optional pipeline description",
processors: [
{
set: {
description: "My optional processor description",
field: "my-keyword-field",
value: "foo",
},
},
],
});response = client.ingest.put_pipeline(
id: "my-pipeline-id",
body: {
"description": "My optional pipeline description",
"processors": [
{
"set": {
"description": "My optional processor description",
"field": "my-keyword-field",
"value": "foo"
}
}
]
}
)$resp = $client->ingest()->putPipeline([
"id" => "my-pipeline-id",
"body" => [
"description" => "My optional pipeline description",
"processors" => array(
[
"set" => [
"description" => "My optional processor description",
"field" => "my-keyword-field",
"value" => "foo",
],
],
),
],
]);curl -X PUT -H "Authorization: ApiKey $ELASTIC_API_KEY" -H "Content-Type: application/json" -d '{"description":"My optional pipeline description","processors":[{"set":{"description":"My optional processor description","field":"my-keyword-field","value":"foo"}}]}' "$ELASTICSEARCH_URL/_ingest/pipeline/my-pipeline-id"{
"description" : "My optional pipeline description",
"processors" : [
{
"set" : {
"description" : "My optional processor description",
"field": "my-keyword-field",
"value": "foo"
}
}
]
}{
"description" : "My optional pipeline description",
"processors" : [
{
"set" : {
"description" : "My optional processor description",
"field": "my-keyword-field",
"value": "foo"
}
}
],
"_meta": {
"reason": "set my-keyword-field to foo",
"serialization": {
"class": "MyPipeline",
"id": 10
}
}
}