« Versioned avro codec plugin docs Avro codec plugin v3.4.1 »

› › ›

Avro codec plugin v3.5.0

edit

IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Avro codec plugin v3.5.0

edit

Plugin version: v3.5.0
Released on: 2025-11-26
Changelog

For other versions, see the overview list.

To learn more about Logstash, see the Logstash Reference.

Getting help

edit

For questions about the plugin, open a topic in the Discuss forums. For bugs or feature requests, open an issue in Github. For the list of Elastic supported plugins, please consult the Elastic Support Matrix.

Description

edit

Read serialized Avro records as Logstash events

This plugin is used to serialize Logstash events as Avro datums, as well as deserializing Avro datums into Logstash events.

Event Metadata and the Elastic Common Schema (ECS)

edit

The plugin behaves the same regardless of ECS compatibility, except adding the original message to [event][original].

Encoding

edit

This codec is for serializing individual Logstash events as Avro datums that are Avro binary blobs. It does not encode Logstash events into an Avro file.

Decoding

edit

This codec is for deserializing individual Avro records. It is not for reading Avro files. Avro files have a unique format that must be handled upon input.

Partial deserialization

Avro format is known to support partial deserialization of arbitrary fields, providing a schema containing a subset of the schema which was used to serialize the data. This codec doesn’t support partial deserialization of arbitrary fields. Partial deserialization might work only when providing a schema which contains the first N fields of the schema used to serialize the data (and in the same order).

Usage

edit

Example usage with Kafka input.

input {
  kafka {
    codec => avro {
        schema_uri => "/tmp/schema.avsc"
    }
  }
}
filter {
  ...
}
output {
  ...
}

Avro Codec Configuration Options

edit

Setting	Input type	Required
`ecs_compatibility`	string	No
`encoding`	string, one of `["binary", "base64"]`	No
`password`	password	No
`proxy`	uri	No
`schema_uri`	string	Yes
`ssl_certificate`	path	No
`ssl_certificate_authorities`	list of path	No
`ssl_cipher_suites`	array	No
`ssl_enabled`	boolean	No
`ssl_key`	path	No
`ssl_keystore_password`	password	No
`ssl_keystore_path`	path	No
`ssl_keystore_type`	string	No
`ssl_supported_protocols`	array	No
`ssl_truststore_password`	password	No
`ssl_truststore_path`	path	No
`ssl_truststore_type`	string	No
`ssl_verification_mode`	string	No
`tag_on_failure`	boolean	No
`target`	string	No
`username`	string	No

`ecs_compatibility`

edit

Value type is string
Supported values are:
- disabled: Avro data added at root level
- v1,v8: Elastic Common Schema compliant behavior ([event][original] is also added)

Controls this plugin’s compatibility with the Elastic Common Schema (ECS).

`encoding`

edit

Value can be any of: binary, base64
Default value is base64

Set encoding for Avro’s payload. Use base64 (default) to indicate that this codec sends or expects to receive base64-encoded bytes.

Set this option to binary to indicate that this codec sends or expects to receive binary Avro data.

`password`

edit

Value type is password
There is no default value for this setting.

Password for HTTP basic authentication when fetching remote schemas. Used together with username.

`proxy`

edit

Value type is uri
There is no default value for this setting.

The address of a forward HTTP proxy to use when contacting a remote schema registry.

`schema_uri`

edit

This is a required setting.
Value type is string
There is no default value for this setting.

schema path to fetch the schema from. This can be a http or file scheme URI example:

http - http://example.com/schema.avsc
file - /path/to/schema.avsc

`tag_on_failure`

edit

Value type is boolean
Default value is false

tag events with _avroparsefailure when decode fails

`ssl_certificate`

edit

Value type is path
There is no default value for this setting.

Path to PEM encoded certificate file for client authentication (mutual TLS). You may use this setting or ssl_keystore_path, but not both simultaneously.

Example

ssl_certificate => "/path/to/client.crt"

`ssl_certificate_authorities`

edit

Value type is a list of path
There is no default value for this setting.

Path to PEM encoded CA certificate file(s) for server verification. This is an alternative to using ssl_truststore_path. You may use this setting or ssl_truststore_path, but not both simultaneously.

Example

ssl_certificate_authorities => ["/path/to/ca.crt"]

`ssl_cipher_suites`

edit

Value type is array
There is no default value for this setting.

The list of cipher suites to use, listed by priorities. Supported cipher suites vary depending on which version of Java is used.

`ssl_key`

edit

Value type is path
There is no default value for this setting.

Path to PEM encoded private key file for client authentication. Must be used together with ssl_certificate. The private key must be unencrypted (passphrase-protected keys are not supported).

Example

ssl_key => "/path/to/client.key"

`ssl_enabled`

edit

Value type is boolean
There is no default value for this setting.

Enable SSL/TLS secured communication to remote schema registry. When using HTTPS schema URIs, SSL is automatically enabled.

`ssl_keystore_path`

edit

Value type is path
There is no default value for this setting.

The path to the JKS or PKCS12 keystore file for client certificate authentication. Use this when the schema registry requires mutual TLS (mTLS) authentication.

`ssl_keystore_password`

edit

Value type is password
There is no default value for this setting.

The password for the keystore file specified in ssl_keystore_path.

`ssl_keystore_type`

edit

Value type is string
There is no default value for this setting.

The format of the keystore file. It must be either jks or pkcs12.

`ssl_supported_protocols`

edit

Value type is array
Default value is [] (uses Java defaults)
Valid values are: TLSv1.1, TLSv1.2, TLSv1.3

List of allowed SSL/TLS protocol versions. When not specified, the JVM defaults are used.

`ssl_truststore_path`

edit

Value type is path
There is no default value for this setting.

The path to the JKS or PKCS12 truststore file containing certificates to verify the schema registry server’s certificate.

Example

input {
  kafka {
    codec => avro {
        schema_uri => "https://schema-registry.example.com:8081/schemas/ids/1"
        ssl_truststore_path => "/path/to/truststore.jks"
        ssl_truststore_password => "${TRUSTSTORE_PASSWORD}"
    }
  }
}

`ssl_truststore_password`

edit

Value type is password
There is no default value for this setting.

The password for the truststore file specified in ssl_truststore_path.

`ssl_truststore_type`

edit

Value type is string
There is no default value for this setting.

The format of the truststore file. It must be either jks or pkcs12.

`ssl_verification_mode`

edit

Value type is string
Default value is "full"
Valid options are: full, none

Options to verify the server’s certificate:

full: Validates that the provided certificate has an issue date that’s within the not_before and not_after dates; chains to a trusted Certificate Authority (CA); has a hostname or IP address that matches the names within the certificate. (recommended)
none: Performs no certificate validation. Warning: Disabling this severely compromises security (https://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf)

Example

input {
  kafka {
    codec => avro {
        schema_uri => "https://schema-registry.example.com:8081/schemas/ids/1"
        ssl_certificate_authorities => ["/path/to/ca.crt"]
        ssl_verification_mode => "full"
    }
  }
}

`target`

edit

Value type is string
There is no default value for this setting.
This is only relevant when decode data into an event

Define the target field for placing the values. If this setting is not set, the Avro data will be stored at the root (top level) of the event.

Example

input {
  kafka {
    codec => avro {
        schema_uri => "/tmp/schema.avsc"
        target => "[document]"
    }
  }
}

`username`

edit

Value type is string
There is no default value for this setting.

Username for HTTP basic authentication when fetching remote schemas. Used together with password.

Example

input {
  kafka {
    codec => avro {
        schema_uri => "https://schema-registry.example.com:8081/schemas/ids/1"
        username => "registry_user"
        password => "${REGISTRY_PASSWORD}"
    }
  }
}

« Versioned avro codec plugin docs Avro codec plugin v3.4.1 »