Elastic Common Schema and OpenTelemetry — A path to better observability and security with no vendor lock-in

By

Elastic Observability and Security Teams

ecs-otel-announcement-1.jpeg

At KubeCon Europe, it was announced that Elastic Common Schema (ECS) has been accepted by OpenTelemetry (OTel) as a contribution to the project. The goal is to achieve convergence of ECS and OpenTelemetry’s Semantic Conventions (SemConv) into a single open schema that is maintained by OpenTelemetry. This FAQ details Elastic’s contribution of Elastic Common Schema to OpenTelemetry, how it will help drive the industry to a common schema, and its impact on observability and security.

What is being announced? 

Elastic is contributing its open source project, Elastic Common Schema (ECS), to the OpenTelemetry project under the Cloud Native Computing Foundation (CNCF) to help converge toward a common schema for observability and security data. A common schema helps normalize data to enable better analysis, visualization, and correlation of that data on any security or observability platform. OpenTelemetry’s adoption of ECS benefits the OTel user community with a mature and proven common schema for metrics, logs, traces, resources (hosts, containers, etc.) and security events. 

What do Elastic users need to know?

Elastic will preserve our users’ investments in ECS. The continued evolution of ECS from within OpenTelemetry will give ECS users a clear path to adopt what we expect to be the industry’s most widely used standard for semantic conventions.

Elastic will participate and closely collaborate with the OTel community to properly merge ECS and OpenTelemetry’s Semantic Conventions over time. Elastic will continue to support user data in the current ECS format, although schema evolution will happen on the newly merged schema. Elastic users will have the choice to continue ingesting and using the current (“frozen”) ECS format or migrate to the new schema.

Elastic will provide guidance and tooling for a simple migration for users who decide to migrate to the new schema.

Why is Elastic contributing ECS to OTel?

OpenTelemetry’s Semantic Conventions (SemConv) and Elastic Common Schema convergence provides a common naming scheme for resources, metrics, logs, traces, security events, and audit events, which can be used across codebase, libraries, and platforms. This convergence of ECS with OTel Semantic Conventions will:

  • Align efforts around a single standard poised for broad adoption by producers of event data
  • Drive better visibility and root cause analysis for operations
  • Enable vendors and the community to focus on richer observability and security features versus dealing with data transformation tasks
  • Facilitate cross-organizational analysis across many signal types, including between security and observability
  • Increase OpenTelemetry’s adoption and the continued evolution and convergence of observability and security domains

Why is a common schema needed for organizations?

Too often, organizations struggle with understanding when an issue occurred (visibility) and why it happened (root cause analysis). This operational challenge is due to data that is siloed and structured in different schemas. Organizations spend more time on unnecessary data transformations than understanding the issue, analyzing root cause, or optimizing operations.

With data structured according to a common schema, operations teams will be able to focus on recognizing, resolving, and preventing issues, along with lowering mean time to resolution (MTTR). Operations can also reduce costs by not having duplicated data and not having to process data for normalization. 

What is an illustrative example of a common schema?

A simple illustrative example is when a client’s IP address is sent from multiple sources that are either monitoring or managing telemetry about the client. The observability platform receives this information in multiple formats:

src:10.42.42.42
client_ip:10.42.42.42 
apache2.access.remote_ip: 10.42.42.42 
context.user.ip:10.42.42.42 
src_ip:10.42.42.42

Having an IP address represented in multiple ways introduces complexities in analyzing potential problems or even identifying them. Without clear semantics for the observed data and a common schema, observability solutions struggle with automatically correlating, analyzing, and identifying root causes from the data. Consequently, operations (SREs, DevOps, etc.) must understand these multiple definitions, know how to find them, and then manually normalize the data for analysis.

With a common schema, all the incoming data is in a standardized format. Taking the above example, each of the sources would identify the client’s IP address the same way:

source.ip:10.42.42.42

Observability and security solutions can now leverage a consistently defined data schema for automation of data correlation and analysis. Operations now spends more time understanding the issue, finding root cause, and optimizing operations versus spending time on unnecessary data transformations.

What is Elastic Common Schema (ECS)?

Elastic Common Schema (ECS), an open source (Apache 2.0) specification, is developed with support from the Elastic user community to define a common set of fields to be used when storing event data in Elasticsearch. The goal of ECS is to enable and encourage users of Elasticsearch to normalize their event data, so that they can better analyze, visualize, and correlate the data represented in their events. ECS is the foundation of Elastic Observability and Security solutions and is a proven and widely adopted schema that has evolved and grown over the years since its inception in 2019.

What are OpenTelemetry and OpenTelemetry Semantic Conventions?

OpenTelemetry (OTel) is an open source project providing a collection of specifications, tools, APIs, and SDKs that can be used to generate, collect, process, and export telemetry data (metrics, logs, and traces) to understand software performance and behavior. It has become the second highest velocity project in the CNCF ecosystem.

OpenTelemetry’s Semantic Conventions (SemConv) specify common names for different kinds of operations and data. The benefit of using OpenTelemetry’s SemConv is in following a common naming scheme that can be standardized across a codebase, libraries, and platforms for OTel end users. Additionally, another big benefit is the decoupling of vendor-specific semantics. Thus with OpenTelemetry’s SemConv, data users resolve the vendor lock-in of their data. So, they can easily move between observability solutions (and security solutions with the ECS contribution) without the need to adapt their data collection.

How does the ECS contribution help OpenTelemetry?

OpenTelemetry’s greatest need is to accelerate the definition of schema for describing log and security events. Contributors to ECS have already defined a unified and well-accepted set of logging semantic conventions, which can be adopted in OTel. ECS is widely used to structure logs consumed in observability and security use cases.

The combination will accelerate the integration of vendor-created logging and OTel component logs (for example, OTel Collector Logs Receivers and Processors). The goal is to define vendor-agnostic semantic conventions for the most popular types of systems and support vendor-created or open source components (for example, HTTP access logs, network logs, system access/authentication logs) extending OTel correlation to these new signals. Users will also benefit from using turnkey log integrations that will be fully recognized by OTel-compatible observability and security products and services. 

The maturity of ECS for Security is a great opportunity to improve the utility of data collected with OpenTelemetry for security use cases. Incorporating ECS will enable OTel producers to structure security events.

Will there be any changes to the licensing for ECS?

There will not be any changes to ECS licensing. ECS is Apache 2.0 licensed, as is OpenTelemetry.

Does Elastic support OpenTelemetry today?

Elastic supports OTel natively. Elastic users can send OTel data directly from applications or through the OTel collector into Elastic APM, which processes both OTel SemConv and ECS. With this native OTel support, all Elastic APM capabilities are available with OTel. See Elastic documentation to learn more about OTel integration.

elastic otel microservices

Where can I learn more about ECS and Elastic’s support for OpenTelemetry?

Find additional information and documentation around Elastic’s support and integrations for OpenTelemetry below:

Sign up for Elastic Cloud free trial

Spin up a fully loaded deployment on the cloud provider you choose. As the company behind Elasticsearch, we bring our features and support to your Elastic clusters in the cloud.

Start free trial