Elastic Common Schema: The journey so far
It has been just over two years since we introduced the Elastic Common Schema (ECS), and what a journey it’s been. From categorization fields to request for comments to Threat Intelligence fields, ECS has evolved rapidly over the course of the last two years. In this blog post, I would like to reflect on the ECS journey so far, and look towards the future of ECS.
You may by now be well experienced with ECS as a specification that provides a consistent and customizable way to structure your data in Elasticsearch, facilitating the analysis of data from diverse sources. With ECS, analytics content such as dashboards and machine learning jobs can be applied more broadly, searches can be crafted more narrowly, and field names are easier to remember.
Back in 2019, ECS 1.0 started out with 279 fields. As of ECS 1.9, recently launched in March 2021, we’re now at a staggering 762 fields across 36 field sets. The pace at which ECS has evolved is a testament to everyone involved in ECS, and we’re really excited for what's to come.
ECS has been adopted across the Elastic Stack and solutions. Since we launched Elastic Security in 2019, there’s been an explosion of content mapped to ECS, including integrations and detection rules. Elastic Observability has also leveraged ECS extensively by using the schema to map logs, metrics, and traces.
It takes a village to raise a schema
The Elastic community extends far beyond our employee base. The community has always played a pivotal role to everything we do at Elastic, and ECS is no different. Unlike other schemas out there, ECS has evolved and will continue to evolve thanks to feedback from our community. Since ECS 1.0, we’ve added countless fields and fieldsets to cover new use cases across IT Ops, security analytics, and APM. Many of those additions come as a direct result of feedback from our community.
Categorization fields
Shortly after releasing ECS 1.0, we realized the need for a set of fields to enable analysts to select and filter events for inclusion in common visualizations and analysis by grouping them into a common set of categories. We added four categorization fields in ECS 1.4:
Since then, we’ve evolved the allowed values of the ‘event.category’ to accommodate new categories, including configuration, network, registry, and session events. ‘event.kind’ has also grown from 9 to 16 allowed values, including admin, allowed, connection, and start. One of the biggest successes we’ve seen with categorization fields lies within our 500+ security detection rules. These rules are based on ECS fields and leverage categorization fields to ensure our rules work across multiple data sources.
As ECS progresses, you’re going to see further additions to our categorization fields and as part of that effort, we want to ensure it’s easy to leverage those fields. In ECS 1.9, we added some real-world examples of categorization events to use as references including firewall, file, IDS, and user events.
ECS additions and changes — have your say!
In 2020, we realized there was room for improvement when proposing additions to the schema and facilitating discussion around those additions. So we introduced a request for comments (RFC) via ECS RFC process, which has been a huge success so far. Prior to the RFC process, it was challenging to gather feedback from subject matter experts within both Elastic and the wider community. The Threat Intelligence fields are actively advancing through the RFC process, and the discussions within that RFC were incredible to see — with no less than 164 comments.
As a result, we now have a set of Threat Intelligence fields that have been discussed and agreed upon from a wide range of expertise. Several other RFCs are in progress, which opens up some exciting new use cases for ECS. Proposals include fieldsets such as email, container orchestration engines, sessions, and more. We’re always seeking feedback on these proposals, so please check them out and chime in.
ECS adoption
We’ve seen tremendous adoption of ECS across vendors, partners, and users. Our first major milestone in ECS adoption was reached in the 7.0 release, when we converted our Beats modules to produce ECS-compliant data. Since then, several vendors and partners have continued to adopt ECS. One great example is Corelight, which aligned with the ECS vision early on by adding ECS support to their sensors.
ECS was designed with customization in mind. As a result, countless organizations have taken ECS as a starting point and extended it to accommodate use cases specific to their organizations. Many of these organizations have tried to adopt other schemas in the past, or even create their own. It has been humbling to see the confidence these organizations have in ECS by choosing it as their de facto schema.
One such organization is Uber. Wes O’Connell, Security Engineering Lead, recently commented: “implementing ECS has been a force multiplier for our team and has dramatically streamlined security investigations.” You can learn more about how Uber is reinventing enterprise defense with Elastic in Wes’ ElasticON 2020 talk.
Simplifying mapping to ECS
While adoption has been fantastic, we consistently hear that mapping events to ECS isn’t easy and therefore adopting ECS can be a challenge without prior Elasticsearch experience. This is something we are acutely aware of and have plans to improve the workflow of mapping to ECS. While tools such as ECS-Mapper, which we used to map Crowdstrike events, and ECS-Loggers are a great start, we have more to do. Once events are mapped to ECS, validating ECS compliance is another area we need to improve upon. Using detection rules within Elastic Security is one approach we recently wrote about, but we have plans for more.
Summary
As we celebrate ECS turning two years old, we hope you now have a clear understanding on the journey so far and where ECS is headed. We’ve seen countless organizations adopt ECS as their de facto schema, which is truly humbling to see. If the next two years are as exciting as the last two, ECS is set to have a blast!
We’re always happy to hear your feedback and ideas. You can reach us at the ECS Slack channel, ECS repo and Discuss forums.
If you’re new to Elastic Security, you can experience our latest version on Elasticsearch Service on Elastic Cloud. Also be sure to take advantage of our Quick Start training to set yourself up for success.