Converting a Custom Implementationedit

A common schema helps you correlate and use data from various sources.

Fields for most Elastic modules and solutions (version 7.0 and later) are mapped to the Elastic Common Schema. You may want to map data from other implementations to ECS to help you correlate data across all of your products and solutions.

Before you start a conversion, be sure that you understand the basics below.

Core and extended levelsedit

Make sure you understand the distinction between Core and Extended fields, as explained in the Guidelines and Best Practices.

Core and Extended fields are documented in the ECS Field Reference or, for a single page representation of all fields, please see the generated CSV of fields.

An approach to mapping an existing implementationedit

Here’s the recommended approach for converting an existing implementation to ECS.

  1. Review each field in your original event and map it to the relevant ECS field.

    • Start by mapping your field to the relevant ECS Core field.
    • If a relevant ECS Core field does not exist, map your field to the relevant ECS Extended field.
    • If no relevant ECS Extended field exists, consider keeping your field with its original details, or possibly renaming it using ECS naming guidelines and attempt to map one or more of your original event fields to it.
  2. Review each ECS Core field, and attempt to populate it.

    • Review your original event data again
    • Consider populating the field based on additional meta-data such as static information (e.g. add event.category:authentication even if your auth events don’t mention the word "authentication")
    • Consider capturing additional environment meta-data, such as information about the host, container or cloud instance.
  3. Review other extended fields from any field set you are already using, and attempt to populate it as well.
  4. Set ecs.version to the version of the schema you are conforming to. This will allow you to upgrade your sources, pipelines and content (like dashboards) smoothly in the future.

Using a spreadsheet to plan your migrationedit

Using a spreadsheet to plan the migration from pre-existing source fields to ECS is a common practice. It’s a good way to address each of your fields methodically among colleagues.

If the data source is either a structured log, or if you already have a pipeline producing events with these non-ECS field names, the tool ECS Mapper may help you get started in performing all of these field renames.

After exporting your mapping spreadsheet to CSV, ECS Mapper will convert your field mapping to equivalent pipelines for Beats, Elasticsearch, and Logstash. Learn more at ECS Mapper.