Validating Elastic Common Schema (ECS) fields using Elastic Security detection rules

The Elastic Common Schema (ECS) provides an open, consistent model for structuring your data in the Elastic Stack. By normalizing data to a single common model, you can uniformly examine your data using interactive search, visualizations, and automated analysis.

Elastic provides hundreds of integrations that are ECS-compliant out of the box, but ECS also allows you to normalize custom data sources. Normalizing a custom source can be an iterative and sometimes time-intensive process. However, we can use the Elastic Security detection engine to help quickly identify ECS non-compliance in our events.

Getting up and running

First, let’s define ECS compliance. “ECS-compliant” describes indexed data that has adopted and aligns with the ECS requirements and guidelines. ECS-compliant events ensure the best possible experience from your data across the Elastic Stack and solutions.

If you’d like to experiment with any of the detection rules demonstrated here, the easiest way to get started is by signing up for a free Elastic Cloud trial. All of the sample detection rules covered here can be downloaded from this gist and imported into the detection engine.

Now let’s explore custom detection rules which will alert on ECS non-compliant events.

Missing fields

An ECS event should populate the ecs.version field. By having each event include the targeted ECS version, we know and understand what fields and data types to expect or not expect.

In this first example, detection alerts will be generated if the ecs.version field is missing from any event in the targeted indices:

Shortly after activating the rule, alerts are generated for non-compliant events. With the details from the alerts, we can know exactly which data sources and events are non-compliant:

Beyond a single missing field, detections can also identify the absence of expected fields when others are present. For example, this next detection looks for an event that populates event.category: process but fails to populate some key process.* fields:

Taking a look at the generated alerts, we discover our source isn’t populating the process.pid field:

Allowed values

Data sources can populate the ECS categorization fields to describe and better classify each event. Populating each event with the proper fields and values ensures consistency to group related event types.

This next rule checks each event containing the event.category field is using a valid allowed value:

Events with an invalid value for the event.category field are now generating alerts. In our example, event.category is set to use “auth” instead of the correct value, “authentication”. Now armed with this insight, we can turn our attention to correcting the problem instead of hunting for non-compliant events.

Source and destination pairings

Not all events are network events, but when they are, they should always populate the source and destination as a pair. Given that our custom event source contains network events, here's a rule that ensures source and destination are both populated:

After activating the new detection, the new detection alerts indicate the source.address field is not populated:

Breakdown fields

Some ECS fields can be populated by an ingest pipeline processor like user agent. When the original user_agent field runs through the user_agent ingest processor, the processor will parse and then populate data into the appropriate fields.

This last example looks for the expected field breakdown and will alert if an expected field is missing in an event:

Alerts are created after the rule is activated. Digging into the alert, we confirm that the event is missing the user_agent.name field:

Summary

These examples are just starting points! With custom, ECS compliance-focused detections, the detection engine becomes another tool to continually improve and refine your data ingestion process. Just make sure these ECS detection rules don’t trigger a pager alert!

Ready to get started? Start your free 14-day trial (no credit card required). Or download our products, free, for your on-prem deployment.