01 March 2019

Brewing in Beats: Network conditions in processors

By Monica Sarbu

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

This update is over the last two weeks.

What's new in Beats

Filebeat registry fixes

Some Filebeat bugs have been uncovered when stabilizing and digging further into some of our flaky tests. Due to a race condition a file entry was reintroduced in the registry file after it has been removed. We also discovered that close_removed and clean_removed did not always work correctly on Windows.

PR #10747 fixes these issues by solving the race condition and improving file deletion detection on Windows.

Add sequence number to syslog parser

With the change in #10760, the filebeat syslog input can parse syslog sequence numbers as well. If present the sequence number will be parsed and added to log event.

Beats Processor Conditionals

Andrew Kroh improved support for conditionals in Beats by introducing a new “network condition” (#10743) and a new if-then-else-processor (#10744). The network condition can match IPv4 and IPv6 ranges, but also supports named ranges like loopback, unicast, multicast, private, and more.

What's new in Central Configuration

We are moving to a new return format on the API that will normalize the return values across endpoints #27408.

Once this is completed we will be ready for documenting the API. This is a very exciting enhancement as we know there is a lot of demand for programmatic control of CM both from within the company as well as from beta testing customers.

In addition, we are working to add K/V metadata to enrolled Beats. This will be very useful for solutions built on top of Beats and CM to integrate more fully, and more seamlessly https://github.com/elastic/beats/issues/9881

What's new in Elastic Common Schema (ECS)

New generator

The core of ECS is a few YAML files, out of which lots of other things are generated: documentation, an index template, a csv, a Go library, a JSON file to power Kibana tooltips, etc. The first version of the generator was difficult to maintain, and didn’t support reusable field sets. A new, simpler generator is coming along nicely. It will support everything we need, and make it easier to add new outputs, whether the output is in Python or another language. You can check it out in PR #336.

Intro to ECS

We recently released the first GA version of ECS. Check out the blog post and the recording of the webinar for more details about Elastic Common Schema (ECS). 

Migrating to ECS

We are planning to migrate Beats to ECS, around the Elastic Stack 7.0 official release. There is a new setting in Beats called “migration.enabled” will let people create their 7.x Beat indices with field aliases to maintain backwards compatibility for their existing visualizations in their new 7.x indices. This setting is off by default.