Brewing in Beats: Network conditions in processors
Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
This update is over the last two weeks.
What's new in Beats
Filebeat registry fixes
Some Filebeat bugs have been uncovered when stabilizing and digging further into some of our flaky tests. Due to a race condition a file entry was reintroduced in the registry file after it has been removed. We also discovered that
clean_removed did not always work correctly on Windows.
PR #10747 fixes these issues by solving the race condition and improving file deletion detection on Windows.
Add sequence number to syslog parser
With the change in #10760, the filebeat syslog input can parse syslog sequence numbers as well. If present the sequence number will be parsed and added to log event.
Beats Processor Conditionals
Andrew Kroh improved support for conditionals in Beats by introducing a new “network condition” (#10743) and a new if-then-else-processor (#10744). The network condition can match IPv4 and IPv6 ranges, but also supports named ranges like loopback, unicast, multicast, private, and more.
What's new in Central Configuration
We are moving to a new return format on the API that will normalize the return values across endpoints #27408.
Once this is completed we will be ready for documenting the API. This is a very exciting enhancement as we know there is a lot of demand for programmatic control of CM both from within the company as well as from beta testing customers.
In addition, we are working to add K/V metadata to enrolled Beats. This will be very useful for solutions built on top of Beats and CM to integrate more fully, and more seamlessly https://github.com/elastic/
What's new in Elastic Common Schema (ECS)
The core of ECS is a few YAML files, out of which lots of other things are generated: documentation, an index template, a csv, a Go library, a JSON file to power Kibana tooltips, etc. The first version of the generator was difficult to maintain, and didn’t support reusable field sets. A new, simpler generator is coming along nicely. It will support everything we need, and make it easier to add new outputs, whether the output is in Python or another language. You can check it out in PR #336.
Intro to ECS
Migrating to ECS
We are planning to migrate Beats to ECS, around the Elastic Stack 7.0 official release. There is a new setting in Beats called “migration.enabled” will let people create their 7.x Beat indices with field aliases to maintain backwards compatibility for their existing visualizations in their new 7.x indices. This setting is off by default.