06 June 2018

Brewing in Beats: Introduce Elastic Common Schema

By Monica Sarbu

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Did you know that Beats 6.2 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, check out the Getting Started with Beats webinar.

This update covers the last two weeks.

Elastic Common Schema (ECS)

The first pre-release of ECS has been published this week. ECS is used to provide a common data model when ingesting data into Elasticsearch. Having a common schema allows you correlate data from sources like logs and metrics or IT operations analytics and security analytics. ECS is still under development, so any feedback is welcome. Please check https://github.com/elastic/ecs for more details.

Dissect processor

The Dissect processor is an alternative to grok in Ingest Node and Logstash. The processor can be used to extract structured fields from any text fields, assuming the structure does not vary from line to line. Dissect is not based on regular expressions, but linearly scans and tokenizes the input strings. This makes the processor really fast. More complicated events/logs with varying formats will still require grok.

Adding tokenization to beats will improve event filtering, allowing users to apply conditionals, string formatters in the output, and drop event or event fields based on the original contents.

The processor is similar to the Dissect filter in Logstash, but is executed in the beat itself. It will be available in 6.4.

Improved compatibility with new Docker versions

Docker client code has been updated to negotiate the API version to use when talking to Docker daemon (#7165). This change improves compatibility with latest Docker versions, both when using the Metricbeat module and the add_docker_metadata processor.

Filebeat Elasticsearch fileset

Filebeat now supports a basic Elasticsearch Fileset to parse Elasticsearch logs with the following behaviour:

  • Handles multiline events
  • Extract timestamp, log level, and the component.

This PR was a collaborative effort from Nicolas and our excellent support members Joshua and Jason. This is part of an ongoing effort, see the meta issue for more details.

Fix a data loss issue with the add_docker_metadata processor

Previously when the processor was unable to retrieve the container id we were dropping the current event, instead we now report the error, but the processor is sending the event to Elasticsearch without the fields.  (#7133)

Enhance Packetbeat process monitor

Packetbeat can now report the process associated with captured TCP protocol traffic on Linux and Windows hosts. It will give the full command-line arguments associated with the process. The Packetbeat process monitor no longer requires a regular expression to match traffic to a process because it finds the open sockets held by each running process. You can see the details in #7135.

Jolokia Discovery support

Jolokia Discovery is a mechanism used by Jolokia agents to announce their endpoints and some additional information to clients. This can be useful in dynamic deployments of Java applications. We have been working on a proof of concept of an autodiscover provider to take advantage of this feature in Beats (#7141). It will be available in 6.4 as an experimental feature, and even if it is a natural fit for the JMX metricset of the Jolokia Metricbeat module, we are open to any feedback also in other use cases.

Kubernetes Pod owner metadata

We have enhanced the add_kubernetes_metadata processor to include Pod owner information. This is really useful when doing analysis, as users will be able to query Pod metrics & logs by Deployment, ReplicaSet or StatefulSet, with more coming soon.

Dashboard export

When developing Filebeat or Metricbeat modules, one is well advised to create dashboards as well. The dashboards and all dependencies need to be exported from Kibana, so they can be bundled with the final module. Improving the tooling for module developers, PR #7239 adds a new command for exporting dashboards and dependencies using the Filebeat or Metricbeat binary only.

All Beats changes

Repository: elastic/beats

Metricbeat

Changes in master:

  • Negotiate Docker API version #7165
  • Elasticsearch pending tasks metricset #7144
  • Metricbeat: Set guest as default user in RabbitMQ module #7107
  • Add xpack data structure for Elasticsearch index_summary metricset #7102
  • [Kibana Module] Add additional metric fields #6746
  • Fix system process CPU ticks field mapping #7230
  • Update state_container to latest kube-state-metrics version #7216
  • Refactor http.server metricset #7100
  • Fix system process CPU ticks field mapping #7230
  • [metricbeat] Fix errors in process summary on latest Linux kernels #6306
Packetbeat

Changes in master:

  • Add missing type to http.response.body field #7169
  • Packetbeat process monitor enhancements #7135
  • MINOR: Resolve string concatenation todo in packetbeat/.../http.go #7204
Filebeat

Changes in master:

  • Fix permissions of generated Filebeat filesets #7140
  • Keep event when add_docker_metadata fails to extract container id #7133
  • Add basic Elasticsearch server fileset #7124
  • Do not spam Filebeat logs with errors if harvester_limit is reached #7223
  • Move crawler to the new input package instead of prospector #7215
  • Add dissect to traefik/access metricset for first stage of processing #7209
  • Comply with PostgreSQL database name format #7198
Auditbeat

Changes in master:

  • Auditd module: New metric for kernel lost events #7179
Processors

Changes in master:

  • Unify handling of root and extracted key for dissect #7210
Dashboards

Changes in master:

  • Fix default_field for metricbeat #7015
  • Add support to export dashboard to Beat #7239
  • Extract Kibana dashboards #7224
Infrastructure

Changes in master:

  • Beats preference pane for macOS #6923
Testing

Changes in master:

  • Add more Vagrant machines #7153
  • Rename traefik test directory to run tests #7208
  • Set foundation for testing against 6.0 builds #7203
Packaging

Changes in master:

  • Make beat.reference.yml optional. #7092
Documentation

Changes in master:

  • Clarify max_retries description for Filebeat and Winlogbeat #7180
  • Mention TCP in DNS protocol docs #7174
  • Document username/password options for HTTP based modules #7134
  • Update Filebeat modules development guide #7242
  • Add links to docs about ssl settings #7236
  • Add link to topic that describes more dashboard loading options #7233
  • Fix beat name in Filebeat hints autodiscover example #7226
  • Fix `client_authentication` option in the documentation. #7168
  • Update docs #7164
  • Add file spool to queue docs #6902

Changes in 6.3:

  • Remove mention of Windows XP from docs #7014
  • Clarify max_retries description for Filebeat and Winlogbeat #7180
  • Add link to topic that describes more dashboard loading options #7233
  • Add links to docs about ssl settings #7236
  • Fix conditional coding and keystore CLI example #7246)
  • Close changelog for 6.3.0 #7190

Changes in 6.2:

  • Add link to topic that describes more dashboard loading options #7233
  • Add links to docs about ssl settings #7236)
  • Fix conditional coding and keystore CLI example #7246

Repository: elastic/go-libaudit

Changes in master:

  • Prepare v0.3.0 release #39
  • Add asynchronous status request via `GetStatusAsync` #38
  • Support backlog wait time feature #36
  • Make Close safe to call more than once #35