02 août 2016

Brewing in Beats: Configure the Ingest Node Pipeline

Par Monica Sarbu

New community Beat: Cassandrabeat

Cassandrabeat uses Cassandra’s nodetool cfstats utility to monitor Cassandra database nodes and lag. Please give it a try and let us know what do you think.

Select pipeline for Ingest Node from Beats

By defining the pipeline in Beats, you can dynamically choose the Ingest pipeline per event.

There are three options to define the pipeline. One is to define a single pipeline (under output.elasticsearch.pipeline) for all your events, in which you can access other fields from the event:

output.elasticsearch.pipeline: ‘%{[fields.example]}’

Another one is to define an array of pipeline rules (under output.elasticsearch.pipelines). A pipeline rule can be introduced when a condition is fulfilled. The condition is defined under when:

  - pipeline: 'ok-pipeline'
      http.code: [200, 299]
  - pipeline: 'verybad-pipeline'
      http.code: [500, 999]
  - pipeline: 'default-pipeline'

Joda compatible date-time formatting in libbeat

The Go standard library way of formatting dates uses examples for layouts. This is an interesting alternative to the way formatting dates work in most other languages, but we felt that exposing this to our users would make things more complicated than needed. Not happy with the existing alternatives, Steffen created a new date-time format library, so you are able to use Joda compatible syntax. For example, you can specify a YYY.MMM.dd format to have 2016. Aug.01 as a result.

This makes it possible to have a configuration similar to Logstash when defining, for example, the index-pattern (currently in progress).

Rename redis.index with redis.key

The output.redis.index setting name has historical reasons, from the times that all outputs shared the same configuration options. The meaning of the index varies from output to output and in the case of Redis, it’s the key name. The PR introduced output.redis.key and deprecates output.redis.index.

In addition, output.file.index is deprecated in favor of output.file.filename.

Extend the Kibana dashboards for the System module of Metricbeat

More Kibana dashboards are created for the System module of Metricbeat to serve as an example for your own Kibana dashboards. A navigation bar is added on the left side (see screenshot below) to make it easier to navigate between all the Kibana dashboards created to visualize the data exported by the System module of Metricbeat:

  • An Overview with all the exported data types
  • Load and CPU statistics
  • Memory statistics
  • Per process statistics
  • Network statistics
  • Filesystem statistics

Filebeat: Unmarshal JSON inputs to integer by default

The standard json library in Golang unmarshals the integer values of the json object into floats instead of integers. This leaded to some unexpected behaviour when using the conditions from processors as you couldn’t easily compare a status code from the JSON object as it was translated to float64. The PR overwrites the Unmarshal behaviour and it tries to convert the numbers from the json objects to integers first, and if it fails then it converts them to floats. This way, 1 unmarshals as int64 and 1.0 as float64.

Metricbeat: Enhance load metrics

A new Metricset called load is exported by Metricbeat instead of exporting the load statistics inside the CPU statistics. With this PR  system.cpu.load.1 becomes system.load.1, system.cpu.load.5 becomes system.load.5 and system.cpu.load.15 becomes system.load.15. In addition, the load values divided by the number of cores are exported under system.load.norm.

Filebeat: Fix state remove and sending empty logs

When a very low scan_frequency is set, then it could happen that a state of a finished harvester was overwritten by the prospector and the state is never set to Finished. This is now fixed in that the prospector only sends a state when the state is set to Finished.

In addition, there is a fix to not send empty log lines.

Community Beats: Create pure Go binaries in packaging by default

The PR makes the Beat generator assume the Beat is pure Go (doesn’t have C dependencies). This simplifies the packaging process and produces fully static binaries by default. It’s still possible to create packages for the Beats that require Cgo, but you need to adjust the Makefile. Use the Beats packer Makefile as an example of the possible features.

Add support for cgroup in gosigar

The PR gives you the ability to ask gosigar for cgroup stats by PID. It returns metrics and limits from the blkio, cpu, cpuacct, and memory subsystems. This is part of a larger effort to build a solution on top of Beats to monitor containers.