Brewing in Beats: Configure the Ingest Node Pipeline
New community Beat: Cassandrabeat
Cassandrabeat uses Cassandra’s nodetool cfstats utility to monitor Cassandra database nodes and lag. Please give it a try and let us know what do you think.
Select pipeline for Ingest Node from Beats
There are three options to define the pipeline. One is to define a single pipeline (under
output.elasticsearch.pipeline) for all your events, in which you can access other fields from the event:
Another one is to define an array of pipeline rules (under
output.elasticsearch.pipelines). A pipeline rule can be introduced when a condition is fulfilled. The condition is defined under
output.elasticsearch.pipelines: - pipeline: 'ok-pipeline' when.range: http.code: [200, 299] - pipeline: 'verybad-pipeline' when.range: http.code: [500, 999] - pipeline: 'default-pipeline'
Joda compatible date-time formatting in libbeat
The Go standard library way of formatting dates uses examples for layouts. This is an interesting alternative to the way formatting dates work in most other languages, but we felt that exposing this to our users would make things more complicated than needed. Not happy with the existing alternatives, Steffen created a new date-time format library, so you are able to use Joda compatible syntax. For example, you can specify a
YYY.MMM.dd format to have
2016. Aug.01 as a result.
This makes it possible to have a configuration similar to Logstash when defining, for example, the index-pattern (currently in progress).
Rename redis.index with redis.key
output.redis.index setting name has historical reasons, from the times that all outputs shared the same configuration options. The meaning of the index varies from output to output and in the case of Redis, it’s the key name. The PR introduced
output.redis.key and deprecates
output.file.index is deprecated in favor of
Extend the Kibana dashboards for the System module of Metricbeat
More Kibana dashboards are created for the System module of Metricbeat to serve as an example for your own Kibana dashboards. A navigation bar is added on the left side (see screenshot below) to make it easier to navigate between all the Kibana dashboards created to visualize the data exported by the System module of Metricbeat:
- An Overview with all the exported data types
- Load and CPU statistics
- Memory statistics
- Per process statistics
- Network statistics
- Filesystem statistics
Filebeat: Unmarshal JSON inputs to integer by default
The standard json library in Golang unmarshals the integer values of the json object into floats instead of integers. This leaded to some unexpected behaviour when using the conditions from processors as you couldn’t easily compare a status code from the JSON object as it was translated to float64. The PR overwrites the Unmarshal behaviour and it tries to convert the numbers from the json objects to integers first, and if it fails then it converts them to floats. This way,
1 unmarshals as int64 and
1.0 as float64.
Metricbeat: Enhance load metrics
A new Metricset called
load is exported by Metricbeat instead of exporting the load statistics inside the CPU statistics. With this PR
system.load.15. In addition, the load values divided by the number of cores are exported under
Filebeat: Fix state remove and sending empty logs
When a very low
scan_frequency is set, then it could happen that a state of a finished harvester was overwritten by the prospector and the state is never set to Finished. This is now fixed in that the prospector only sends a state when the state is set to Finished.
In addition, there is a fix to not send empty log lines.
Community Beats: Create pure Go binaries in packaging by default
The PR makes the Beat generator assume the Beat is pure Go (doesn’t have C dependencies). This simplifies the packaging process and produces fully static binaries by default. It’s still possible to create packages for the Beats that require Cgo, but you need to adjust the Makefile. Use the Beats packer Makefile as an example of the possible features.
Add support for cgroup in gosigar
The PR gives you the ability to ask gosigar for cgroup stats by PID. It returns metrics and limits from the
memory subsystems. This is part of a larger effort to build a solution on top of Beats to monitor containers.