Brewing in Beats: Http module in Metricbeat
Welcome to Brewing in Beats! With this weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.
New community Beat: Rsbeat
Rsbeat is built on top of libbeat infrastructure to collect the Redis slow logs and ship them to Elasticsearch directly or via Logstash. The slow logs are accumulated by Redis in memory, so no files are written with information about the slow command executions. Rsbeat interrogates Redis to retrieve the slow logs.
Metricbeat: new http module
Thanks to Christian Galsterer, the creator of the successful Httpbeat, Metricbeat has a generic module for collecting data periodically from multiple HTTP endpoints, and indexing the result in Elasticsearch. The http module is especially useful when no dedicated Metricbeat module is available, but there is a HTTP based monitoring endpoint.
In case the HTTP endpoint returns the result in a json format, you can configure the json
metricset to export all the JSON fields under a configured namespace field as shown in the following example:
{ "@timestamp": "2017-05-01T13:00:24.745Z", "http": { "http_json_namespace": { "date": "05-01-2017", "milliseconds_since_epoch": 1493643625474.000000, "time": "01:00:25 PM" } }, ... }
Dashboards using the Time Series Visual Builder
The system module of Metricbeat gets two new Kibana dashboards, built using the new Time Series Visual Builder that will be added with the new Kibana 5.4 release. You can watch a demo of the time series visual builder during the Kibana visualizations deep dive.
Metricbeat: report the topN processes
This adds the option to only report on the top N processes by CPU and/or memory. It is useful because storing metrics about each and every process from every host can be quite expensive from a storage point of view. Previously it was possible to filter processes by name, which was useful if one knew in advance which are the most interesting processes. This adds a new option which should be quite convenient in practice, because the number of per-process documents gets limited while still allowing to display the top processes.
Add Alibaba Cloud metadata
Thanks to the community contribution, the add_cloud_metadata
processor enhances each event with Alibaba Cloud metadata. This metadata is only available when VPC is selected as the network type of the instance.
Other changes
Repository: elastic/beats
Affecting all Beats
Changes in 5.4:
- @timestamp doesn't get printed when specified in message codec #3721
Changes in master:
- Ensure correct permissions for config files when running
make update
#4137 - Remove duplicate code in glob_watcher_test.go #4117
Processors
Changes in master:
- Make Kubernetes indexers/matchers pluggable #4151
Filebeat
Changes in master:
- Move handling of JSON fields to harvester #4159
- Extract stdin prospector #4158
- Cleanup prospector shutdown #4157
- Disable default prospector and adjust short configs #4105
- Refactor harvester to send events directly to the spooler #4070
- Filebeat: expand double wildcards in prospector #3980
Winlogbeat
Changes in 5.x:
- Deprecate the metrics endpoint in Winlogbeat #4145
Metricbeat
Changes in master:
- New http module #4156
- Fixing multiEventFetch error reporting when no events are returned #4153
- Fixing nil pointer on Prometheus collector when HTTP response is nil #4119
Heartbeat
Changes in master:
Infrastructure
Changes in master:
- Adding goimports support to make check and fmt #4114
Changes in 5.3:
- Fix link to the MacOSX SDK tarball #4120
Changes in 5.4:
- Fix link to the MacOSX SDK tarball #4120
Documentation
Changes in 5.4:
- Backport: Cherrypick #4148, #4108, #4095 into 5.4 #4161
Changes in 5.3:
Changes in master:
- Forward port: Cherrypick #4095 into master #4162
- Wrong "make collect" statement documented instead of "make update" when documentation should be regenerated #4155
- Fix config example #4148
- Remove docker statement #4147
- Fix Jolokia docs #4140
- Add note about editor config to CONTRIBUTING.md #4136
- Add community beats topic to dev guide #4134
Dashboards
Changes in master:
- Update host details dashboard to use Series Agg #4146