Beats 6.4.0 Released
Update: Due to a packaging mistake, the modules.d configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This manifests itself by showing an empty list on the metricbeat modules list command and failures to enable/disable any module. To workaround this issue, run the following command:
sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/
This should affect only new installations, upgrades should be fine because the old modules.d is left in place. The issue will be fixed in the 6.4.1 release.
Welcome to a new Elastic Beats release, bringing you the usual flurry of new features and improvements!
Let’s go through the highlights:
Dissect processor
The dissect filter plugin was introduced in Logstash in November 2016 as a less powerful but faster alternative to the good old Grok. Quoting from the original blog post:
”Essentially, field extraction in Dissect is a kind of 'split' operation and normally a split is done with one delimiter that is expected to be repeated in the source text. In contrast to Grok, which asks you to specify the regex patterns for fields of interest, Dissect asks you to specify the patterns of delimiters between the fields of interest. Dissect queues up the delimiters and searches the source text once from left to right, finding each successive delimiter in the queue. Different search algorithms are used for 1, 2 and 3+ byte delimiters.”
Because it doesn’t use regular expressions, Dissect is very fast and its performance is easier to predict. This makes it a very good fit for Beats, which strive to remain lightweight. So we have added a Dissect processor in Beats, implementing the same language that Logstash implements. A third implementation, inside the Elasticsearch Ingest Node, is currently in the works. A common test suite insures that the implementations don’t diverge. I think you can tell from this that we’re betting on Dissect at Elastic, so expect it to improve over time.
Here is an example usage inside the Beats configuration:
processors:
- dissect:
tokenizer: "%{key1} - %{key2}"
field: "message"
target_prefix: "extracted"
Seccomp syslog filtering (beta)
Starting with this version, on Linux 3.17 and later, Beats can take advantage of secure computing mode, also known as seccomp. Seccomp restricts the system calls that a process can issue.
The kernel exposes a large number of system calls that are not used by Beats. By installing a seccomp filter, you can limit the total kernel surface exposed to Beats (principle of least privilege). This minimizes the impact of unknown vulnerabilities that might be found in the process and it helps hardening the Beats installation.
The way it works is that Beats load a seccomp BPF filter at process start-up that drops the privileges to invoke specific system calls. Once a filter is loaded by the process it cannot be removed.
The seccomp policy can be customized through the configuration file, which can be used to further restrict the syscall surface area. This is an example blacklist policy that prohibits execve, execveat, fork, and vfork syscalls:
seccomp:
default_action: allow
syscalls:
- action: errno
names:
- execve
- execveat
- fork
- vfork
In related news, you can use Auditbeat to report any seccomp violations that occur on the system. The kernel generates an event for each violation and Auditbeat reports the event. The event.action value will be violated-seccomp-policy and the event will contain information about the process and system call.
Jolokia autodiscover provider (experimental)
Autodiscover is the Beats feature where modules can be enabled automatically when new processes and endpoints become available. An autodiscover provider creates the events that trigger the enabling or disabling of modules. The first two autodiscover providers are Docker and Kubernetes, which work based on the Docker/Kubernetes events.
The Jolokia autodiscover provider uses Jolokia Discovery to find agents running in your host or your network. The Jolokia Discovery mechanism is based on multicast UDP and is supported by any Jolokia agent since version 1.2.0, it is enabled by default when Jolokia is included in the application as a JVM agent, but disabled in other cases as the OSGI or WAR (Java EE) agents.
The data from the discovery event can be used in the module configuration via templates. Here is an example:
metricbeat.autodiscover:
providers:
- type: jolokia
interfaces:
- name: br*
interval: 5s
grace_period: 10s
- name: en*
templates:
- condition:
contains:
jolokia.server.product: "tomcat"
config:
- module: jolokia
metricsets: ["jmx"]
hosts: "${data.jolokia.url}"
namespace: test
jmx.mappings:
- mbean: "java.lang:type=Runtime"
attributes:
- attr: Uptime
field: uptime
This configuration starts a jolokia module that collects the uptime of each tomcat instance discovered. Discovery probes are sent using all interfaces starting with br and en, for the br interfaces the interval and grace_period is reduced to 5 and 10 seconds respectively.
Check out our blog post on how to monitor Java applications with Metricbeat and Jolokia for more details.
New Filebeat and Metricbeat Modules: Envoyproxy, Traefik, Kibana, and others
Like every Beats release, 6.4 introduces a few new modules as well as several significant improvements to the existing modules.
Worth noting are the Envoy Proxy and the Traefik modules in Metricbeat. Both are popular open source reverse proxies used in cloud native infrastractures, especially on top of Kubernetes. Speaking of Kubernetes, the kubernetes module in Metricbeat has got a new metricset: apiserver, which collects the metrics from the Kubernetes API Server component, which exposes the Kubernetes API.
The Beats modules that are monitoring the Elastic stack were also significantly improved, making it easier to monitor the Elastic stack with the Beats: To start, Filebeat now has a module for reading the Elasticsearch logs and one for the Kibana logs. The Elasticsearch Filebeat module has several file sets: server, GC, audit log, slow log, deprecation. The module for Elasticsearch in Metricbeat got a lot of new metricsets: index, index_summary, shard, ml_jobs, index_recovery, cluster_stats and the existing metricsets were enhanced.
Other modules that were improved include RabbitMQ, PostgreSQL and MongoDB.
Auditbeat: XXH64 hashing and Kubernetes deployment
The file integrity module in Auditbeat can now do XXH64 hashes. This hash algorithm is non-cryptographic but extremely fast. Our benchmarks show that it’s 5x faster than any other hash function offered by Auditbeat today.
With this version, Auditbeat joins Filebeat and Metricbeat in the list of Beats for which we offer sample Kubernetes manifest files. For now, the sample manifest files deploy the file integrity module only.
Packetbeat: Process monitoring improvements
Packetbeat is able, since its first version, to correlate the network connections to the process owning that connection. This can be very useful in a number of situations. In 6.4, this functionality is getting a significant upgrade:
- Add support for Windows process monitoring (it used to work only on Linux).
- Report the full command-line for all processes (no longer needed to explicitly configure them)
- Disable the monitor when using file input.
- Get rid of polling for processes and sockets.
- Add UDP support.
- Add the process monitoring capabilities to the flows feature.
If process monitoring is enabled, the following fields are added to each flow:
proc: Process name for the server-side of the flow.cmdline: Command-line for the local process on the server-side of the flow.client_proc: Process name for the client-side of the flow.client_cmdline: Command-line for the local process on the client-side of the flow.
Feedback
If you want to make use of the new features added in Beats 6.4.0, please download it, install it, and let us know what you think on Twitter (@elastic) or in our forum.
Enjoy!