Brewing in Beats: Optimize matching regular expressions
Welcome to the weekly Brewing in Beats ! With this weekly series, we are keeping you up to date with all the changes in Beats, including the latest commits, releases and other learning resources.
Optimize matching regular expressions
Steffen wasn’t happy with the performance of the regular expressions implementation in the Go standard library so he optimized some of the most common use cases by factors up to 10x and even 100x. The idea of the optimization is that in Beats we accept regular expressions in places where most people use them for simple substring searches (e.g. exclude_lines, include_lines, or multiline), so we can automatically detect these cases in regexps and switch to faster implementations. See the PR for benchmark results.
Community Beat: Apexbeat
Apexbeat extracts configurable contextual data and metrics from Java applications via the APEX toolkit and indexes them into Elasticsearch.
Community Beat: Connbeat
Connbeat collects TCP connections metadata to index them in Elasticsearch. For each connection, it gives you details about the IP, port of the endpoints involved, together with details about the local process. You can use it to monitor the connections in Docker instances, and you get extra metadata about your Docker container. The exported data are similar to the ones provided by the system.socket metricset in Metricbeat. For now, it works only on Linux systems.
Initial setup of Filebeat modules
With this change, the -setup option is added to the Filebeat command line for loading the Ingest Node pipeline at startup if the Elasticsearch output is configured. In case Elasticsearch is not available, then Filebeat fails to start with an error.
For example, the following command includes the -setup
option to initialize the Filebeat Nginx module before running Filebeat:
filebeat -e -modules nginx -setup
In the near future, we are planning to load also the Kibana dashboards, in addition to the Ingest Node pipeline, when using the -setup
flag.
Other features and fixes
All Beats
- Pass additional metadata to the outputs. As the first use case, the pipeline is passed as metadata to the Elasticsearch output #3359
- Cleanup before_build script that it’s executed before creating Beats packages #3386
- As Elasticsearch is planning to remove the support for types in the future, the
_type
field is marked as deprecated in 5.3 and it will be removed in 6.0. Instead of_type
, you can use thetype
field. #3409 - Generate the templates before testing to use the latest version #3425
Filebeat
- Make optional the module directory with all the Filebeat modules #3405
Heartbeat
- Set default settings in the http monitor check #3402
- Update the full configuration file and use Elasticsearch as the default monitored http endpoint #3401
- Fix failure when ICMP is configured and IPv6 support is disabled on boot or not available in kernel #3414
Metricbeat
- Update docker client to the latest version #3398
- Add HTTP helper to be used for HTTP based metricsets #3413
Generators
- Set the default period to 10s in the Metricbeat module generator #3377