Brewing in Beats: Mac installer for Beats

Welcome to Brewing in Beats! With these weekly series, we're keeping you up to date with what's new in Beats, including the latest commits and releases.

Did you know that Beats 6.3 is already available? Try it and let us know what you think. 

Better MacOS integration for Beats

We are improving our MacOS integration. We have created an installer package for each Beat and are bundling a preference pane that provides a minimal interface for controlling each Beat that you have installed. You can see more screenshots here.

beats-mac-installer.png

New galera_status metricset

Thanks to a contribution from the community we add support to Galera-specific metrics in the mysql module of Metricbeat. This will allow monitoring things like the flow control performance or the synchronization status of the nodes of a Galera cluster. These metrics will be collected by enabling the new galera_status metricset, that will be available on 6.4.0.

Autodiscover processors hint

We have added support to define processors from hints, when using Autodiscover. Both for Docker & Kubernetes, users will be able to use processors to attach processors to the config for each container from an annotation. For example, defining a dissect pattern, for instance:

co.elastic.logs/processors.dissect.tokenizer: "%{key2} %{key1}"

HTML Escaping

By default Beats escape HTML and XML contents in JSON strings. This is a problem for users, wanting to view, process, and ingest the original HTML/XML contents as is. With #7445 we introduce settings to disable HTML escaping.

Refactoring the Beat Packaging Build

We did a refactoring of the Beat packaging build to make it easier to extend with new platform targets and package types. As a nice side-effect packaging Beats is now faster. Due largely to higher CPU utilization the beats build gets about a 2x speedup in the release-manager CI jobs saving ~15m.

New Auditbeat Sub-Commands

In Auditbeat we had requests for adding a way to view the current audit rules loaded to the kernel and for listing the audit status. So we have added CLI commands to the Auditbeat executable to list this information. See the details in elastic/beats#7361.

All changes

Repository: elastic/beats

Affecting all Beats

Changes in 6.3:

  • Fix duplication of dynamic fields on reconnect #7352

Changes in master:

  • Set logp default for keepfiles to 7 #7495
  • Fix configuration-ssl docs for key param #7490
  • Minor: Small fixes for add_kubernetes_metadata #7466
  • Add additional types to kubernetes metadata #7457
  • Fix a panic when dealing with remaining at the end of a delimiter #7449
  • Html escaping #7445
  • Refactor asset generation #7441
  • Enforce normalization on Namespace and Name and check for nil factory #7426
  • missing license headers #7425
  • Use type alias to avoid remarshalling Pod events in Kubernetes watcher #7393
  • Refactor: Introduce a Global Registry #7392
  • Add processor definition support for hints builder #7387
  • Fix duplication of dynamic fields on reconnect #7352
  • libbeat: Refactor error handling in schema.Apply() #7335
Metricbeat

Changes in 6.3:

  • Fix wrong param in Metricbeat Kibana module reference config #7373

Changes in master:

  • Remove reference to removed error type #7491
  • Disable labels dedotting in default docker module configuration #7485
  • Missing headers on the mysql/galera_status #7450
  • Fix naming of fields for galera status #7435
  • Metricbeat module for Traefik #7413
  • Metricbeat: MongoDB TLS connection support #7401
  • Metricbeat: Galera module and status Metricset #6892
Packetbeat

Changes in master:

  • Update TLS protocol cipher suites (#7455) #7498
Filebeat

Changes in master:

  • Deprecation fileset for Elasticsearch filebeat module #7474
  • Add Slowlog fileset for the Elasticsearch module #7473
  • Add generation of generated files for Filebeat module tests #7430
  • Correctly parse ISO8601 dates with and without Timezone. #7379
  • Filebeat: Elasticsearch module: Audit log  #7365
Auditbeat

Changes in master:

  • Auditbeat: Add commands to show kernel rules and status #7361
Testing

Changes in master:

  • Add kubernetes to services which are kept running #7458
  • Improve autodiscover tests resiliency to timing issues #7434
Packaging

Changes in master:

  • Refactor Beat packaging and cross-building #7388
Documentation

Changes in 6.3:

  • Add missing docs for co.elastic.logs/disable hint #7406
  • Add warning about autodiscover and template scoping #7315
  • add note about overwriting dashboards #6828,
  • Update getting started intro and links #7357,
  • Minor doc fixes #7360,
  • Fix broken doc links #7370)
  • Document username/password options for HTTP based modules #7134

Changes in master:

  • Recommend using beat.hostname to fix visulations that use host #7471
  • Clarify autodiscover docs in relation to startup process #7468
  • Use hostNetwork for Metricbeat Deployment #7463
  • Clarify purpose of breaking changes doc #7453
  • Fix doc on output.kafka.version setting #7446
  • Add missing docs for kubernetes autodiscover provider parameters #7436
  • Add a NOTE to the to_syslog option to make it clear that the option is #7428
  • Updating docs to use new Fetch method signature #7421
  • Add missing docs for co.elastic.logs/disable hint #7406
  • Add warning about autodiscover and template scoping #7315

Changes in 6.2:

  • [docs] TLS extended key usage notes #6428
  • Minor doc fixes #7360