Brewing in Beats: Osquery module in Filebeat

Did you know that Beats 6.1 is already available? Try it and let us know what you think. If you are curious to see the Beats in action, we just published the Getting Started with Beats webinar.

This update covers the last two weeks.

Filebeat: Osquery module

Osquery is an OS instrumentation framework that uses SQL as an interface for querying the underlying data. This PR adds a new Filebeat module that collects and indexes the result logs written by osqueryd. Osquery writes the results in JSON, which makes it fairly easy to ingest to the Elastic stack. The module uses the JSON decoding support in Filebeat, and then renames the fields to match the Beats naming conventions (most fields prefixed with osquery.result).

This module comes with Dashboards for two of the Osquery packs: it-compliance and ossec-rootkit. It is therefore recommended to enable these packs in the osqueryd configuration.

The osquery module is scheduled to be released in 6.2.

Kubernetes autodiscovery

Similar to the Docker autodiscovery provider, which was introduced in 6.1, we are now adding a Kubernetes autodiscovery provider. It works by listening to Kubernetes events and allowing the user to define custom configs for them. For instance, it's able to detect a new Pod running NGINX started and enable the NGINX module for it, in both Filebeat and Metricbeat.

The Kubernetes autodiscovery provider is contributed by the long-term Beats contributor Vijay Samuel from eBay and is targeted for the 6.2 release.

Here are sample configurations:

Filebeat: (discover all workloads in default namespace)

filebeat.autodiscover:
  providers:
    - type: kubernetes
      in_cluster: false
      kube_config: /home/docker/config
      host: minikube
      templates:
        - condition:
            equals:
              kubernetes.namespace: default
          config:
            - type: docker
              containers.ids:
                - "${data.kubernetes.container.id}"

Metricbeat: query for pods with label project: prometheus

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      in_cluster: false
      kube_config: /home/docker/config
      host: minikube
      templates:
        - condition:
            equals:
              kubernetes.labels.project: prometheus
          config:
            - module: prometheus
              metricsets: ["stats"]
              hosts: "${data.host}:9090"

Add docker metadata support for Auditbeat

The add_docker_metadata processor from libbeat is the main tool that we have for enhancing the Beats events with Docker metadata (container name, image, labels, etc.). With this PR, add_docker_metadata can do the correlation also based on the process id, which means that it can be used with Auditbeat data.  This new feature will be available in 6.2.

Bonus: follow the PR link to find the secret link between the Spanish cuisine and Docker.

Filebeat prospector renaming to input

The prospector naming in Filebeat dates back from the logstash-forwarder days. But as we’re adding multiple prospector types, in which some of them deal with network connections rather than files, we decided to rename them to inputs. This PR does the renaming in the code. From the configuration file POV, both the prospector and input names will work interchangeably, but prospector generates a deprecation warning. The meta ticket for tracking the renaming progress can be found here.

Other changes:

Repository: elastic/beats

Affecting all Beats

Changes in master:

• Refuse to store dotted keys to prevent cyclic reference in our configuration. #6077
• Fix CPU usage report in process.GetOne #6076
• Adapt kibana index-pattern cleanup to changed dir structure. #6069
• move compile time version information #6050
• New metric: RSS #6039
• Remove dots from keys in monitoring events #6038
• permit reporter defaultParams overrides #6032
• New beat info: ephemeral_id #6028

Metricbeat

Changes in master:

• Fix uwsgi kibana folder name #6082
• Fix metricbeat mongodb collstats schema #6072
• Fix process cgroup memory values #6034
• Change kubernetes.node.cpu.allocatable.cores to float #6130
• Fix float32 normalizeValue panic #6129
• Use official client in Docker module #6113
• Fix the consumergroup of kafka module incorrectly work #5880

Filebeat

Changes in master:

• Add a parameter to docker prospector to filter on stream #6057
• Module generator tools for Filebeat #5544

Auditbeat

Changes in master:

• Renamed auditd module fields #6080
• Warn if auditd is running #6023
• Skip trailing zeroes in kernel version #6109
• Update go-libaudit to v0.0.7 #6104
• Update Auditbeat docs / fields / dashboards #6089

Changes in 6.x:

• Skip trailing zeroes in kernel version #6109

Dashboards

Changes in master:

• Fix osquery Kibana folder name #6081

Infrastructure

Changes in 6.x:

• Revert "move compile time version information (#6050)" #6087

Documentation

Changes in master:

• Add kubernetes autodiscover docs #6065
• Improve generated documentation #6037
• Add faq about disabling ssl to capture mysql data #5846
• Promote Docker/Kubernetes modules & processors to GA #6105
• Clarify docs about reloading config files #6046

Changes in 6.2:

• Upgrade branch to 6.2 #6139

Changes in 6.1:

• Bump docs version for 6.1.2 #6075
• Close changelog 6.1.2 #6074
• Add faq about disabling ssl to capture mysql data #5846
• Clarify naming of multiline settings #6014

Changes in 5.6:

• Bump docs version 5.6.6 #6073
• Close changelog 5.6.6 #6071

Repository: elastic/gosigar

Changes in master:

• Add changelog for #91 #94
• Add missing error checks and file handle closes #92
• Prepare 0.8.0 Release #96
• Add getrusage for Unix and partially for Windows #95

Repository: elastic/go-libaudit

Changes in master:

• Decoding fixes and event format changes #17
• Unregister PID from kernel on client close #19
• Ignore unexpected notifications in getReply #18