<p>The Beats 6.5 release is here! This is probably our most feature rich minor release since we’ve started the Beats project. Central config, dedicated UIs for visualizing Beats data, two net-new Beats, Heartbeat GA, and <a href="https://www.elastic.co/guide/en/beats/libbeat/6.5/release-notes-6.5.0.html">the list goes on and on</a>. A lot of the new functionality is introduced as Beta, so we look forward to your early feedback on these features.<span></span>
</p><h2>Beats central management (Beta)</h2><p>You can now control your Beats from Kibana! You can enable and configure Filebeat and Metricbeat modules, stop modules, and re-configure the output. It’s all controlled via configuration tags, so it’s easy to manage tens of thousands of Beats (if you are wondering, yes, there are many deployments with tens of thousands of Beats).
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt57e975f3fb47966c/5c304ec7e7a0881470c3a46d/Image_from_iOS.jpg" data-sys-asset-uid="blt57e975f3fb47966c" alt="Image from iOS.jpg">
</p><p>To use central management, you still need to install Filebeat/Metricbeat like before, but instead of editing the configuration file, you “enroll” the Beat to Kibana. The enrollment can be token based or password based. In the token based enrollment, the user needs to copy-paste and run a command like:
</p><pre class="prettyprint">$ filebeat enroll &lt;kibana-url&gt; &lt;token&gt;
</pre><p> where <code>&lt;token&gt;</code> can be copy-pasted from the UI.
</p><p> When doing password based enrollment, the <code>enroll</code> command will prompt for the password for the <code>elastic</code> user. This can also be scripted. After enrolment, a unique token for that particular Beat instance is saved. That means that a Beat can retrieve its own configuration, but nothing else. The enrolment API allows you to automate this process, so this works for large deployments as well.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte4781905e1e894ca/5c304cffc05f480270a742c8/Screen_Shot_2018-11-08_at_19.12.01.png" data-sys-asset-uid="blte4781905e1e894ca" alt="Screen Shot 2018-11-08 at 19.12.01.png">
</p><p>After enrollment, it’s really easy to configure a new glob in Filebeat or enable a new module Metricbeat:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb0afa02543fd4d46/5c304ebf29d13af10bc2ab34/Screen_Shot_2018-11-01_at_10.35.55_AM.png" data-sys-asset-uid="bltb0afa02543fd4d46" alt="Screen Shot 2018-11-01 at 10.35.55 AM.png">
</p><p>In this Beta release, only Filebeat and Metricbeat are supported.
</p><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/dad2fH1uGqgqbLxjC4BPrK.jpg" data-uuid="dad2fH1uGqgqbLxjC4BPrK" data-v="4" data-type="inline">
</div><h2>Infrastructure and Logs&nbsp;Kibana apps (Beta)</h2><p>After installing Beats and Kibana 6.5, you will notice two new icons on the Kibana navigation bar. These are the Infrastructure UI and the Logs UI, two new Kibana apps that are the perfect companions for Metricbeat and Filebeat.
</p><p>Below is a screenshot of the Infrastructure app, using a diagram that we call "waffle map". Each box in the waffle map can be a host/VM instance, a docker container, or a pod. The view comes with a search bar to filter the boxes that have a specific characteristic and with a "group by" functionality to group the boxes by certain fields like availability zone or the Kubernetes namespace. The color of each box represents the value of the selected metric, in the case of the screenshot, inbound traffic.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7c003bdd60b4ea60/5c6132ca00babade4322377f/screenshot-systems-monitoring.jpg" data-sys-asset-uid="blt7c003bdd60b4ea60" alt="screenshot-systems-monitoring.jpg">
</p><p>If you click on one of these boxes, you can get more details about the host itself as in the screenshot below. Here you can see not only &nbsp;metrics like CPU usage, load, memory usage, and others, but also the logs that are coming from that host. If the host is a Kubernetes node, you can also see Kubernetes specific metrics.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt14a145c0c17e95d2/5c304ea1b7017c456eff0995/screenshot-metrics-infrastructure-monitoring.jpg" data-sys-asset-uid="blt14a145c0c17e95d2" alt="screenshot-metrics-infrastructure-monitoring.jpg">
</p><p>The Logs&nbsp;UI is designed to give you a view to your log files that resembles the good old <code>tail -f</code>, but still give you the fantastic power of Elasticsearch. Compared to the now venerable Discover tab, the Logs&nbsp;UI sorts the entries from older to newer, has infinite scrolling in both directions, live streaming, and a minimap for quick navigation and viewing the log volume. The search bar at the top allows easy searching and filtering using auto-complete. You have to try it, I think you’ll agree that this makes scrolling through logs a lot more pleasant.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4ae83f5d8bdc8cea/5c304cf4c599e169662410fe/Screen_Shot_2018-11-08_at_19.31.42.png" data-sys-asset-uid="blt4ae83f5d8bdc8cea" alt="Screen Shot 2018-11-08 at 19.31.42.png">
</p><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/jS82ptTDJhQXBUN9KgqTT6.jpg" data-uuid="jS82ptTDJhQXBUN9KgqTT6" data-v="4" data-type="inline">
</div><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/Z1HogQG1idP8Wnf74mJLHi.jpg" data-uuid="Z1HogQG1idP8Wnf74mJLHi" data-v="4" data-type="inline">
</div><h2>Functionbeat (Beta)</h2><p>Functionbeat is not only a new Beat, but a new <strong>kind</strong> of Beat. Instead of running on your servers, it runs, well... serverless. The idea is that it can be deployed as an AWS Lambda function, so that if you want to move logs from, say, Cloudwatch to our <a href="https://www.elastic.co/cloud">hosted Elasticsearch Service</a>, you can do that without having to deploy an EC2 instance or similar.
</p><p>The way it works is that you download a tar.gz on your computer, like with any Beat, but instead of running it locally, you deploy it to Lambda.
</p><p>Here is a sample config:
</p><pre class="prettyprint">functionbeat.provider.aws.deploy_bucket: "functionbeat-deploy" functionbeat.provider.aws.functions: - name: cloudwatch type: cloudwatch_logs triggers: - log_group_name: /aws/lambda/functionbeat-cloudwatch_logs cloud.id: … cloud.auth: ...
</pre><p> If you then run <code>./functionbeat deploy cloudwatch</code>, it will create an AWS Lambda function, via Cloudformation, that copies the logs from that Cloudwatch group to the configured Elasticsearch instance.
</p><p> The initial release runs in AWS Lambda, but we plan adding support for Azure and GCP as well. The supported triggers are Cloudwatch Logs and SQS. Functionbeat also supports Kinesis and the API Gateway but doesn’t yet expose CLI function to deploy them.
</p><p> Implementation wise, Functionbeat is built on top of the same libbeat platform as the rest of the Beats. This means that features like filtering and index/pipeline configuration work the same way as in Filebeat, for example. In the future, it will also be possible to run modules on top of Functionbeat.
</p><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/jybcyNJ1XavEmErHEKXJ26.jpg" data-uuid="jybcyNJ1XavEmErHEKXJ26" data-v="4" data-type="inline">
</div><h2>Journalbeat (Experimental)</h2><p> As journald is very popular on Linux distros, we want to provide a very good experience around ingesting the logs stored in it, just like we have for text logs with Filebeat and Windows Events with Winlogbeat. A community Journalbeat already exists, and plenty of people are using it, but it is time for Elastic to provide an official version. We want to thank <a href="https://github.com/mheese/">@mheese</a> for maintaining Journalbeat for such a long time as a community Beat.
</p><p> Just like Filebeat and Winlogbeat, Journalbeat maintains in a registry file how far it progressed in the logs, so it knows where to continue on restarts. It also offers “at-least-once” guarantees and is sensitive to back-pressure. It collects all the metadata stored in the journal, like the process executable, process id, or the user, and you can easily filter by any of these fields.
</p><p> Journalbeat is currently marked as experimental because we have the intention to transform it into a Filebeat input. We do encourage everyone that needs Journald support to try it and provide us with feedback.
</p><h3>Known issues</h3><ul>
	<li>Journalbeat cannot be stopped properly when the output is unreachable and it is attempting to connect</li>
	<li>Reference configuration is incorrect. Default value of journalbeat.seek is cursor. Furthermore, journalbeat.matches needs to be journalbeat.include_matches instead.</li>
	<li>Journalbeat does not put the registry file under data folder specified by path.data.</li>
</ul><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/GuXSD51XEhAW8tQsmVXL65.jpg" data-uuid="GuXSD51XEhAW8tQsmVXL65" data-v="4" data-type="inline">
</div><h2>Heartbeat is now GA</h2><p> During its Beta phase, we’ve seen good adoption and interesting use cases for Heartbeat, so we are very happy to announce its general availability starting with the 6.5.0 release. As a reminder, Heartbeat is a lightweight daemon that you install on a remote server to periodically check the status of your services and determine whether they are reachable. Heartbeat tests service availability via TCP, HTTP, and ICMP checks.
</p><p> Heartbeat now supports configuration reloading in the same way that Filebeat and Metricbeat do. This makes it easy to dynamically add/remove monitors by writing/removing configuration files on disk. Taking it one step further, it is now also possible to use the libbeat Autodiscover functionality. This means that Heartbeat can discover new services, for example, by listening to Docker or Kubernetes events. In the future, as we add more Autodiscover providers, this will allow Heartbeat to discover, for example, AWS EC2 and ELB instances.
</p><p> When monitoring TLS endpoints one often wants to be alerted when an x509 certificate is approaching its expiration date. We now record a variety of certificate metadata in our TLS checks, including certificate expiration date. Combine this with a watch to create a reminder to renew your certificates before they expire.
</p><p>For more details refer to the <a href="https://www.elastic.co/blog/uptime-monitoring-with-heartbeat-and-the-elastic-stack">Uptime Monitoring with the&nbsp;Elastic Stack blog</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7958c2db6d8fac2a/5c304cd06d4494e30b7b54b3/simple-heartbeat-dashboard_(1).jpg" data-sys-asset-uid="blt7958c2db6d8fac2a" alt="simple-heartbeat-dashboard (1).jpg">
</p><h2>Suricata Integration via Filebeat module</h2><p><a href="https://suricata-ids.org/">Suricata</a> is an open source real-time intrusion detection system (IDS) and network monitoring system (NMS). Integrating Suricata with the Elastic stack is already a common choice for security professionals. With this release, we’re making this integration easier by providing a Filebeat module that collects the Suricata Eve logs. The module renames several fields so they match with the (in progress) <a href="https://github.com/elastic/ecs">Elastic Common Schema (ECS)</a>. The module also comes with a couple of Kibana dashboards.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt38e3f496deaf4fa1/5c304cf029d13af10bc2ab26/kibana-suricata.png" data-sys-asset-uid="blt38e3f496deaf4fa1" alt="kibana-suricata.png">
</p><h2>Modules improvements</h2><p>Like any Beats release, 6.5 brings several improvements to our existing Filebeat and Metricbeat modules, as well as new modules. To name a few:
</p><ul>
	<li>Kafka Metricbeat module is now GA and both the Metricbeat and the Filebeat modules now support Kafka 2.0.0. Metricbeat module also includes a dashboard with consumer group lag visualization.</li>
	<li>Filebeat Docker input is marked as GA.</li>
	<li>Cross-cluster-replication support in the Elasticsearch Metricbeat module</li>
	<li>The Metricbeat <code>redis</code> module now collects more memory stats and supports changes introduced in redis 5.0.</li>
	<li>Added new metrics about cache size to Metricbeat <code>memcached</code> module.</li>
	<li>New <code>replstatus</code> and <code>metrics</code> metricsets in the MongoDB Metricbeat module.</li>
	<li>New <code>haproxy</code> module in filebeat with support for default log formats (default, tcplog and httplog).</li>
	<li>New <code>process</code> metricset in <code>php_fpm</code> Metricbeat module to collect list of processes in php fpm pools using the full status report.</li>
	<li>New <code>socket_summary</code> metricset in system module to collect a summary of open connections and listening ports.</li>
</ul>

blog-beats-banner.jpg

blog-beats-thumb.jpg

Beats 6.5.0 released

Welcome to the 6.4.0 Elastic Beats release, bringing you the usual flurry of new features and improvements!

<html><head></head><body><p><strong>Update:</strong> Due to a packaging mistake, the <code>modules.d</code> configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This manifests itself by showing an empty list on the <code>metricbeat modules list</code> command and failures to enable/disable any module. To workaround this issue, run the following command:
</p><pre class="prettyprint">sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/
</pre><p>This should affect only new installations, upgrades should be fine because the old modules.d is left in place.&#xA0;The issue will be fixed in the 6.4.1 release.
</p><p><br>
</p><p>Welcome to a new Elastic Beats release, bringing you the usual flurry of new features and improvements!
</p><ul>
	<li><a href="/downloads/beats">Download link</a></li>
	<li><a href="/guide/en/beats/libbeat/6.4/release-notes-6.4.0.html">Release notes</a></li>
</ul><p>Let&#x2019;s go through the highlights:
</p><h2>Dissect processor
</h2><p>The dissect filter plugin <a href="/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs">was introduced in Logstash in November 2016</a> as a less powerful but faster alternative to the good old Grok. Quoting from the original blog post:
</p><blockquote>&#x201D;Essentially, field extraction in Dissect is a kind of &apos;split&apos; operation and normally a split is done with one delimiter that is expected to be repeated in the source text. In contrast to Grok, which asks you to specify the regex patterns for fields of interest, Dissect asks you to specify the patterns of delimiters between the fields of interest. Dissect queues up the delimiters and searches the source text once from left to right, finding each successive delimiter in the queue. Different search algorithms are used for 1, 2 and 3+ byte delimiters.&#x201D;
</blockquote><p>Because it doesn&#x2019;t use regular expressions, Dissect is very fast and its performance is easier to predict. This makes it a very good fit for Beats, which strive to remain lightweight. So we have added a Dissect processor in Beats, implementing the same language that Logstash implements. A third implementation, inside the Elasticsearch Ingest Node, is currently in the works. A common test suite insures that the implementations don&#x2019;t diverge. I think you can tell from this that we&#x2019;re betting on Dissect at Elastic, so expect it to improve over time.
</p><p>Here is an example usage inside the Beats configuration:
</p><pre class="prettyprint"> processors:
 - dissect:
    tokenizer: &quot;%{key1} - %{key2}&quot;
    field: &quot;message&quot;
    target_prefix: &quot;extracted&quot;
</pre><h2>Seccomp syslog filtering (beta)
</h2><p>Starting with this version, on Linux 3.17 and later, Beats can take advantage of secure computing mode, also known as seccomp. Seccomp restricts the system calls that a process can issue.
</p><p>The kernel exposes a large number of system calls that are not used by Beats. By installing a seccomp filter, you can limit the total kernel surface exposed to Beats (principle of least privilege). This minimizes the impact of unknown vulnerabilities that might be found in the process and it helps hardening the Beats installation.
</p><p>The way it works is that Beats load a seccomp BPF filter at process start-up that drops the privileges to invoke specific system calls. Once a filter is loaded by the process it cannot be removed.
</p><p>The seccomp policy can be customized through the configuration file, which can be used to further restrict the syscall surface area. This is an example blacklist policy that prohibits execve, execveat, fork, and vfork syscalls:
</p><pre class="prettyprint"> seccomp:
  default_action: allow 
  syscalls:
  - action: errno 
    names: 
    - execve
    - execveat
    - fork
    - vfork
</pre><p>In related news, you can use <a href="/products/beats/auditbeat">Auditbeat</a> to report any seccomp violations that occur on the system. The kernel generates an event for each violation and Auditbeat reports the event. The event.action value will be violated-seccomp-policy and the event will contain information about the process and system call.
</p><h2>Jolokia autodiscover provider (experimental)
</h2><p>Autodiscover is the Beats feature where modules can be enabled automatically when new processes and endpoints become available. An autodiscover provider creates the events that trigger the enabling or disabling of modules. The first two autodiscover providers are Docker and Kubernetes, which work based on the Docker/Kubernetes events.
</p><p>The Jolokia autodiscover provider uses <a href="https://jolokia.org/reference/html/protocol.html#discovery">Jolokia Discovery</a> to find agents running in your host or your network. The Jolokia Discovery mechanism is based on multicast UDP and is supported by any Jolokia agent since version 1.2.0, it is enabled by default when Jolokia is included in the application as a JVM agent, but disabled in other cases as the OSGI or WAR (Java EE) agents.
</p><p>The data from the discovery event can be used in the module configuration via templates. Here is an example:
</p><pre class="prettyprint"> metricbeat.autodiscover:
  providers:
    - type: jolokia
      interfaces:
      - name: br*
        interval: 5s
        grace_period: 10s
      - name: en*
      templates:
      - condition:
          contains:
            jolokia.server.product: &quot;tomcat&quot;
        config:
        - module: jolokia
          metricsets: [&quot;jmx&quot;]
          hosts: &quot;${data.jolokia.url}&quot;
          namespace: test
          jmx.mappings:
          - mbean: &quot;java.lang:type=Runtime&quot;
            attributes:
            - attr: Uptime
              field: uptime
</pre><p>This configuration starts a jolokia module that collects the uptime of each tomcat instance discovered. Discovery probes are sent using all interfaces starting with br and en, for the br interfaces the interval and grace_period is reduced to 5 and 10 seconds respectively.
</p><p>Check out our <a href="/blog/monitoring-java-applications-with-metricbeat-and-jolokia">blog post&#xA0;on how to monitor Java applications with Metricbeat and Jolokia</a> for more details.
</p><h2>New Filebeat and Metricbeat Modules: Envoyproxy, Traefik, Kibana, and others
</h2><p>Like every Beats release, 6.4 introduces a few new modules as well as several significant improvements to the existing modules.
</p><p>Worth noting are the Envoy Proxy and the Traefik modules in Metricbeat. Both are popular open source reverse proxies used in cloud native infrastractures, especially on top of Kubernetes. Speaking of Kubernetes, the kubernetes module in Metricbeat has got a new metricset: apiserver, which collects the metrics from the Kubernetes API Server component, which exposes the Kubernetes API.
</p><p>The Beats modules that are monitoring the Elastic stack were also significantly improved, making it easier to monitor the Elastic stack with the Beats: To start, Filebeat now has a module for reading the Elasticsearch logs and one for the Kibana logs. The Elasticsearch Filebeat module has several file sets: <code>server</code>, <code>GC</code>, <code>audit log</code>, <code>slow log</code>, <code>deprecation</code>. The module for Elasticsearch in Metricbeat got a lot of new metricsets: <code>index</code>, <code>index_summary</code>, <code>shard</code>, <code>ml_jobs</code>, <code>index_recovery</code>, <code>cluster_stats</code> and the existing metricsets were enhanced.
</p><p>Other modules that were improved include RabbitMQ, PostgreSQL and MongoDB.<br>
</p><h2>Auditbeat: XXH64 hashing and Kubernetes deployment
</h2><p>The file integrity module in Auditbeat can now do <a href="https://cyan4973.github.io/xxHash/">XXH64</a> hashes. This hash algorithm is non-cryptographic but extremely fast. Our benchmarks show that it&#x2019;s 5x faster than any other hash function offered by Auditbeat today.
</p><p>With this version, Auditbeat joins Filebeat and Metricbeat in the list of Beats for which we offer sample Kubernetes manifest files. For now, the <a href="https://github.com/elastic/beats/tree/master/deploy/kubernetes/auditbeat">sample manifest files</a> deploy the file integrity module only.
</p><h2>Packetbeat: Process monitoring improvements
</h2><p>Packetbeat is able, since its first version, to correlate the network connections to the process owning that connection. This can be very useful in a number of situations. In 6.4, this functionality is getting a significant upgrade:
</p><ul>
	<li>Add support for Windows process monitoring (it used to work only on Linux).</li>
	<li>Report the full command-line for all processes (no longer needed to explicitly configure them)</li>
	<li>Disable the monitor when using file input.</li>
	<li>Get rid of polling for processes and sockets.</li>
	<li>Add UDP support.</li>
	<li>Add the process monitoring capabilities to the flows feature.</li>
</ul><p>If process monitoring is <a href="/guide/en/beats/packetbeat/6.4/configuration-processes.html">enabled</a>, the following fields are added to each flow:
</p><ul>
	<li><code>proc</code>: Process name for the server-side of the flow.</li>
	<li><code>cmdline</code>: Command-line for the local process on the server-side of the flow.</li>
	<li><code>client_proc</code>: Process name for the client-side of the flow.</li>
	<li><code>client_cmdline</code>: Command-line for the local process on the client-side of the flow.</li>
</ul><h2>Feedback
</h2><p>If you want to make use of the new features added in Beats 6.4.0, please <a href="/downloads/beats">download it</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p><p>Enjoy!
</p></body></html>

Beats 6.4.0 Released

We're pleased to release Beats 6.3.0 featuring configuration via Kubernetes and Docker annotations, Filebeat Syslog input, and a beta for spooling to disk.

<html><head></head><body><p>We&apos;re pleased to release Beats 6.3.0. Enjoy!
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats">Download link</a></li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/6.3/release-notes-6.3.0.html">Release notes</a></li>
</ul><p>Here are the highlights:
</p><h3>Configure Beats via Kubernetes/Docker annotations<br></h3><p>You probably got used to the fact that each 6.x minor release is making Kubernetes monitoring even nicer, so we couldn&#x2019;t disappoint you with this one. The goodie that we have prepared in 6.3 is that you can now configure Beats not only via the Beats manifest files but also via Kubernetes annotations (or Docker labels) that are assigned to the monitored Pods.
</p><p>To use this, enable Autodiscover with Hints enabled in the Beats configuration:
</p><pre class="prettyprint">filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
</pre><p>Then, when creating a new deployment, you can use Kubernetes annotations to &#x201C;hint&#x201D; to the Beats how the new deployment should be monitored. For example, if you deploy an Nginx container, you can add the following Kubernetes annotations to it:
</p><pre class="prettyprint"> annotations:
   co.elastic.logs/module: nginx
   co.elastic.logs/fileset: access
   co.elastic.metrics/module: nginx
   co.elastic.metrics/period: 10s
   co.elastic.metrics/hosts: &quot;${data.host}:8080&quot;
</pre><p>When Filebeat and Metricbeat receive the notification that a new Pod is created, they retrieve the annotations above and start the respective Nginx modules with the given configuration. Any Beats settings can be modified this way, and it&#x2019;s a great way of keeping the logging and monitoring configuration near the application deployment configuration.
</p><h3>Filebeat Syslog input</h3><p>Starting with 6.3, you can send logs to Filebeat using the syslog protocol, over UDP or TCP. The internal Syslog parser is a state machine built with Ragel, which makes it both efficient and flexible enough to deal with the usual variations from the Syslog standard. For the moment, RFC  3164 is supported. RFC 5424 is planned for a future release.
</p><p>Here is a sample configuration:
</p><pre class="prettyprint">filebeat.inputs:
- type: syslog
    protocol.tcp:
      # The host and port to receive the new event
      host: &quot;localhost:9000&quot;
      # Character used to split new message
      #line_delimiter: &quot;\n&quot;
      # Maximum size in bytes of the message received over TCP
      #max_message_size: 20MiB
      # The number of seconds of inactivity before a remote connection is closed.
      #timeout: 300s
</pre><h3>Spooling to disk (Beta)<br></h3><p>Starting with this release, all Beats get an optional disk queue where they can spool their events. Any event to be published will be written to the spool first and only forwarded to Elasticsearch or Logstash afterwards. With spooling to disk, events not yet published are still available after Beats have been restarted or in case outputs have been stalled for a long time due to network issues.
</p><p>The spooling to disk feature can be used to provide &#x201C;at least once&#x201D; guarantees for Beats like Metricbeat and Auditbeat, that don&#x2019;t read from a natural queue (like Filebeat does when tailing files).
</p><p>This feature is currently in Beta, so please avoid it in production, but do try it on non-production systems and give us feedback on it.
</p><h3>Add host metadata</h3><p>Everyone loves Beats processors like <code>add_cloud_metadata</code> or <code>add_kubernetes_metadata</code>. Add a line in your Filebeat or Metricbeat configs, and suddenly all events are populated with things like the instance type, region, kubernetes namespace, and so on. So we&#x2019;ve asked ourselves why don&#x2019;t we have something similar for non-cloud non-kubernetes hosts.
</p><p>The new <code>add_host_metadata</code> processor adds the following fields to your event:
</p><ul>
	<li>host.name</li>
	<li>host.id</li>
	<li>host.architecture</li>
	<li>host.os.platform</li>
	<li>host.os.version</li>
	<li>host.os.family</li>
</ul><p>These fields can be used in any filtering and aggregation in Kibana.
</p><p>To support the new fields, we had to add a new object to the index template. This change may cause mapping conflicts for some Logstash users, even if you aren&#x2019;t using the new processor. For more information, see the Beats <a href="/guide/en/beats/libbeat/6.3/breaking-changes-6.3.html#breaking-changes-mapping-conflict]">breaking changes</a> documentation.&#xA0;</p><h3>Wait, there&#x2019;s more: IIS, Logstash, MongoDB, Etcd modules</h3><p>The 6.3 release increases our collection of Filebeat and Metricbeat modules. Filebeat comes with the following new modules:
</p><ul>
	<li dir="ltr">MongoDB</li>
	<li dir="ltr">Logstash</li>
	<li dir="ltr">IIS</li>
</ul><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta48cd409c1f8576c/5c30580dca38a16366a38042/staging.demo.elastic.co_app_kibana_(2).png" data-sys-asset-uid="blta48cd409c1f8576c" alt="staging.demo.elastic.co_app_kibana (2).png">
</p><span id="docs-internal-guid-5d0a0182-d2eb-0a25-f1b1-2fe7a1f3a26a"><p>Metricbeat comes with the following new modules:
</p><ul>
	<li dir="ltr">Etcd</li>
	<li dir="ltr">Graphite</li>
	<li dir="ltr">Logstash</li>
	<li dir="ltr">Munin</li>
</ul><h2>Feedback</h2><p>If you want to make use of the new features added in Beats 6.3.0, please <a href="https://www.elastic.co/downloads/beats">download it</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p><p><br>
</p></span></body></html>

Beats 6.3.0 Released

<html><head></head><body><p>We&#x2019;re pleased to announce the Beats 6.2.0 release, coming with plenty of new features including central monitoring, a passwords keystore, Auditbeat GA, Kubernetes autodiscovery, and a Filebeat module for Osquery.
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats">Download</a></li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/6.2/release-notes-6.2.0.html">Release notes</a></li>
</ul><h2>Beats central monitoring (X-Pack Basic)</h2><p>The Beats 6.2 release brings Beats central monitoring, as part of X-Pack Basic (Free). The way it works is that Beats are sending their monitoring data like CPU usage, number of published events, number of dropped events to the Elastic cluster, and then you can use the dedicated UI (X-Pack Basic) to monitor the Beats instances.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte0b4bcfc20f5e132/5c5844b952256ff20b4afdb6/beats-overview.png" data-sys-asset-uid="blte0b4bcfc20f5e132" alt="beats-overview.png">
</p><p>Below is a list of all your Beats instances, together with information like the type of the Beat, the output enabled, version, etc. The list offers filtering, so you can filter for a specific Beat or a specific Beat type.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7c43ff3a96096b72/5c5844b2a4f817ef0b0e83c2/Beats-list-monitoring.png" data-sys-asset-uid="blt7c43ff3a96096b72" alt="Beats-list-monitoring.png">
</p><p>If you click on one of the Beats, you are redirected to a different page that shows more information about the evolution of the published events, memory usage, system load, bytes sent to easily identify any issues with the Beat.
</p><h2><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta4c44d178c83d7f6/5c5844ad8d25f6030ca04665/Beats-details-monitoring.png" data-sys-asset-uid="blta4c44d178c83d7f6" alt="Beats-details-monitoring.png"><br></h2><p>By default, the central monitoring is disabled. To enable it, install the Xpack Elasticsearch and Kibana plugins and enable Xpack monitoring in the configuration file of the Beat.
</p><pre class="prettyprint">xpack.monitoring.enabled: true
</pre><p>If Elasticsearch is configured as the output, then the monitoring data is sent to the same Elasticsearch cluster configured in the output. In case a different output is configured, then you need to configure the monitoring Elasticsearch cluster under the <code>xpack.monitoring.elasticsearch</code> section.
</p><h2>Secrets keystore</h2><p>The Beats configuration files often contain sensitive information. For example, in Metricbeat, you need to define the username and password to connect to the Elasticsearch output or credentials to connect to each individual service that you monitor.
</p><p>Currently, the sensitive information is found in the plaintext YAML configuration file, but &#xA0;starting with the 6.2 release, you can define them into an obfuscated keystore on disk and only reference them from the configuration file. Future enhancements will also allow encrypting the keystore with a password.
</p><p>New CLI commands are provided, to make it easier to create the keystore and add or remove keys to the keystore. Here are some sample commands:
</p><pre class="prettyprint"># create new keystore
metricbeat keystore create
# add a new key to the store.
metricbeat keystore add output.elasticsearch.password
# remove a key from the store
metricbeat keystore remove output.elasticsearch.password
# list the configured keys without the sensitive information
metricbeat keystore list
</pre><p>After you create the keystore, you can reference keys from the keystore in the configuration file by using the same syntax that we use for the environment variables:
</p><pre class="prettyprint">password: &quot;${output.elasticsearch.password}&quot;
</pre><h2>Auditbeat GA</h2><p>We have first released Auditbeat as Beta in the <a href="https://www.elastic.co/blog/beats-6-0-0-released">Beats 6.0</a> release. Since then, we have worked hard at rounding the rough edges, cleaning up the configuration files, squashing bugs, and collecting your feedback. And the feedback has been constantly great, so we are happy to announce the general availability of Auditbeat as part of the Beats 6.2 release.
</p><p>Being GA means that Auditbeat is recommended for production, fully supported by Elastic, and that we will only be doing breaking changes in major releases.
</p><p>If you didn&#x2019;t hear about Auditbeat before, you can think of it as a friendlier version of <a href="https://www.systutorials.com/docs/linux/man/8-auditd/">auditd</a> that is perfectly integrated with the Elastic Stack. It is based on the Linux audit framework, which means it can hook into every system call and capture them on particular conditions. The Linux Audit Framework can send multiple messages for a single auditable event. For example, a rename syscall causes the kernel to send eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). Auditbeat is able to combine all of the data from each of the messages into a single event.
</p><p>You can use Auditbeat to very efficiently detect things like short-lived connections and processes, unauthorised attempts to open files, privilege escalations, and so on. Auditbeat automatically correlates events together, resolves UIDs ito names, and outputs JSON objects directly to Elasticsearch or Logstash.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcfa08d17b0209fff/5bc43e461952de9159147ccf/auditbeat-file-integrity-dashboard.png" data-sys-asset-uid="bltcfa08d17b0209fff" alt="auditbeat-file-integrity-dashboard.png">
</p><h2>Filebeat Osquery module</h2><p><a href="https://github.com/facebook/osquery">Osquery</a> is an OS instrumentation framework that uses SQL as an interface for querying the underlying data. The new osquery module in Filebeat collects and indexes the result logs written by <code>osqueryd</code>. Osquery writes the results in JSON, which makes it fairly easy to ingest to the Elastic stack, but the module simplifies the collection and the visualization aspect.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt4358277c350263ce/5c3074a1aab458d30b314302/osquery.png" data-sys-asset-uid="blt4358277c350263ce" alt="osquery.png">
</p><p>This module comes with dashboards for two of the Osquery packs: <code>it-compliance</code> and <code>ossec-rootkit</code>. It is therefore recommended to enable these packs in the osqueryd configuration.
</p><h2>Feedback</h2><p>If you want to make use of the new features added in Beats 6.2.0, please <a href="https://www.elastic.co/downloads/beats">download the latest stable version</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p></body></html>

Beats 6.2.0 released

<html><head></head><body><p>We&#x2019;re pleased to announce the Beats 6.1.0 release. This is the latest stable version and it comes with lots of new modules and an exciting Autodiscovery feature.
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats">Download</a></li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/6.1/release-notes.html">Release notes</a></li>
</ul><h2>Docker Autodiscovery</h2><p>Beats 6.1 brings in the first phase of autodiscovery support. Autodiscovery allows the user to configure providers, that watch for system changes and emit events to a common bus. Based on these events, the Autodiscovery system detects situations when there is something new that we can monitor and instantiates new Beats modules for it. In general, Autodiscovery allows the Beats to react and adapt to changes in the ever more dynamic infrastructures.
</p><p>The first provider watches for Docker events. It supports config mapping from container metadata to config templates, so new modules are created when a container starts.
</p><pre class="prettyprint">metricbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals.docker.container.image: redis
          config:
            - module: redis
              metricsets: [&quot;info&quot;, &quot;keyspace&quot;]
              hosts: &quot;${data.host}:${data.port}&quot;
</pre><p>The above is an example configuration that instantiates the Metricbeat Redis module every time a new redis container (defined by having the redis image) is started. Note that the connection information (host/ports) is filled in by the autodiscovery support via a template.
</p><p><span></span>
</p><p>Future releases will add more Autodiscovery providers, for example for Kubernetes events and package managers.
</p><p><span></span>
</p><h2>New Metricbeat and Filebeat modules</h2><p>Each Beats release adds a few new Metricbeat and Filebeat modules, but 6.1 really sets the bar higher. Many of these modules are contributed by our users (Thank You!). Let&#x2019;s go through the list:
</p><ul>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-module-graphite.html">Graphite module</a> in Metricbeat, contributed by <a href="https://github.com/vjsamuel">@vjsamuel</a>. The Graphite protocol is wonderful in its simplicity and very commonly used by monitoring systems. This module makes easy to use Metricbeat as a gateway for Graphite data into the Elastic stack.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-metricset-http-server.html">HTTP server metricset</a>, by the same <a href="https://github.com/vjsamuel">@vjsamuel</a>. This allows Metricbeat to open an HTTP server and accept arbitrary data in the JSON format. You can use this, for example, for creating custom metrics collection in your programming language of choice, and then push it to the local Metricbeat via this metricset.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-module-etcd.html">Etcd module </a>in Metricbeat, contributed by <a href="https://github.com/berfinsari">@berfinsari</a>. Etcd is the distributed key-value where Kubernetes stores all of its data, so this fits very well with all the Kubernetes improvements that we added in 6.0.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-module-logstash.html">Logstash module in Metricbeat</a> as well as <a href="https://www.elastic.co/guide/en/beats/filebeat/6.1/filebeat-module-logstash.html">in Filebeat.</a> What about monitoring the Elastic Stack with Beats? Besides Elasticsearch and Kibana, Beats 6.1 is able to monitor Logstash as well, by collecting its metrics and logs.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/filebeat/6.1/filebeat-module-postgresql.html">Postgres module</a> in Filebeat. This module collects and parses the error and the slow-logs generated by PostgreSQL.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/filebeat/6.1/filebeat-module-kafka.html">Kafka module</a> in Filebeat. This module collects and parses logs and stacktraces from Kafka.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-metricset-system-uptime.html">System uptime metricset</a> in Metricbeat. A simple but useful one, reporting the host uptime in milliseconds. It&#x2019;s available on all our supported operating systems.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-metricset-windows-service.html">Windows service metricset</a> in Metricbeat, contributed by <a href="https://github.com/martinscholz83">@martinscholz83</a>. This metricset collects information about which services are running and data about each of them.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-metricset-ceph-osd_tree.html">OSD tree</a> metricset for the Ceph module in Metricbeat, contributed by <a href="https://github.com/elaron">@elaron</a>. The Metricbeat support for &#xA0;Ceph monitoring gets even better with this new metricset.</li>
	<li><a href="https://www.elastic.co/guide/en/beats/metricbeat/6.1/metricbeat-metricset-rabbitmq-queue.html">RabbitMQ queue metricset</a> in Metricbeat, contributed by <a href="https://github.com/kvalev">@kvalev</a>. This metricset provides basic RabbitMQ queue information.</li>
</ul><h2>TLS support in Packetbeat</h2><p>Packetbeat 6.1 adds <a href="https://www.elastic.co/guide/en/beats/packetbeat/6.1/exported-fields-tls.html">support for the TLS protocol</a>, which is one of the most anticipated Packetbeat features.
</p><p>It doesn&#x2019;t mean decrypting traffic, but it parses the initial handshake and extracts data like ciphers supported by the client and the server, the client and server certificate chains, the subject alternative name (SAN), validity dates, raw certificates, and so on. This data is super valuable for debugging TLS issues and also for intrusion detection and auditing.
</p><p>The implementation also comes with support for the <a href="https://en.wikipedia.org/wiki/Application-Layer_Protocol_Negotiation">ALPN</a> extension to TLS, which allows Packetbeat to detect, for example, whether HTTP/2 or HTTP/1 are used as an application protocol on top of the TLS connection.
</p><p><img src="https://api.contentstack.io/v2/uploads/5a2e9bb20ae920bd28b15c0b/download?uid=blt55f1abc557bc4f11" data-sys-asset-uid="blt55f1abc557bc4f11" alt="tls-dashboard.png">
</p><h2>Docker JSON-file prospector in Filebeat</h2><p>Filebeat 6.1 comes with an (experimental) Docker prospector that implements the default <a href="https://docs.docker.com/engine/admin/logging/json-file/">JSON logging driver</a>. Filebeat could already read Docker logs via the log prospector with JSON decoding enabled, but this new prospector makes things easier for the user. It abstracts the format, so there is no need to manually configure JSON decoding. Here is an example config, which captures the logs from a single container specified by its ID:
</p><pre class="prettyprint">prospectors:
   - type: docker
     containers.ids:
       - c3ec7a0bd9640151a768663b7e78c115d5b1a7f87fba572666bacd8065893d41
</pre><p>It also parses the timestamp from the JSON file, something that wasn&#x2019;t previously possible with Filebeat alone (it required Logstash or Ingest Node). This new prospector works great with the Docker Autodiscovery provider.
</p><h2>New Auditbeat dashboards </h2><p>Auditbeat 6.1 comes with several <a href="https://github.com/elastic/beats/blob/deddfc47584a9b28207ed108c058ed6fb9540e2b/auditbeat/auditbeat.yml#L15">use cases</a> in the default configuration file, which makes it easier to get started with. To match the use cases, we also have three new dashboards:
</p><ul>
	<li>Overview - A general dashboard showing a summary of all events.</li>
	<li>Executions - A dashboard showing process executions (execve and execveat) syscalls.</li>
	<li>Sockets - A dashboard showing information about related sockets and remote connectivity (e.g. bind, connect, accept, recvfrom).</li>
</ul><p><img src="https://api.contentstack.io/v2/uploads/5a2e9c2273340ef831ac62a2/download?uid=blt54e84667df359d1a" data-sys-asset-uid="blt54e84667df359d1a" alt="auditbeat-kernel-sockets-dashboard (1).png">
</p><h2>Feedback</h2><p>If you want to make use of the new features added in Beats 6.1.0, please <a href="https://www.elastic.co/downloads/beats">download the latest stable version</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p><p><span></span>
</p><p><span></span>
</p></body></html>

Beats 6.1.0 released

<p>Today is a big day for Elastic and the entire community: the Elastic Stack 6.0 is now generally available (GA). In this blog post we’ll highlight the main new features of Beats 6.0.
</p><p>If you’ve been following the Alpha and Beta blog posts, you know already what’s in, so we won’t stay in your way. Go to the Beats product page to <a href="https://www.elastic.co/downloads/beats">Download Beats Now</a>.
</p><p>Before upgrading from Beats 5.x, review the <a href="https://www.elastic.co/guide/en/beats/libbeat/6.0/breaking-changes-6.0.html">Breaking Changes</a> docs and the <a href="https://www.elastic.co/guide/en/beats/libbeat/6.0/upgrading-5-to-6.html">Upgrade</a> guide. If you are planning to upgrade the whole Elastic stack at once, we also have a <a href="https://www.elastic.co/guide/en/elastic-stack/6.0/upgrading-elastic-stack.html">guide</a> for that.
</p><h2>Logs and metrics out of Kubernetes and Docker</h2><p>Filebeat and Metricbeat have gained several processors and modules that make container observability with the Elastic stack a breeze.
</p><p>You can use the <code>add_docker_metadata</code> and <code>add_kubernetes_metadata</code> processors to enhance application logs and metrics with Docker and Kubernetes metadata. These processors query the Docker and Kubernetes APIs and enhance the events with the container name, image, pod name, labels, and so on. Depending on the Beat, these processors can use different logic to obtain the metadata. For example Filebeat takes the path of the log file, extracts the container ID from it, and uses the ID to retrieve metadata about the container/pod from which the log message originated. See this blog post about <a href="https://www.elastic.co/blog/enrich-docker-logs-with-filebeat">how to use Filebeat to collect Docker logs</a>.
</p><p>Metricbeat gets a new Kubernetes module, which works by interrogating periodically the Kubernetes API. It gives you details about the running container’s pods, including the CPU usage, memory usage, bytes exchanged over the network, and info about the file system. The sample Kibana dashboard provided with the module shows you at a glance the monitoring status of your Kubernetes cluster.
</p><p><img src="https://api.contentstack.io/v2/uploads/59119c94b65131b00781e1ae/download?uid=blt92344bcc82605436" data-sys-asset-uid="blt92344bcc82605436" alt="kubernetes dashboard kibana" style="max-width:100%;"><br>
</p><p>We also provide Kubernetes deployment manifests for Metricbeat and Filebeat. You can find more details about deploying Filebeat and Metricbeat on Kubernetes in the docs, but as a sneak peak, it’s as easy as:
</p><pre class="prettyprint">curl -L -O https://raw.githubusercontent.com/elastic/beats/6.0/deploy/kubernetes/filebeat-kubernetes.yaml
# edit the YAML file to set the Elasticsearch connection information
kubectl create -f filebeat-kubernetes.yaml
</pre><p>The above commands install Filebeat as a DaemonSet, ensuring one agent is running on each Kubernetes node, and configure Filebeat to pick up the logs from <code>/var/lib/docker/containers/</code>, unwrap them from the JSON objects, and automatically enhance them with Kubernetes metadata.
</p><h2>Auditbeat - easy operational security</h2><p>You can think of Auditbeat as a friendlier version of <a href="https://www.systutorials.com/docs/linux/man/8-auditd/">auditd</a> that is perfectly integrated with the Elastic Stack. It is based on the Linux audit framework, which means it can hook into every system call and capture them on particular conditions. You can use Auditbeat to very efficiently detect things like short-lived connections and processes, unauthorised attempts to open files, privilege escalations, and so on. Auditbeat automatically correlates events together, resolves UIDs ito names, and outputs JSON objects directly to Elasticsearch or Logstash.
</p><p><img src="https://api.contentstack.io/v2/uploads/597efc4abe23be06494fedf0/download?uid=blte8fac57bba3f4f15" data-sys-asset-uid="blte8fac57bba3f4f15" alt="auditbeat file integrity dashboard" style="max-width:100%;"><br>
</p><p>Auditbeat also has a file integrity module. It watches files and directories for changes, and when a file changes it computes the MD5, SHA1, and SHA256 hashes and publishes them to Elasticsearch. The hashes can be compared against known malicious files, as shown in this <a href="https://www.elastic.co/blog/malware-analysis-wannacry-elastic-stack">blog post about detecting WannaCry</a>. This functionality is available on Windows, macOS, and Linux.
</p><h2>New commands and configuration layout</h2><p>We rethought the way the Metricbeat and Filebeat modules are enabled and configured. Instead of one huge configuration file, we now have a <code>modules.d</code> directory with individual configuration files for each module. The Beats also get commands to list, enable, or disable modules. For example:
</p><p><img src="https://api.contentstack.io/v2/uploads/5a043745a8d13dfb2bd63a39/download?uid=blt896c82fed74cb436" data-sys-asset-uid="blt896c82fed74cb436" alt="metricbeat demo gif" style="max-width:100%;">
</p><pre class="prettyprint">$ metricbeat modules list
$ metricbeat modules enable redis
$ metricbeat modules disable redis
</pre><p>And these are not the only useful new commands. There are also commands to export the configuration, export the Elasticsearch mapping template, do a test fetch for a Metricbeat module, and even test the connectivity with Logstash or Elasticsearch.
</p><h2>More efficient Metricbeat storage</h2><p>We took a good look at the data that Metricbeat generates and how it is stored into Elasticsearch, and worked our way towards using considerably less storage while still providing the same, or almost the same, value. We have changed the number of default shards to 1 for the Metricbeat indices, because the amount of data typically doesn’t require more. We have added a new feature that captures only the first N processes by memory and CPU time, instead of all processes, which significantly reduces the number of documents created. And we have reduced the polling frequency for the more static metricsets, like the filesystem one.
</p><p>These changes are complemented nicely by the storage efficiency improvements that went into Elasticsearch 6.0, especially the <a href="https://www.elastic.co/blog/minimize-index-storage-size-elasticsearch-6-0?blade=tw&hulk=social">better support for sparse fields</a>. Together, the improvements add up to 85% less storage used by Metricbeat 6.0 in default configuration compared to Metricbeat 5.5.
</p><h2>Internal pipeline refactoring and better performance</h2><p>Beats 6.0 also comes with a refreshed internal pipeline architecture. While this change is mostly internal, meant to simplify and improve the Beats overall architecture, it does have some visible effects.
</p><p>The new pipeline is asynchronous by default, meaning, for example, that while Filebeat is waiting for a network acknowledgement from Logstash/Elasticsearch, it continues to read and process lines from disk. This brings an increase in the maximum throughput.
</p><p>The Filebeat internal spooler is removed, as its functionality is covered by the new pipeline. This means that tuning Filebeat for performance is easier (only one internal queue size to play with). Be aware that this also implies some breaking changes in the configuration file.
</p><p>Another effect of the pipeline refactoring is that you can no longer enable two outputs at the same time. In previous versions of the Beats, it was possible to enable simultaneously multiple outputs of different types (e.g. one Logstash and on Elasticsearch), but not multiple outputs of the same type. In a back-pressure sensitive shipper, like Filebeat, having multiple outputs generally means that the slowest output decides the rhythm. This is not ideal and often took our users by surprise. We therefore removed the possibility, which also simplified our internal architecture, and instead we recommend using multiple Filebeat instances or using Logstash as an intermediary with multiple outputs.
</p><h2>Tell me more</h2><p>Besides the highlights above, Beats 6.0 come with a ton of other small features, improvements, and modules. Please read the <a href="https://www.elastic.co/guide/en/beats/libbeat/6.0/release-notes-6.0.0.html">full release notes</a> for details.
</p><h2>Thank you and community credits</h2><p>The 6.0 release was a big undertaking, and it wouldn’t have been possible without the continuous support from our community. We’d like to thank everyone who has contributed code or documentation, but also everyone who has tried out the alpha and beta releases.
</p><p>We’d like to nominate the following community members in particular:
</p><ul>
	<li>Vijay Samuel (<a href="https://github.com/vjsamuel">@vjsamuel</a>), from eBay, for his constant high quality contributions to Beats over the last two years, especially around the Kubernetes support. You can read more about how eBay is using Beats to monitor their Kubernetes app on their <a href="http://www.ebaytechblog.com/2017/10/04/beats-ebay-collectbeat-a-journey-where-company-and-community-come-together/">blog</a>.</li>
	<li>Amanda H. L. de Andrade (<a href="https://github.com/amandahla">@amandahla</a>) for the RabbitMQ, Vsphere, and CEPH modules</li>
	<li>Martin Scholz (<a href="https://github.com/martinscholz83">@martinscholz83</a>) for the Windows module in Metricbeat and other improvements</li>
</ul><p>On behalf of the whole community: Thank you and enjoy 6.0!
</p>

Beats 6.0.0 GA released

<p>Today we are pleased to announce the &nbsp;Beats 5.6 release, the latest stable release of the 5.x series.
</p><p>Before going through the highlights, let’s get those handy links out of the way:
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats">Download 5.6.0</a></li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/5.6/release-notes-5.6.0.html">Beats 5.6.0 release notes</a></li>
</ul><h2 rel="line-height:1.38;margin-top:18pt;margin-bottom:6pt;">Load Machine Learning jobs from Filebeat modules</h2><p>Filebeat 5.6 release comes with the support to load Machine Learning (ML) job configurations directly from the Filebeat modules. This means that besides Kibana dashboards and Elasticsearch Ingest Node pipelines, Filebeat modules can also contain ML configurations to be used for anomaly detection.
</p><p style="margin-left: 20px;"><span style="color: rgb(38, 38, 38);"><span style="font-size: 14px;"></span>Survey: With this feature, we are further enriching the modules experience for our&nbsp;community. Which Filebeat & Metricbeat&nbsp;modules should we target next to include&nbsp;preconfigured machine learning jobs? &nbsp;<a href="http://go.es.io/2gEBoLN">Let us know by filling this short survey</a>. It won't take more than&nbsp;</span><span style="color: rgb(38, 38, 38);">5 minutes, we promise.&nbsp;</span></p><p><span style="color: rgb(89, 89, 89);"></span>As part of this release, Filebeat comes with five ML job configurations which were already tuned to work perfectly with the data collected by the Nginx module. If you have Xpack installed and ML enabled, simply run:</p><pre class="prettyprint">filebeat -e -modules=nginx -setup
</pre><p>And you will see the following job configurations created for you:
</p><ul>
	<li>Detect unusual visitor rate</li>
	<li>Detect unusual response code rates</li>
	<li>Detect low request rate</li>
	<li>Detect unusual IP address - high distinct count of urls</li>
	<li>Detect unusual IP addresses - high request rates</li>
</ul><p>After you let Filebeat&nbsp;ingest some data, simply click the Run button on the ML jobs, and <strong>BOOM</strong>: anomaly detection for your access logs.
</p><p><img src="https://api.contentstack.io/v2/uploads/59aff216765ca3280d7cb522/download?uid=blt3073ee2cb2d0bd8a" data-sys-asset-uid="blt3073ee2cb2d0bd8a" alt="Screen Shot 2017-09-06 at 12.59.29.png">
</p><p>And that’s not all! For any incident detected by ML, you can drill down in dedicated Kibana dashboards which are ready made for you to investigate these incidents.
</p><p><img src="https://api.contentstack.io/v2/uploads/59aff22c259271800dd88e3e/download?uid=bltacff2bc2cf4be7de" data-sys-asset-uid="bltacff2bc2cf4be7de" alt="27842771-3097b33c-610d-11e7-984c-4e1d65cac80b (1).png">
</p><h2>Preparing for a smooth transition to 6.0</h2><p>If you are on a 5.x version of Beats, you will need to upgrade to 5.6 before making the jump to 6.0. This is because we’ve made some changes to 5.6 to smoothen the upgrade experience.
</p><p>Notably, the <code>_all.norms</code> setting is no longer disabled in the Beats 5.6 mapping templates. This is because Elasticsearch is removing support for the <code>_all</code> field in 6.0, and having settings for it in the template is problematic at upgrade time.
</p><p>Kibana 6.0 will use a new internal format and a new API for importing and exporting dashboards. To make sure Beats 5.6 will work well with Kibana 6.0, we added support to import the dashboards via the new API in this release already. For this, you can run the `import_dashboards` script with the <code>-kibana</code> flag:
</p><pre class="prettyprint">./scripts/import_dashboards -kibana http://localhost:5601
</pre><p>If Kibana 5.x is used, the dashboards are imported directly into Elasticsearch.
</p><pre class="prettyprint">./scripts/import_dashboards
</pre><h2>Feedback</h2><p>If you want to make use of the new features added in Beats 5.6.0, please <a href="https://www.elastic.co/downloads/beats">download the latest stable version</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p>

Beats 5.6.0 released

<html><head></head><body><p>Today the Elastic stack 6.0 is leaving the Alpha phase behind and boldly enters the Beta. The Beta1 brings tons of improvements to your favorite Beat, be it Filebeat, Heartbeat, Metricbeat, Packetbeat, or Winlogbeat. Not only that, but we also have a surprise: the first public release of Auditbeat.
</p><p>Before going through the highlights, let&#x2019;s get those handy links out of the way:
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats">Download 6.0.0-beta1</a></li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/6.0/release-notes-6.0.0-beta1.html">Beats 6.0.0-beta1 release notes</a></li>
	<li><a href="https://www.elastic.co/blog/beats-6-0-0-alpha2-released">Beats 6.0.0-alpha2 blog post</a></li>
	<li><a href="https://www.elastic.co/blog/beats-6-0-0-alpha1-released">Beats 6.0.0-alpha1 blog post</a></li>
</ul><h2>Say Heya to Auditbeat</h2><p>If you read the <a href="https://www.elastic.co/blog/beats-6-0-0-alpha2-released">Alpha2 release blog post</a>, you know that we have been working on harnessing the auditing framework of the Linux kernel. The audit framework unlocks tons of valuable data, including detection of short lived processes, network connections, unauthorized file accesses, and so on. Check out the <a href="https://www.elastic.co/elasticon/conf/2017/sf/security-at-slack">Security @ Slack</a> talk from Elastic{ON} for an inspiring presentation on how the audit data and the Elastic stack are used against attackers.
</p><p>In Beta1, we&#x2019;re doubling down on this approach by adding a new Beat to our collection: Auditbeat. Auditbeat works as an alternative to the auditd daemon, but one that is designed to integrate perfectly with the Elastic stack. If you have tried to write parsing rules for the auditd logs, you know it&#x2019;s not exactly straightforward. Auditbeat not only does that for you, but it also automatically groups related events together in one document, so that everything is ready to visualize in Kibana. Because it&#x2019;s built on top of libbeat, our mature framework for building Beats, it can send the events via secure and back-pressure sensitive connections to Logstash, Elasticsearch, and Kafka.
</p><p><img src="https://api.contentstack.io/v2/uploads/597efc4abe23be06494fedf0/download?uid=blte8fac57bba3f4f15" data-sys-asset-uid="blte8fac57bba3f4f15" alt="auditbeat-file-integrity-dashboard.png" style="max-width:100%;"><br>
</p><p>In addition to collecting audit logs from the Linux kernel, Auditbeat also has a file integrity module. It watches files and directories for changes, and when a file changes it computes the MD5, SHA1, and SHA256 hashes and publishes them to Elasticsearch. The hashes can be compared against known malicious files, like it is done in this <a href="https://www.elastic.co/blog/malware-analysis-wannacry-elastic-stack">blog post about detecting WannaCry</a>. This functionality is available on Windows, macOS, and Linux.
</p><h2>New commands and configuration layout</h2><p>We rethought the way the Metricbeat and Filebeat modules are enabled and configured. Instead of one huge configuration file, we now have a <code>modules.d</code> directory with individual configuration files for each module. The Beats also get commands to list, enable or disable modules. For example:
</p><pre class="prettyprint">$ metricbeat modules list
$ metricbeat modules enable redis
$ metricbeat modules disable redis
</pre><p>And these are not the only useful new commands. There are also commands to export the configuration, the Elasticsearch mapping template, to do a test fetch for a Metricbeat module, and even to test the connectivity with Logstash or Elasticsearch. Here is an example run for the <code>test output</code> command:
</p><pre class="prettyprint">$ filebeat test output
elasticsearch: https://61fc3e1983ebd7b13e9d6098e4651345.us-east-1.aws.found.io:9243...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 54.235.139.222, 54.221.244.80, 54.243.122.128, 54.204.28.14, 54.221.223.25, 54.243.118.44, 54.243.142.98, 54.235.122.205
    dial up... OK
  TLS...
    security: server&apos;s certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 5.5.0
</pre><h2>Add Docker metadata to the Docker logs</h2><p>The <code>add_docker_metadata</code> processor, which we introduce in alpha2, got a little smarter. A little smarter, and a lot more useful. You might know that when using the default logging driver in Docker, the log files are placed under a path like <code>var/lib/docker/containers/&lt;container_id&gt;/*.log</code>. The <code>add_docker_metadata</code> processor can now extract the container ID from the file path and use it to query the Docker API for the metadata.
</p><p>This makes it <a href="https://www.elastic.co/blog/enrich-docker-logs-with-filebeat">really easy to collect and annotate the Docker logs</a>.
</p><h2>Internal pipeline refactoring</h2><p>6.0.0-beta1 also comes with a refreshed internal pipeline architecture. While this change is mostly internal, meant to simplify and improve the Beats overall architecture, it does have some visible effects.
</p><p>The new pipeline is asynchronous by default, meaning, for example, that while Filebeat is waiting for a network acknowledgement from Logstash/Elasticsearch, it continues to read and process lines from disk. We expect this to bring an increase in the maximum throughput.
</p><p>The Filebeat internal spooler is removed, as its functionality is covered by the new pipeline. This means that tuning Filebeat for performance is easier (only one internal queue size to play with). Be aware that this also implies some <a href="https://www.elastic.co/guide/en/beats/libbeat/6.0/release-notes-6.0.0-beta1.html#_breaking_changes">breaking changes</a> in the configuration file.
</p><p>Another effect of the pipeline refactoring is that you can no longer enable two outputs at the same time. In previous versions of the Beats, it was possible to enable simultaneously multiple outputs of different types (e.g. one Logstash and on Elasticsearch), but not multiple outputs of the same type. In a back-pressure sensitive shipper, like Filebeat is, having multiple outputs generally means that the slowest output decides the rhythm. This is not ideal and often took our users by surprise. We therefore removed the possibility, which also simplified our internal architecture, and instead we recommend using Logstash for duplicating the events to multiple sinks.
</p><h2>Become a Pioneer</h2><p>A big &#x201C;Thank You&#x201D; to everyone that has tried the alpha releases and posted issues or provided feedback. We&#x2019;d like to also remind you that if you post a valid, non-duplicate bug report during the alpha/beta period against any of the Elastic stack projects, you are entitled to a special gift package. You can find more details about the Pioneer Program in the <a href="https://www.elastic.co/blog/elastic-pioneer-program-6-0">Pioneer Program announcement post</a>.</p></body></html>

auditbeat-file-integrity-dashboard.png

Beats 6.0.0-beta1 released

Beats 6.0.0-alpha2 has been released and includes new features like the audit module in Metricbeat and a new processor to enhance events with Docker metadata.

<html><head></head><body><p dir="ltr">It&apos;s time for the second Alpha version of the upcoming 6.0 release of the Elastic stack.&#xA0;The Beats 6.0.0-alpha2 release comes with new features, some rethinking of old features, and&#xA0;bug fixes. This blog post covers the highlights.
</p><p dir="ltr">Quick links:
</p><ul>
	<li dir="ltr"><a href="https://www.elastic.co/downloads/beats">Download the 6.0.0-alpha2 release</a></li>
	<li dir="ltr"><a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-6.0.0-alpha2.html">Check out the full release notes</a></li>
</ul><h2 dir="ltr">Auditd alternative in Metricbeat, simpler to use</h2><p dir="ltr">Auditd is the userspace component to the Linux Auditing System. It can audit every syscall in the Linux Kernel, and write audit records to log files. For example, it writes a log line every time a user accesses a file, starts a process, or connects to a socket. It&#x2019;s part of the Linux kernel starting with 2.6.14. The power of auditd is that by inspecting the audit log files, you can track security-relevant events, detect misuse or unauthorized activities.&#xA0;
</p><p dir="ltr">The auditd log format is a bit challenging to parse, but the auditd Filebeat module (available starting with Filebeat 5.4) parses the individual log lines for you.&#xA0;
</p><p dir="ltr">Another issue with Auditd is that can generate a lot of data, so you need to be specific on what you look for in order to make use of it. This is partly because the Linux Audit Framework sends multiple messages for a single auditable event. For example, a rename syscall causes the kernel to sent eight separate messages. Each message describes a different aspect of the activity that is occurring (the syscall itself, file paths, current working directory, process title). To be more valuable, these events need to be correlated together.&#xA0;
</p><p dir="ltr">So, we decided to create our own version of auditd that is easier to use and better integrated with the Elastic stack. For this, we created the audit module in Metricbeat. It establishes a subscription to the kernel to receive the events as they occur. Messages for one event can be interleaved with messages from another event. The audit module buffers the messages in order to combine related messages into a single event even if they arrive interleaved or out of order.&#xA0;
</p><p dir="ltr">The Linux kernel only supports a single subscriber to the audit events so the audit Metricbeat module cannot be used simultaneously with a service like auditd. Auditd should be disabled if the audit module is being used.&#xA0;
</p><p dir="ltr">The audit module is based on <a href="https://github.com/elastic/go-libaudit">go-libaudit</a> library, developed by us. It comes with two sample applications: audit and auparse. The audit application registers to receive audit events from the kernel and outputs the data it receives to stdout. The auparse application parses the log files from the Linux auditd process or the output of the audit application and it combines related log messages that are a part of the same event.&#xA0;
</p><p dir="ltr">To make use of the audit module, you need to install manually the audit rules to the Kernel. The audit rules are where you configure the activities that are audited. These rules are configured as either syscalls or files that should be monitored. For example you can track all <code>connect</code> syscalls or file system writes to <code>/etc/passwd</code>. As a next step, we are planning to have the audit module load automatically the audit rules, to make it easier for the user.
</p><p dir="ltr">The audit module is a beta feature, and is subject to change.
</p><h2 dir="ltr">New processor: add_docker_metadata</h2><p dir="ltr">Inspired by the <code>add_kubernetes_metadata</code> processor, which we added with Alpha1, the <code>add_docker_metadata</code> processor queries the Docker API to enhance the events with the Docker name, image, and labels. To lookup the metadata, the processor currently needs the container ID to be present in one of the event fields. This is the case, for example, with the cgroups events created by the system module in Metricbeat.
</p><pre class="prettyprint">processors:
 - add_docker_metadata:
     match_fields: [&quot;system.process.cgroup.id&quot;]
     host: &quot;unix:///var/run/docker.sock&quot;
</pre><p dir="ltr">TLS configuration for the connection with the Docker server is also possible.
</p><h2 dir="ltr">Optimized Metricbeat default configuration</h2><p dir="ltr">This release introduces a new default <code>metricbeat.yml</code>, one that is better suited for the majority of metrics use cases. The new configuration takes advantage of the <em>top N</em> way of filtering processes, which we introduced in Alpha1, and it also uses the new way of generating the Elasticsearch mapping template to set the number of shards to 1 and to enable the <em>best_compression</em> codec. Finally, the new configuration sets better defaults for the filesystem data collection.
</p><p dir="ltr">We expect this new default configuration to consume dramatically less Elasticsearch storage per day-host, while delivering almost the same value.
</p><h2 dir="ltr">Metricbeat new modules</h2><p dir="ltr">Just like Alpha1, this release comes with a set of new Metricbeat modules and improvements. Notable are the <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-vsphere.html">Vsphere</a> and <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-rabbitmq.html">RabbitMQ</a> modules, both contributed by <a href="https://github.com/amandahla">@amandahla</a>, the MacOS support for the diskio metricset, and several improvements to the <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-kubernetes.html">Kubernetes</a> module.
</p><h2 dir="ltr">Feedback</h2><p>If you want to make use of the new features added in Beats 6.0.0, please <a href="https://www.elastic.co/downloads/beats">download the latest preview release</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p></body></html>

pexels-photo-374563.jpeg

Beats 6.0.0-alpha2 released

Beats 6.0.0-alpha1 has been released and includes new features like kubernetes monitoring, more modules in Metricbeat, or a better upgrade experience.

<html><head></head><body><p dir="ltr">We&#x2019;re starting our road towards the 6.0 release of the Elastic stack, with the first alpha version being released today. In the Beats team, we&#x2019;ve been baking a lot of new features that we&#x2019;re really excited to put out there for the first time. This blog post will cover some of the highlights.
</p><p dir="ltr">Quick links:
</p><ul>
	<li dir="ltr"><a href="https://www.elastic.co/downloads/beats">Download the 6.0.0-alpha1 release</a></li>
	<li dir="ltr"><a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-6.0.0-alpha1.html">Check out the full release notes</a></li>
</ul><h2 dir="ltr">Kubernetes monitoring </h2><p>With this Beats release we&#x2019;re turning the Elastic stack into a perfect monitoring companion for Kubernetes. Both Filebeat and Metricbeat now have integrations with Kubernetes to collect pod logs and metrics, as well as enhance them with Kubernetes specific metadata.
</p><p>Metricbeat gets a <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-kubernetes.html">Kubernetes module</a>, which works by interrogating periodically the kubelets. It gives you details about the running containers pods, like the CPU usage, memory usage, bytes exchanged over the network, about filesystem, or about the logs capacity.
</p><p>The provided sample Kibana dashboard shows you at a glance the monitoring status of your Kubernetes cluster.
</p><p dir="ltr">
<img src="https://api.contentstack.io/v2/uploads/59119c94b65131b00781e1ae/download?uid=blt92344bcc82605436" data-sys-asset-uid="blt92344bcc82605436" alt="kubernetes-dashboard.png">
</p><p dir="ltr">Filebeat, and other Beats as well, is now able to automatically attach Kubernetes specific metadata to the log lines by using a <a href="https://www.elastic.co/guide/en/beats/filebeat/master/kubernetes.html">new processor</a>. By default, the Kubernetes processor adds the following metadata:
</p><ul>
	<li>pod name</li>
	<li>pod namespace</li>
	<li>container name and pod labels</li>
</ul><p>Depending on the Beat, the Kubernetes processor uses slightly different logic to obtain this metadata. For example in Filebeat, it takes the path of the log file, extracts the container ID from it and uses it to retrieve metadata about the pod from which the log message originated. In Metricbeat, it uses the IP and port of the endpoints to resolve which containers are part of which pods.
</p><p>The Kubernetes support in Beats is a wonderful example of open source collaboration. <a href="https://github.com/vjsamuel">Vijay Samuel</a>, from Ebay, worked closely with the Elastic engineers to contribute upstream a lot of the code they had for monitoring their Kubernetes cluster with Beats. Vijay&#x2019;s experience was essential to shape these features, so we&#x2019;d like to thank him and Ebay on behalf of the Elastic community.
</p><h2 dir="ltr">More Metricbeat modules</h2><p>Besides the Kubernetes module which we already mentioned, Metricbeat 6.0.0-alpha1 comes with a bunch of new modules: memcached, perfmon, dropwizard, HTTP, vSphere, Elasticsearch, and Kibana.
</p><p>An interesting one is the <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-module-http.html">HTTP module</a>. We liked the idea of the <a href="https://github.com/christiangalsterer/httpbeat">Httpbeat</a>, a community Beat created by <a href="https://github.com/christiangalsterer">Christian Galsterer</a>. It offers a generic solution for collecting data periodically from multiple HTTP endpoints and indexing the responses in Elasticsearch. &#xA0;Christian contributed it as a module in Metricbeat to be used for monitoring application that export their metrics over HTTP. &#xA0;For example, if the HTTP endpoint returns the result in a JSON format, you can configure the json metricset from this module to index the contents of each JSON object into Elasticsearch.
</p><p><a href="https://github.com/maddin2016/perfmonbeat">Perfmonbeat</a> is another interesting community Beat, created by <a href="https://github.com/maddin2016">Martin Scholz</a>, that we adopted in Metricbeat as part of the Windows module. It comes with the <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-metricset-windows-perfmon.html">perfmon metricset</a> to collect performance counters from Windows. It uses the PDH functions to collect performance data.
</p><h2 dir="ltr">Metricbeat - Report the top N processes</h2><p>One of the most appreciated Metricbeat features is that it reports CPU and memory statistics about the individual processes, which gives you a lot of insight into what is taking up the resources. However, often hundreds of processes are running on each monitored system, and that tends to consume significant amounts of disk space on the Elasticsearch side.
</p><p>From the first days of Metricbeat, it was possible to filter the process statistics by the process names or other fields, which is useful when you know which of the processes you want to monitor in advance. With the 6.0.0-alpha1 release, we&#x2019;re adding the option to only report the top N processes by CPU and/or memory. We found that this option strikes a good balance between the need to limit the amount of consumed disk space and the need to have granular enough metrics, so we&#x2019;ve made it the default in the Metricbeat configuration.
</p><h2 dir="ltr">Dashboards using the Time Series Visual Builder</h2><p>The system module of Metricbeat gets two new Kibana dashboards, built using the new Time Series Visual Builder that was released with Kibana 5.4. It is the perfect match for the Metricbeat data, so we plan to recreate many of the existing Metricbeat dashboards using the new time series capabilities in Kibana.
</p><p>For more details, you can watch a demo of the time series visual builder during the <a href="https://www.elastic.co/elasticon/conf/2017/sf/kibana-visualizations-deep-dive">Kibana visualizations deep dive</a>.
</p><p dir="ltr">
<img src="https://api.contentstack.io/v2/uploads/59119b689b6ac1d10719c559/download?uid=blt171fc471693e4414" data-sys-asset-uid="blt171fc471693e4414" alt="Metricbeat host overview Kibana (1).png">
</p><h2 dir="ltr">Better upgrade experience</h2><p>With this release, we&#x2019;re changing the default format of the Elasticsearch index names to include the Beat version. For example, instead of <code>filebeat-2017.05.09</code> we are now using <code>filebeat-6.0.0-alpha1-2017.05.09</code> which allows us to apply a different mapping template depending on the Beat version. This will make rolling upgrades easier, where you have multiple Beats versions running in parallel. This change doesn&#x2019;t affect the current Kibana dashboards as the <code>beatname-*</code> index pattern matches the new template.
</p><p>Another improvement is that the mapping template is now generated by the Beat on startup. This makes it easier to adjust and optimize the template depending on the Elasticsearch version. It also makes it easier to add custom fields to the template, by editing a human friendly <code>fields.yml</code> file instead of a large JSON document.
</p><h2 dir="ltr">Feedback</h2><p>If you want to make use of the new features added in Beats 6.0.0, please <a href="https://www.elastic.co/downloads/beats">download the latest preview release</a>, install it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/c/beats">forum</a>.
</p></body></html>

kubernetes-dashboard.png

Beats 6.0.0-alpha1 released

screenshot-computer-screen-green-code.jpg

thumb-computer-screen-green-code.jpg

Grokking the Linux authorization logs

With the Brewing in Beats series, we're keeping you up to date with all that's new in Beats, from the details of pull requests to learning resources.

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p><h3>Pass arrays or dicts via -E and env variables</h3><p>It is now possible (again, since this was also possible in versions &lt; 5.0, but we had a regression) to specify an <a href="https://github.com/elastic/beats/pull/3045">array or dictionary via environment variables or the -E CLI flag</a>. This is particularly useful when specifying multiple output hosts. For example, this works as expected now: <code>output.kafka.hosts: ${KAFKA_HOSTS}</code> if <code>KAFKA_HOSTS</code> is an environment variable in the form <code>&#x201C;host1:9092&#x201D;,&#x201C;host2:9092&#x201D;</code>. This feature didn&apos;t make the cut for the 5.1 release, but will be available in a next minor release.</p><h3>Metricbeat Kafka module improvements</h3><p>Preparing for the first release of the Kafka module, we&#x2019;ve got <a href="https://github.com/elastic/beats/pull/3029">several fixes and improvements to it</a>, including TLS support and SASL authentication (available in Kafka 0.10).
</p><h3>Winlogbeat fix for &#x201C;invalid bounds&#x201D; error message</h3><p>When reading a batch of large event log records the Windows function EvtNext returns errno 1734 (&quot;The array bounds are invalid.&quot;). This seems to be a bug in Windows because there is no documentation about this behavior. The <a href="https://github.com/elastic/beats/pull/3112">fix</a> handles the error by resetting the event log subscription handle (so events are not lost) and then retries the EvtNext call with maxHandles/2.
</p><h3>Winlogbeat benchmarks and performance improvements</h3><p>Andrew is working on <a href="https://github.com/elastic/beats/pull/3107">analyzing the performance</a> of Winlogbeat and fixing some performance issues, like the <a href="https://github.com/elastic/beats/pull/3113">number of allocations when converting from UTF16</a>.
</p><h3>Automatic tests with ES/LS 2.x</h3><p>We realized we don&#x2019;t have automatic integration tests with any of the 2x versions of Elasticsearch and Logstash, but list them on our support matrix, so we&#x2019;re <a href="https://github.com/elastic/beats/pull/3057">fixing that</a>.
</p><h3>Filebeat: restate publish_async as experimental</h3><p>The <code>publish_async</code> feature allows Filebeat to continue reading batches of log lines while waiting for confirmation that the current batch was processed by the next stage. This option can improve throughput but also cause larger memory and CPU consumption. We&#x2019;re now <a href="https://github.com/elastic/beats/pull/3096">making it clear</a> in the docs and via a warning that this feature is considered experimental at the moment.</p></body></html>

brewing.jpg

brewing_small.jpg

Brewing in Beats: Metricbeat Kafka module improvements

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.</p><h3>New community Beat: Redditbeat</h3><p>Redditbeat, by <a href="https://github.com/voigt">@voigt</a>, indexes into Elasticsearch new Reddit submissions of one or multiple sub-reddits. Because why not.
</p><h3>Libbeat processor for decoding JSON</h3><p>Filebeat can decode JSON since version 5.0, but it didn&#x2019;t have so far a good way of handling JSON-in-JSON situations. This can happen, for example, if one is using the Docker JSON logging driver and the application also outputs in JSON. Thanks to a community contribution by <a href="https://github.com/suraj-soni">@suraj-soni</a>, libbeat got a new processor that can <a href="https://github.com/elastic/beats/pull/2605">decode JSON from an arbitrary field</a>. This is useful for Filebeat, but can be used also, for example, by Packetbeat to decode the body of HTTP requests.
</p><h3>Metricbeat: raw MySQL fields</h3><p>The general approach we take with Metricbeat is to carefully curate the available metrics and only export the most actionable ones, which we usually put in the sample dashboards. We also rename these metrics to follow a common naming scheme and make sure to select the correct Elasticsearch type and Kibana format for each of them.
</p><p>While this approach gives a good out of the box experience that people usually like, it has the disadvantage that if someone needs a more obscure metric, it&#x2019;s often not readily available. To compensate, we&#x2019;re experimenting with adding the option to automatically capture all metrics with their original names (or changed just enough to be usable in Elasticsearch/Kibana) under a `raw` sub-document. The first module to <a href="https://github.com/elastic/beats/pull/3001">get the `raw` option is MySQL</a>.
</p><h3>Metricbeat: Kafka module merged</h3><p>The <a href="https://github.com/elastic/beats/pull/2969">Kafka module</a> is now merged and will be available in the 5.1 release as experimental.
</p><h3>Metricbeat: MongoDB module improvements</h3><p>The MongoDB module was <a href="https://github.com/elastic/beats/pull/2999">refreshed</a> to extract the key metrics from the relatively new wiredTiger storage engine and to work well with MongoDB 3.4. We now also have a sample dashboard for MongoDB.
</p><h3>Metricbeat: HAProxy module improvements</h3><p>The fields created by the HAProxy module were <a href="https://github.com/elastic/beats/pull/3003">reviewed</a> to match our naming conventions and got better documented.
</p><h3>Metricbeat: Docker module improvements</h3><p>The same reviewing work was done for the <a href="https://github.com/elastic/beats/pull/3015">CPU metricset</a> from the Docker module. Also, the Docker labels are now exported as <a href="https://github.com/elastic/beats/pull/3024">a dictionary rather than an array.</a> This makes them easier to use in Kibana. Additionally, the <a href="https://github.com/elastic/beats/pull/3040">ports data was removed</a> from the container metricset. Special thanks to <a href="https://github.com/rikatz">@rikatz</a> for early testing the Docker module and suggesting improvements on it.
</p><h3>Metricbeat: parse and sanitize the connection URL</h3><p>Metricbeat now offers a safer and standardized method for modules to <a href="https://github.com/elastic/beats/pull/2958">sanitize the connection URLs</a> (the contents of the `hosts` field) so that we have a good way of extracting the sensitive information from them.
</p><h3>Filebeat: faster shutdown when dealing with a large number of files</h3><p>Filebeat is now able to <a href="https://github.com/elastic/beats/pull/3035">interrupt a running scan</a> when receiving the shutdown signal. This makes the shutdown more responsive when the scan has to deal with lots of files.
</p><h3>Metricbeat: fix service timeout on startup</h3><a href="https://github.com/maddin2016">@maddin2016</a> fixed a bug where the Metricbeat startup was taking too long and <a href="https://github.com/elastic/beats/pull/3056">could cause a service timeout</a> when being started under windows.</body></html>

blog-brewing_thumb.jpg

Brewing in Beats: Processor for decoding JSON

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>New community Beat: Consulbeat</h3> <p>This new <a href="https://github.com/Pravoru/consulbeat">Beat</a> collects service health metrics from <a href="https://www.consul.io/">Consul</a> and pushes them to the Elastic stack. </p> <h3>Metricbeat Kafka module (in progress)</h3> <p>Not merged yet, but Nicolas has made good progress on adding support for monitoring Kafka with Metricbeat. The work is partially inspired by <a href="https://github.com/gingerwizard/kafkabeat">Kafkabeat</a>, so hat tip to Dale. </p> <h3>Fix: Remove duplicate fields from the index patterns</h3> <p>We had an issue where several fields, including @timestamp, were <a href="https://github.com/elastic/beats/issues/2941">showing up multiple times in Kibana</a>. This happened because we build up the Elasticsearch mapping template and the Kibana index pattern from several fragments, so some duplicates sneaked up on us. We&#x2019;ve <a href="https://github.com/elastic/beats/pull/2973">added code</a> to automatically check if the same field is accidentally defined in multiple places, so this shouldn&#x2019;t happen again. </p> <h3>Filebeat: tail_files is applied only once at start</h3> <p>A <a href="https://github.com/elastic/beats/issues/2613">common</a> problem with using the tail_files option was that people used it to skip old files but, because it was applied on any new file, it could easily drop the first few lines from each file. We&#x2019;ve now <a href="https://github.com/elastic/beats/pull/2932">changed the behaviour</a> to only&#xA0;apply the tail_files option at the Filebeat start. </p> <h3>Remove intermediary files during generation</h3> <p>We&#x2019;ve got a lot of auto-generated files in the repository, to the point that it is hard for occasional contributors, and sometimes even ourselves, to figure out where things need to be changed. So we&#x2019;re looking to reduce the number of auto-generated files that we commit in git, and this <a href="https://github.com/elastic/beats/pull/2931">PR</a> takes a first step and removes the intermediary fields.yml files used for generating the templates, docs, and index patterns. </p> <h3>Packetbeat code cleanup</h3> <p>Steffen continued his work on applying coding standards to the older parts of our code base, with a massive <a href="https://github.com/elastic/beats/pull/2972">PR</a> against Packetbeat, mostly renaming things and reducing the exported fields. </p> <h3>Godocs for go-ucfg</h3> <p><a href="https://github.com/elastic/go-ucfg"><u>Go-ucfg</u></a> is our fairly advanced (we worry about it becoming self aware soon) library for handling configuration files. For the moment it just got <a href="https://github.com/elastic/go-ucfg/pull/71"><u>better docs</u></a>. </p></body></html>

blog-brewing-banner.jpg

Brewing in Beats: Kafka module in Metricbeat

It’s finally here! Beats 5.0.0 is released.

<html><head></head><body><p>It&#x2019;s finally here! After about 11 months of work, five Alpha versions, one Beta, and one RC, we have a new major version of your favorite data shippers. And it&#x2019;s not only Beats that are releasing today. In fact, starting with this release, all the projects from the Elastic stack <a href="/blog/elastic-stack-5-0-0-released">are releasing at the same time</a> and with the same version numbers down to the patch level. That&#x2019;s why we&#x2019;ve got the nice jump from 1.3.1 to 5.0.0 in version numbers for Beats.
</p><p>If you&#x2019;ve been following the alphas and betas, you know already what&#x2019;s in, so we won&#x2019;t stay in your way if you&#x2019;re just looking for the <a href="https://www.elastic.co/downloads/beats">Download Now</a> link. If not, then this blog post highlights the biggest changes coming with 5.0.
</p><p>If you are upgrading from 1.x, please read the <a href="https://www.elastic.co/guide/en/beats/libbeat/5.0/breaking-changes.html">Breaking Changes</a> docs and the <a href="https://www.elastic.co/guide/en/beats/libbeat/5.0/upgrading.html">Upgrading</a> guide.
</p><h2>Metricbeat - You know, for metrics</h2><p>Metricbeat replaces Topbeat as the primary tool for collecting metrics in the Elastic stack. Like Topbeat, Metricbeat collects &#x201C;top&#x201D; like statistics about host and per process resources (CPU, memory, disk, network). Unlike Topbeat, Metricbeat also collects metrics from systems such as Apache, HAProxy, MongoDB, MySQL, Nginx, PostgreSQL, Redis, or Zookeeper, with more to come in the near future.
</p><p>The system module in Metricbeat offers all the functionality of Topbeat, and it also extends this functionality with new features like per container metrics, disk IO metrics, and network IO metrics. The configurations of the two Beats are not compatible, but <a href="https://www.elastic.co/guide/en/beats/libbeat/current/upgrading-1-to-5.html#_upgrading_topbeat_to_metricbeat">migrating</a> from one to another is fairly simple.
</p><p>And the good news doesn&#x2019;t stop here: Metricbeat takes advantage of the new number types and storage implementation of Elasticsearch 5.0, which make numbers <a href="https://www.elastic.co/blog/searching-numb3rs-in-5.0">faster to index, faster to search, more disk-efficient and more memory-efficient</a>.
</p><h2>Filebeat - under the hood improvements</h2><p>We&#x2019;ve refactored the way Filebeat stores the state of of harvested files (which files are opened, inode numbers for each, how far we&#x2019;ve read in each one, etc.). We&#x2019;ve also cleaned up the settings around when the files should be closed, ignored or forgotten from the state. We expect these changes to improve the robustness of Filebeat against corner cases (for example inode reuse or very quick file rotation), so we recommend that you upgrade.
</p><p>In addition, Filebeat can now natively decode JSON objects from log lines. This is useful for <a href="https://www.elastic.co/blog/structured-logging-filebeat">structured logging</a>, where the logging library writes the metadata directly formatted as JSON.
</p><h2>Packetbeat Flows</h2><p>Packetbeat now reports statistics like packet count and byte count about IP and TCP flows, regardless of the upper layer protocols. This opens Packetbeat to a new set of use cases, giving insights into how the traffic is flowing through the network and offers a limited level of visibility into encrypted traffic.
</p><p>In addition, Packetbeat got support for a few more application protocols, including AMQP, Cassandra, EDNS, DNSSEC, and NFS.
</p><h2>Filter with the power of processors</h2><p>Beats &#x201C;processors&#x201D; are simple rules that manipulate the events just before they are shipped from the Beat. While similar in nature with the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.0/ingest.html">Ingest Node</a> processors or Logstash filters, don&#x2019;t expect the same power. The Beats processors are... well, lightweight, and focus on filtering the data to avoid sending too much of it over the network when it will be discarded anyway by Logstash.
</p><p>That said, processors give you great flexibility in choosing the data that should be shipped. Here is a quick example:
</p><pre class="prettyprint">processors:
- drop_event:
   when:
     equals:
       http.code: 200
- drop_fields:
    fields: [&#x201C;http.phrase&#x201D;]
</pre><h2>Kafka Output</h2><p>We listened to your feedback and we&#x2019;ve added native Kafka output support in Beats, at the same time greatly improving the existing Redis output. This means that if you are passing all messages through a Kafka queue anyway, you no longer need an extra Logstash instance to convert between Beats and Kafka.
</p><p>The Beats Kafka output is compatible with Kafka versions 0.8, 0.9, and 0.10 and supports encryption, authentication, as well as dynamic topic and partition selection.
</p><h2>Configuration flexibility</h2><p>Beats 5.0 uses the same YAML-based configuration format, but improved with many small features that make the deployment of Beats easier and more pleasant. The new features include better namespacing, reference variables, better regular expression support, and the ability to overwrite any setting from the command line with the <code>-E</code> flag.
</p><p>You can read more about the new Beats configuration language in the <a href="https://www.elastic.co/guide/en/beats/libbeat/5.0/config-file-format.html">docs</a>.
</p><h2>Tell me more</h2><p>Do you want to read more on what&#x2019;s new in 5.0? Have a look at the <a href="https://www.elastic.co/guide/en/beats/libbeat/5.0/release-notes-5.0.0.html">release notes</a> and the blog posts for the pre-releases:
</p><ul>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-alpha1-released">Beats 5.0.0-alpha1</a></li>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-alpha2-released">Beats 5.0.0-alpha2</a></li>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-alpha3-released">Beats 5.0.0-alpha3</a></li>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-alpha4-released">Beats 5.0.0-alpha4</a></li>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-alpha5-released">Beats 5.0.0-alpha5</a></li>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-beta1-released">Beats 5.0.0-beta1</a></li>
	<li><a href="https://www.elastic.co/blog/beats-5-0-0-rc1-released">Beats 5.0.0-rc1</a></li>
</ul><h2>Thank you and community credits</h2><p>And now, it&#x2019;s party time!
</p><p><img src="https://api.contentstack.io/v2/assets/58093e9e1eb333c83f32cbf9/download?uid=bltdc2103ec2b30512b" data-sys-asset-uid="bltdc2103ec2b30512b" alt="Beats team 5.0" style="max-width:100%">
</p><p><span></span>
</p><p>First of all, we would like to thank everyone that has tried the 5.0 pre-releases and posted the <a href="https://github.com/elastic/beats/issues?q=is%3Aissue+label%3A%22Pioneer+Program%22+is%3Aclosed">issues they found</a>. Each one of you will receive a small thank you gift from Elastic.
</p><p>We&#x2019;d also like to give special credit to the following community members for contributing significant code to the 5.0 release:
</p><ul>
	<li><a href="https://github.com/kofemann">@kofemann</a> for the NFS support in Packetbeat</li>
	<li><a href="https://github.com/mrkschan">@mrkschan</a> for the Nginx module in Metricbeat</li>
	<li><a href="https://github.com/McStork">@mkstork</a> for the EDNS and DNSSEC support in Packetbeat</li>
	<li><a href="https://github.com/hartfordfive">Al Lefebvre</a> for the HAProxy module in Metricbeat</li>
	<li><a href="https://github.com/raboof">Arnout Engelen</a> for the IPv6 support in Packetbeat&#x2019;s socket to process mapping code</li>
	<li><a href="https://github.com/chrsblck">Chris Black</a> for the Redis module in Metricbeat</li>
	<li><a href="https://github.com/cyrilleverrier">Cyrille Verrier</a> for packaging improvements</li>
	<li><a href="https://github.com/radoondas">Radovan Ondas</a> for the Apache module in Metricbeat</li>
	<li><a href="https://github.com/Keylor42">Thomas Paulmyer</a> for the AMQP support in Packetbeat</li>
</ul>On behalf of the whole community: Thank you and enjoy 5.0</body></html>

Beats 5.0.0 Released

<html><head></head><body><h3>Heartbeat &#x1F493;</h3>
<p>Heartbeat is the newest Beat to <a href="https://github.com/elastic/beats/pull/2743">join the main beats repository</a>. Heartbeat does active uptime monitoring via ICMP, TCP, and HTTP and stores the availability status and the round trip times in Elasticsearch. It already supports things like SSL, authentication, or proxies.
</p>
<p><strong></strong>
</p>
<p>Heartbeat is also able to reload the list of monitored targets and to resolve all the IP addresses behind a DNS name and test them all. This is useful, for example, to check the uptime of all the hosts behind an ELB.
</p>
<p>Heartbeat development is still ongoing. We plan to release a first officially supported versions in the next few months.
</p>
<h3>Libbeat: Add EC2, GCE, or DigitalOcean metadata</h3>
<p>The new Beats processor does a one time query of the respective cloud APIs and adds a meta object in all events created by the Beat, making it easy to filter in Kibana by things like the region or the machine type. See the <a href="https://github.com/elastic/beats/pull/2728">pull request</a> for example objects from each of the supported cloud platforms.
</p>
<h3>Beats-tester: test default configuration</h3>
<p>Now that we can easily overwrite config settings with -E, we can <a href="https://github.com/elastic/beats-tester/pull/45">use the default configuration files in the packaging tests</a>, and only overwrite what we need to be different in tests. This has the advantage that we test the default configuration files in realistic conditions.
</p>
<h3>Fix requested privileges on Windows</h3>
<p>Fixed an issue where the Metricbeat system module was <a href="https://github.com/elastic/gosigar/pull/50">requesting too broad privileges</a>. We expect this to fix <a href="https://github.com/elastic/beats/issues/1897">cases</a> where Metricbeat was not able to read process information.
</p>
<h3>Fix &#x201C;actual free&#x201D; memory on Windows</h3>
<p>In the Metricbeat system module for Windows, <a href="https://github.com/elastic/gosigar/pull/49">fixed &#x201C;actual free&#x201D;</a> to refer available physical memory, not available virtual memory.
</p>
<h3>Fix for the Kafka output</h3>
<p>Fixes an issue where the Kafka output would <a href="https://github.com/elastic/beats/pull/2737">continuously resend the same batch</a> of messages if one of them was too large to be accepted by Kafka.
</p>
<h3>Fix for Filebeat</h3>
<p>Fixed an issue where if If <code>clean_removed</code> and <code>clean_inactive</code> were used together, states <a href="https://github.com/elastic/beats/pull/2750">were not directly removed</a> from the registry.
</p></body></html>

Brewing in Beats: Heartbeat, for uptime monitoring

Today we’re publishing the first public release candidate for the 5.0 version of the stack. Let your excitement grow, the 5.0 day is approaching!

<html><head></head><body><p>Today we&#x2019;re publishing the first public release candidate for the 5.0 version of the stack. Let your excitement grow, the 5.0 day is approaching!<br></p><p><strong></strong>
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats">Download links</a></li>
	<li>Beats 5.0.0-rc1 <a href="https://www.elastic.co/guide/en/beats/libbeat/5.0/release-notes-5.0.0-rc1.html">release notes</a></li>
	<li>Beats 5.0.0-beta1 <a href="https://www.elastic.co/blog/beats-5-0-0-beta1-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-beta1.html">release notes</a></li>
	<li>Beats 5.0.0-alpha5 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha5-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha5.html">release notes</a></li>
	<li>Beats 5.0.0-alpha4 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha4-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha4.html">release notes</a></li>
	<li>Beats 5.0.0-alpha3 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha3-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha3.html">release notes</a></li>
	<li>Beats 5.0.0-alpha2 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha2-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha2.html">release notes</a></li>
	<li>Beats 5.0.0-alpha1 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha1-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha1.html">release notes</a></li>
</ul><p><strong></strong>
</p><p>Here is what changed in RC1:
</p><h3>Make future upgrades easier</h3><p>In order to simplify adding new fields between minor versions, we have done two changes:
</p><p><strong></strong>
</p><ul>
	<li>Our Elasticsearch mapping templates now automatically set unknown strings to the <code>keyword</code> type.</li>
	<li>We make sure we always include a dot when outputting floats to Elasticsearch. If we know that the type of a field is float, but the current value has no decimals (e.g. 42), we will serialize it with zeros after the dot (in this case, 42.0000) before shipping the event to Elasticsearch.</li>
</ul><p><strong></strong>
</p><p>With these changes, Elasticsearch will be able to guess the types of future fields, even if the updated mapping is not loaded yet, so the upgrade procedure will be simpler.
</p><h3>Other fixes</h3><ul>
	<li>The default Metricbeat configuration disables the <code>load</code> metricset on Windows.</li>
	<li>The <code>drop_fields</code> filter now correctly ignores missing fields and removes the ones that exist.</li>
	<li>It&#x2019;s again possible to set a custom BPF filter in Packetbeat.</li>
	<li>Fixed an encoding issue in Filebeat when a different encoding was set then the actual encoding of the file.</li>
	<li>Fixed type conversions when interpreting the configuration file.</li>
</ul></body></html>

beats_full_width.png

beats_blog.png

Beats 5.0.0-rc1 released

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>Filebeat: -once flag to exit after finishing current work</h3> <p>When the <code>-once</code> flag is used, Filebeat starts all configured harvesters and prospectors and runs each prospector until the harvesters are closed. It is recommended to use the flag in combination with <code>close_eof</code> so harvester directly close when the end of the file is reached. This could be used for measuring the Filebeat performance or for loading a fixed set of files. </p> <h3>Kibana dashboards for containerized processes</h3> <p>Since recently, Metricbeat can report cgroup data for any running process (see this <a href="https://www.elastic.co/blog/monitoring_container_resource_usage_with_metricbeat">blog post</a> for details), and now we&#x2019;ve also added <a href="https://github.com/elastic/beats/pull/2555">sample dashboards</a> for this data. </p> <h3>Added /net/ and /ns/ support to procfs library</h3> <p>Andrew <a href="https://github.com/elastic/procfs/pull/1">added new features</a> to the procfs Golang library, which does proc file system parsing and which we plan to use in Metricbeat. </p> <h3>Updated to Go 1.7.1</h3> <p>Our compiler images used in packaging, docker files used in tests, and our CI tools configurations were all <a href="https://github.com/elastic/beats/pull/2533">updated to use Go 1.7.1</a> (from 1.7). </p> <h3>Filebeat: fix concurrent harvesters</h3> <p>In case newly started harvesters did not persist their first state before the next scan started, it could have happened that multiple harvesters were started for the same file. This could have been caused by a large number of files or the output blocking. There are more details in the <a href="https://github.com/elastic/beats/pull/2541">PR</a>. </p> <h3>Release 1.3.1</h3> <p>We&#x2019;ve released the&#xA0;<a href="https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-1.3.1.html">patch version 1.3.1</a>, which includes a&#xA0;<a href="https://github.com/elastic/beats/pull/2509">concurrency fix</a>&#xA0;in Filebeat, discovered and fixed by&#xA0;<a href="https://github.com/amyangfei">@amyangfei</a>, a contributor. </p></body></html>

Brewing in Beats: Dashboards for containerized processes

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>Metricbeat: new HAProxy module</h3> <p>Thanks to a community contribution by <a href="https://github.com/hartfordfive">@hartfordfive</a>, Metricbeat got an <a href="https://github.com/hartfordfive">HAProxy module</a> &#x1F389; </p> <h3>Filebeat: The return of the Symlinks</h3> <p>Filebeat used to somehow unintentionally follow symlinks when opening files. So we deprecated them in 1.3 and were planning to remove the support for symlinks in 5.0. However, we&#x2019;ve got some pretty strong pushback from Kubernetes users (K8s uses symlinks to<a href="https://github.com/elastic/beats/issues/1686#issuecomment-244931069"> map log files with useful file names</a>), so we&#x2019;ve listened and <a href="https://github.com/elastic/beats/pull/2478">added back support for them</a>. However, the symlink support is off by default and marked as experimental at this point. </p> <h3>Filebeat: close_removed and clean_removed are now on by default</h3> <p>This will keep the registry file from growing to quickly. For background: in 1.x the registry file was a map using the filename as a key. In 5.x the registry file is a list. A side effect of this change is that the registry file grows even if the same file names are reused (think of the usual syslog file names). By making <a href="https://github.com/elastic/beats/pull/2488">these two options on by default</a>, we&#x2019;re making sure the registry file stays small. </p> <h3>Metricbeat: Improvements to the MySQL module</h3> <p>A community contribution by <a href="https://github.com/claygorman">@claygorman</a> adds several <a href="https://github.com/elastic/beats/pull/2484">thread related metrics</a> to the MySQL module in Metricbeat. </p> <h3>Store the version in the Elasticsearch templates</h3> <p>We now <a href="https://github.com/elastic/beats/pull/2445">store the Beats version in the `_meta` section</a> of the Elasticsearch templates we provide. This can help with troubleshooting and perhaps we could use it in the future to provide automatic mappings upgrades. </p> <h3>import_dashboards: automatically select the right Beat</h3> <p>Now that the import_dashboard scripts were rewritten in Go and download the beats-dashboards package when executed, we wanted to make sure that it does the &#x201C;right thing&#x201D; when called without any arguments. So we now <a href="https://github.com/elastic/beats/pull/2429">adjust</a> the <a href="https://github.com/elastic/beats/pull/2458">binary</a> at build time to have a different `-beat` argument depending on the package. </p> <h3>Packetbeat: Improve handling of HTTP messages larger than 10MB</h3> <p>Old implementation dropped messages larger than 10MB (hard coded), to protect against memory DoS. However, in some cases the transaction was still recorded, but the parsing of the headers was incomplete. <a href="https://github.com/elastic/beats/pull/2470">This PR</a> makes the support for large messages explicit, by adding a mode to the parser that &quot;sees&quot; the segments without storing them. </p></body></html>

brewing-1-gold-brewer.jpg

Brewing in Beats: HAProxy module in Metricbeat

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>Kafka output: configurable partitioning strategies</h3> <p>The Kafka output in libbeat now <a href="https://github.com/elastic/beats/pull/2284">implements</a> several strategies for selecting the Kafka partition: hash, round robin, random. The hash partitioner works on a key configured via the string formatter (i.e. can be a field or a combination of fields). </p> <h3>Long running Filebeat tests</h3> <p>To detect potential issues with long running filebeat instances, a <a href="https://github.com/elastic/beats/tree/master/filebeat/tests/open-file-handlers">test set up</a> was created. Metricbeat is used to monitor Filebeat and report the open file handlers from the operating system to Elasticsearch. </p> <h3>Log total internal metrics values on shutdown</h3> <p>The Beats now <a href="https://github.com/elastic/beats/pull/2349">dump their internal metrics</a> on shutdown at INFO level. This completes the <a href="https://github.com/elastic/beats/issues/1931">effort</a> of reducing the log verbosity on INFO/WARN while still providing the needed information for troubleshooting. </p> <h3>Docs on Metricbeat container support</h3> <p>Besides improving the documentation on cgroups, <a href="https://github.com/elastic/beats/pull/2311">this</a> includes a <a href="https://www.elastic.co/guide/en/beats/metricbeat/master/_running_metricbeat_in_a_container.html">guide</a> on how to run Metricbeat in a container to monitor the other containers and the services they run. </p> <h3>Packetbeat: IPv6 support for matching processes</h3> <p>When running on the same host as the monitored application, Packetbeat can correlate the network traffic with the process that created the traffic based on data in the proc file system. This <a href="https://github.com/elastic/beats/pull/2254">PR</a>, coming from a&#xA0;community contribution by <a href="https://github.com/raboof">@raboof</a>, makes this work also when IPv6 addresses are involved. </p> <h3>Packetbeat: display IP addresses when listing the devices</h3> <p>Via another <a href="https://github.com/elastic/beats/pull/2327">community contribution</a>, this time by <a href="https://github.com/noricube">@noricube</a>, the <code>-devices</code> flag from Packetbeat now also lists the IP addresses. This makes it easier to figure out which device is which on Windows. </p> <h3>Sample dashboards import/export improvements</h3> <p>We&#x2019;ve got <a href="https://github.com/elastic/beats/pulls?utf8=%E2%9C%93&amp;q=is%3Apr%20closed%3A%3E2016-08-21%20is%3Aclosed%20dashboards">several small fixes and improvements</a> for the scripts that we use to save, package, and load the sample dashboards. </p> <h3>Fix panic on empty host array</h3> <p><a href="https://github.com/elastic/beats/pull/2326">This</a> was affecting the libbeat Logstash output, in the master version only. </p></body></html>

Brewing in Beats: Configurable Kafka partitioning strategies

We’re happy to announce the fifth alpha release of the Beats 5.0 series.

<html><head></head><body><p>We&#x2019;re happy to announce the fifth alpha release of the Beats 5.0 series. As we slowly approach the GA release, we&apos;re polishing the Beats with lots of small improvements and fixes. &#xA0;You can find all the details in the release notes below, or read the rest of the blog post for the highlights. </p> <p><strong></strong> </p> <ul> <li><a href="https://www.elastic.co/downloads/beats">Download Beats 5.0.0-alpha5</a></li> <li><a href="https://www.elastic.co/guide/en/beats/libbeat/5.0/release-notes-5.0.0-alpha5.html">Beats 5.0.0-alpha5 release notes</a></li> <li>Beats 5.0.0-alpha4 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha4-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha4.html">release notes</a></li> <li>Beats 5.0.0-alpha3 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha3-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha3.html">release notes</a></li> <li>Beats 5.0.0-alpha2 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha2-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha2.html">release notes</a></li> <li>Beats 5.0.0-alpha1 <a href="https://www.elastic.co/blog/beats-5-0-0-alpha1-released">blog post</a> and <a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha1.html">release notes</a></li> </ul> <p><strong></strong> </p> <p>IMPORTANT: This is an alpha release and is intended for testing purposes only. Please do not deploy in production. Yada yada. </p> <h3>Automatically load the right Elasticsearch template</h3> <p>Elasticsearch 5.0 comes with several <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/breaking_50_mapping_changes.html">mapping changes</a>, but it is still able to use the old 2.x templates for backwards compatibility. This is good news because it means that you can use Beats 1.x with either Elasticsearch 2.x or 5.x. However, because we want the Beats 5.x to use the new mapping features, but also be able to work with Elasticsearch 2.x, we added a feature that can query the Elasticsearch version and automatically load the best template for you. This way, you have complete freedom in planning your rolling upgrades. </p> <h3>More filtering flexibility</h3> <p>Beats 5.0.0-alpha5 introduces simple processors (previously called &quot;generic filters&quot;) that allow you to flexibly choose which events or fields to drop based on simple conditions. Starting with alpha5, these conditions can also be combined with logical operators (AND/OR/NOT) so they don&#x2019;t have to be so simple anymore. You can then express conditions like &#x201C;drop all logs about 200 and 404 responses, except if the response time is larger than 100 milliseconds&#x201D;. </p> <h3>Filebeat registry file cleanup + more options</h3> <p>Filebeat stores its state about the files it reads in a registry file on disk. This way, it can avoid shipping the same log lines after a restart. With the 5.0.0-alpha5 release, we introduce new configuration options that allow you to configure when to remove entries from this registry file. This means you can make sure that the registry file doesn&#x2019;t grow forever. On the same note, other Filebeat configuration settings were renamed for consistency and we now have a short <a href="https://www.elastic.co/guide/en/beats/filebeat/master/_how_filebeat_works.html">guide</a> explaining what each Filebeat component does. </p> <h3>Override settings from the CLI via the -E flag</h3> <p>Do you like how Elasticsearch allows you to set any configuration setting from the command line by using the <code>-E</code> flag (yes, it used to be <code>-D</code>, but was renamed in 5.0 to avoid confusion with the JVM flags)? Well, the same is now possible for all Beats. For example, you can quickly enable the console output by adding <code>-E output.console.pretty=true</code>. </p> <h3>More configuration flexibility</h3> <p>On the same theme with the <code>-E</code> flag, you can now specify multiple configuration files by repeating the <code>-c</code> flag. Settings in subsequent config files override those that precede them. You can use this, for example, for setting defaults in a base configuration file, and overwrite settings via local configs. </p> <p><strong></strong> </p> <p>We have also standardized on using <code>enabled: false</code> as a way to disable most things in the configuration file. For example, if you want to disable a Packetbeat protocol without commenting out 10 lines of config, add <code>enabled: false</code>. Want to disable an output? The same <code>enabled: false</code> does the trick. </p> <h3>New defaults for the logging verbosity</h3> <p>We did some rethinking of the Beats logging strategy. Part of this, we&#x2019;ve switched the default log level from ERROR to INFO and reduced the verbosity of most warnings and info messages. To compensate, we added a set of internal metrics which also get logged to the INFO level every 30 seconds. This means you get good visibility into what is happening inside the Beats without getting huge files during periods of high traffic. </p> <h2>Become a Pioneer</h2> <p>A big Thank You to everyone who has tried the previous alpha releases and <a href="https://github.com/elastic/beats/issues/new">posted issues</a> or <a href="https://discuss.elastic.co/c/beats">provided feedback</a>. We&#x2019;d like to also remind you that if you post a valid, non-duplicate bug report during the alpha/beta period against any of the Elastic stack projects, you are entitled to a <a href="https://www.elastic.co/blog/elastic-pioneer-program">special gift package</a>. </p> <span></span> <span></span> <span></span></body></html>

Beats 5.0.0-alpha5 released

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p><h3>New community beat: Burrowbeat</h3><p><a href="https://github.com/goomzee/burrowbeat">Burrowbeat</a> is a new Beat that can be used to monitor consumer lags in Kafka queues. It is based on the <a href="https://github.com/linkedin/Burrow/wiki/What-is-Burrow">Burrow</a> project.
</p><h3>Configurable index patterns</h3><p>The Beats now offer <a href="https://github.com/elastic/beats/pull/2119">a lot more flexibility over the index names</a>. They used to enforce daily indices of the form <code>beatname-%{+yyyy.MM.dd}</code>, which is still the default. But it&#x2019;s now possible to do weekly or hourly indices using a syntax similar to Logstash, or something like <code>beatname</code> to work with the new <a href="https://www.elastic.co/blog/managing-time-based-indices-efficiently">rollover APIs</a> from Elasticsearch, or even <code>beatname-%{type}-%{+yyyy.MM.dd}</code> to automatically split the data into separate indices based on any field from the events.
</p><h3>Use scaled floats for percentages</h3><p>After adding support for half floats a few weeks back, Adrien <a href="https://github.com/elastic/elasticsearch/pull/19264">did it again</a> and added scaled floats to Elasticsearch &#x1F389;. Scaled floats are stored as longs behind the scenes, which makes them benefit of the compression scheme used for integers in Lucene. This makes them a great fit for the way we store percentages in Beats: a number between 0 and 1 that gets formatted as a percentage by Kibana.
</p><p>This <a href="https://github.com/elastic/beats/pull/2156">PR</a> switches our percentages to use scaled_float. We had quite a few of those, so we can expect a significant improvement in the storage footprint.
</p><h3>Automatically generate Kibana index patterns</h3><p>The sample Beats dashboards contain the Kibana index patterns, which allow us to define custom formatting for some of our fields (think of percentages). Providing the index pattern also saves the user a step while getting started. We used to create these index patterns by exporting them from Kibana, in a mostly manual process. To improve on this, Monica created a script to <a href="https://github.com/elastic/beats/pull/2122">generate the index patterns</a> from our <code>fields.yml</code> files, which are the primary source data for everything that the Beats export.
</p><h3>Lookup functionality</h3><p>The <a href="https://github.com/elastic/beats/issues/2186">lookup feature</a> allows attaching arbitrary metadata to events by calling external scripts. The scripts are called with parameters that can be taken from the original event, and the results cached for that parameters. This can be used, for example, to attach extra metadata to every file read by Filebeat by calling the Kubernetes APIs. The same could be used to add custom metadata to every process monitored by Metricbeat. Since the results are cached, the external script is called only once per file/process.
</p><p><a href="https://github.com/elastic/beats/pull/2153">This functionality is now merged</a> into an experimental branch called <code>x-exec-lookup</code>, which we plan to merge into master after we add several security checks around calling external scripts.
</p><h3>Filebeat: shorter close_inactive default</h3>The <code><a href="https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#close-inactive">close_inactive</a></code> option sets after what time interval Filebeat closes the open files that don&#x2019;t receive updates. The old default used to be 1h, which could mean that we keep a lot of open files in the case of quickly rotating files. This <a href="https://github.com/elastic/beats/pull/2157">PR</a> changes the default to 5m. If a file receives updates after it was closed, it is picked up again by Filebeat, so the lower default doesn&#x2019;t mean any risk of data loss.</body></html>

Brewing in Beats: Configurable index patterns

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p><h3>In progress: add cgroup information to gosigar</h3><p>As part of a larger effort to make container monitoring easy with the Beats, <a href="https://github.com/elastic/gosigar">gosigar</a> (the library we use in Metricbeat) is getting support for <a href="https://github.com/elastic/gosigar/pull/37">parsing and understanding</a> the data from <code>/proc/cgroups</code>. This gives us not only the CPU and memory usage of each container, but also the configured limits for each. This is a feature requested, for example, by our Cloud team to properly report on the cpu usage as a percentage of the configured limit.
</p><h3>Metricbeat: Conversion helpers refactoring</h3><p>The effort started last week to make most of the parsing and conversion work in Metricbeat declarative was <a href="https://github.com/elastic/beats/pull/2040">completed</a> this week. The bulk of the work involved in writing and maintaining a Metricbeat module lies in renaming and converting the metrics, so we hope that doing this declaratively will be an important productivity win. The Metricbeat <a href="https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-developer-guide.html">developer guide</a> was also <a href="https://github.com/elastic/beats/pull/2057">updated</a>.
</p><h3>How Filebeat works documentation</h3><p>Getting confused about what a Prospector or what a Harvester is? Don&#x2019;t know the difference between <code>close_inactive</code> and <code>clean_inactive</code>? We <a href="https://github.com/elastic/beats/pull/1963">now</a> have a <a href="https://www.elastic.co/guide/en/beats/filebeat/master/_how_filebeat_works.html">single docs page</a> explaining the high level Filebeat concepts and docs <a href="https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#close-options">sections</a> to explain all the relevant configuration options.
</p><h3>Path variables available in the configuration file</h3><p>It is now possible to refer to <a href="https://github.com/elastic/beats/pull/2042">path variables</a> (e.g. path.home, path.data, path.config) in the configuration file. The benefit is that it is a bit more clear to write <code>filebeat.registry_file: ${path.data}/registry</code> than just the equivalent <code>filebeat.registry_file: registry</code>.
</p><h3>Format string support in libbeat</h3><p>In several planned features (the lookup feature, <a href="https://github.com/elastic/beats/issues/2064">dynamic pipeline selection</a>, dynamic <a href="https://github.com/elastic/beats/issues/2074">index pattern</a>) it is useful to be able to format strings based on event fields. This <a href="https://github.com/elastic/beats/pull/2065">PR</a> adds supports in libbeat for using a format similar to the one used by Logstash. E.g. `pipeline: %{[fields.type]}`.
</p><h3>Metricbeat: add PGID to process data</h3><p>Coming from a <a href="https://github.com/nickivanov">community contributor</a>, this <a href="https://github.com/elastic/beats/pull/2021">PR</a> adds the process group id information to the per process docs.
</p><h3>Quickly switch Packetbeat protocols on/off in config</h3><p>We have standardized on using <code>enabled: false</code> as a way to disable most things in the configuration file. For example, if you want to disable a Packetbeat protocol without commenting out 10 lines of config, add <code>enabled: false</code>. Want to disable an output? Add <code>enabled: false</code>.
</p><h3>Filebeat cleanups</h3><p>The effort (<a href="https://github.com/elastic/beats/issues/1913">meta ticket</a>) of cleaning up the Filebeat code continued last week with this <a href="https://github.com/elastic/beats/pull/2020">PR</a> doing lots of small improvements.
</p><h3>Potential Filebeat file descriptor leak fixed</h3><p>A potential FD leak was <a href="https://github.com/elastic/beats/issues/2028">discovered</a> and <a href="https://github.com/elastic/beats/pull/2029">fixed</a>. The file descriptors would have gotten garbage collected eventually, but it&#x2019;s better to close them as soon as they are not needed anymore.
</p><h3>Kibana Dashboard for the Metricbeat network stats</h3><p>This <a href="https://github.com/elastic/beats/pull/2071">PR</a> adds a new sample dashboard that can be used with the networking stats collected by the Metricbeat system module.
</p></body></html>

Brewing in Beats: How Filebeat works

With this series, we're keeping you up to date with all that's new in Beats.

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p><h3>New community Beat: Saltbeat</h3><p>Salt is an open source automation framework, similar to Ansible for example. <a href="https://github.com/martinhoefling/saltbeat">Saltbeat</a> is a forwarder for messages from the salt master event bus to Logstash and Elasticsearch.
</p><h3>Set configuration options via the CLI</h3><p>The Beats are <a href="https://github.com/elastic/beats/pull/1986">getting a -E flag</a>, similar to the Elaticsearch -E option, that allows overwriting configuration options from the command line. For example, you can now quickly enable the console output by adding <code>-E output.console.pretty=true</code>.
</p><h3>Combine processor conditions with and/or/not</h3><p>The Beats processors (a.k.a generic filters), which are new in 5.0, now support <a href="https://github.com/elastic/beats/pull/1983">combining conditions with logical operators</a>. This means you can express complex conditions for filtering out the interesting events. For example, to drop all events that have codes 200 or 404 on a given URL:
</p><pre class="prettyprint">drop_event.when:
  and:
    - equals.request_url: &#x201C;/test&#x201D;
    - or:
      - equals.http.code: 404
      - equals.http.code: 200
</pre><h3>Logging verbosity cleanup + more internal metrics</h3><p>We started with a <a href="https://github.com/elastic/beats/issues/1734">bug report</a> (thanks Lee) that highlighted how sometimes we fail to provide the operator with important information and started doing a <a href="https://github.com/elastic/beats/issues/1931">largish effort</a> to rethink what we log and with what verbosity level. Part of this, we <a href="https://github.com/elastic/beats/pull/1948">reduced</a> the numbers of WARN and INFO messages that we produce, so we can switch the default log level <a href="https://github.com/elastic/beats/pull/1952">from ERROR to INFO</a>. To compensate, we replaced the most verbose warnings/infos with <a href="https://github.com/elastic/beats/pull/1955">internal metrics</a>. These internal metrics are dumped periodically in logs and will provide data for the future central monitoring system.
</p><h3>Environment variable expansion based on go-ucfg</h3><p>The previous implementation for environment variable expansion in our configuration files was done via textual replacement before parsing. This was easy and elegant to implement, but could lead issues when the <code>${var}</code> sequence shows up accidentally (e.g. in a password field). By <a href="https://github.com/elastic/beats/pull/1984">passing the expansion responsibility</a> to our new <a href="https://github.com/elastic/go-ucfg">go-ucfg library</a>, we can be more selective on where we accept variable expansions.
</p><p>This also comes with a breaking change: we used to replace unresolved variables with an empty string, the shell way. Now unresolved variables result in a configuration error, which should help catching configuration bugs earlier.
</p><h3>Specify multiple configuration files</h3><p>Another improvement that we owe to go-ucfg is that you can now specify multiple configuration files by <a href="https://github.com/elastic/beats/pull/1985">repeating the -c flag</a>. You can use this, for example, for setting defaults in a base configuration file, and overwrite settings via local configs.
</p><h3>Filebeat fix for very quick file rotation</h3>In case a file was renamed after the state was read but before the file was opened by the harvester it could happen that the wrong file was opened. This lead to the <a href="https://github.com/elastic/beats/pull/1954">issue</a>, that the wrong file was read including reporting the wrong state for a file. In general this was rather unlikely to happen, but can happen when <code>scan_frequency</code> is very low and number of files rotated is very high.</body></html>

Brewing in Beats: Set configuration options from the CLI

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats.

<html><head></head><body><p>Welcome to Weekly Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in&#xA0;<a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p>
<h3>New Community Beat: Logstashbeat</h3>
<p>Logstash now has a monitoring API, so it can be monitored with Beats :-). <a href="https://github.com/consulthys" target="_blank">Valentin Crettaz</a> has created a new community Beat called <a href="https://github.com/consulthys/logstashbeat" target="_blank">Logstashbeat</a> that gathers stats (events, jvm and process) from a running Logstash instance and ship them to Elasticsearch.
</p>
<h3>New Community Beat: Nginxupstreambeat</h3>
<p>Nginxustreambeat is a Beat used for Nginx upstream server status monitoring. The Nginx <a href="https://github.com/2Fast2BCn/nginx_upstream_check_module" target="_blank">upstream check module</a>, by the same <a href="https://github.com/2Fast2BCn" target="_blank">author</a>, &#xA0;reads the statistics periodically and indexes them in Elasticsearch. This is very useful when using Nginx as a reverse proxy to load-balance among multiple upstream servers.
</p>
<h3>New Community Beat: Lambdabeat</h3>
<p><a href="https://github.com/michaelbaamonde">Mike Baamonde</a>, one of our Elastic colleagues, created&#xA0;<a href="https://github.com/michaelbaamonde/lambdabeat">Lambdabeat</a>&#xA0;to collect various&#xA0;AWS Lambda metrics and index them in&#xA0;Elasticsearch. Lambdabeat fetches data from the&#xA0;Cloudwatch API, so you don&apos;t need to&#xA0;deploy it&#xA0;anywhere in particular, but it does require valid AWS credentials.
</p>
<h3>New Generators, now in the main Beats repository</h3>
<p>The Beat generator proved to be extremely useful in quickly bootstrapping new Beats and keeping them up-to-date with our changes. As we&#x2019;re adding different types of generators, like the one that generates a <a href="https://github.com/elastic/beats/pull/1770" target="_blank">Metricbeat powered Beat</a>, or one for generating a Beat that <a href="https://github.com/elastic/beats/pull/1771" target="_blank">implements a single Packetbeat protocol</a>, we figured it makes more sense to <a href="https://github.com/elastic/beats/pull/1769" target="_blank">group these generators</a> in a single place. This place is inside the main <a href="https://github.com/elastic/beats/tree/master/generate" target="_blank">Beats repository.</a>
</p>
<h3>Filebeat: symlinks are no longer followed</h3>
<p>Following symlinks to log files in Filebeat was asking for trouble, for example it could happen that the same file was read twice. We will be <a href="https://github.com/elastic/beats/pull/1767" target="_blank">removing this feature</a> with the next alpha release. Symlinks are now completely ignored from now on.
</p>
<h3>Fixed backoff logic in outputs</h3>
<p>Coming from a community contributor, this <a href="https://github.com/elastic/beats/pull/1814" target="_blank">PR</a> fixes a bug in the backoff retry code in libbeat.
</p>
<h3>Metricbeat: automatically document events</h3>
<p>Every Metricbeat module so far comes with Dockerized tests to verify it against a real instance of the monitored system. We now take <a href="https://github.com/elastic/beats/pull/1785" target="_blank">advantage of this</a> and automatically generate samples of the events by running the integration tests.
</p>
<h3>MongoDB Metricbeat module</h3>
<p>A <a href="https://github.com/elastic/beats/pull/1837" target="_blank">MongoDB module</a> for Metricbeat is currently in progress.
</p></body></html>

Brewing in Beats: Logstashbeat, Nginxupstreambeat, Lambdabeat

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>New community Beat: CloudTrailBeat</h3> <p>An interesting new Beat was created by <a href="https://github.com/aidan-">Aidan Rowe</a>. <a href="https://github.com/aidan-/cloudtrailbeat">CloudTrailBeat</a> relies on a combination of SNS, SQS and S3 to create a processing &apos;pipeline&apos; to process new log events quickly and efficiently. The Beat polls the SQS queue for notification of when a new CloudTrail log file is available for download in S3. Each log file is then downloaded, processed and sent to the configured receiver (logstash, elasticsearch, etc). </p> <h3>Metricbeat: export CPU usage fields as percentages</h3> <p>With <a href="https://github.com/elastic/beats/pull/1705">this change</a>, the system module in Metricbeat exports all CPU usage fields (user, system, idle, etc.) as percentages from the total CPU time. This makes it easier to make sense of the values. The old &#x201C;ticks&#x201D; units are also available, but disabled by default. </p> <h3>Metricbeat: export network IO stats</h3> <p>This new <a href="https://github.com/elastic/beats/pull/1732"><u>metricset</u></a> provides network IO metrics collected from the operating system. An event is generated for each network interface. This metricset is available on Darwin (aka OSX), Windows, Linux, and FreeBSD. </p> <h3>Filebeat new registry file format</h3> <p>The previous format for the registry file was a dictionary with the file paths as keys. This had the potential to lead to overwrite and conflicts on file rotation. As the file path is also stored inside the state object this information was duplicated. Now <a href="https://github.com/elastic/beats/pull/1703"><u>a pure array is stored in the registry</u></a> which makes the format more flexible and brings it close to the format used by Logstash. </p> <p>The migration from the old format happens automatically on the first run. </p> <h3>Lumberjack code extracted in a library</h3> <p>We now have a <a href="https://github.com/urso/go-lumber"><u>go-lumber</u></a> library that implements both the client and the server side of the Lumberjack protocol, which we use between Beats and Logstash. The unit tests for logstash output now use <a href="https://github.com/elastic/beats/pull/1715"><u>lumberjack server implementation from go-lumber</u></a>, simplifying test code quite a bit. Custom incomplete lumberjack server has been removed from output plugin. The above referenced PR also does the first steps towards replacing the default JSON encoding with custom code that does JSON encoding without reflection and with fewer allocations. </p> <h3>Generator for Metricbeat modules/metricsets</h3> <p>Adding a MetricSet requires creating several files with a predefined file structure for automation. To simplify the task of creating a MetricSet and make sure the correct structure is used, this task is now <a href="https://github.com/elastic/beats/pull/1738">automated with a script</a>. </p> <h3>Normalizing Metricbeat fields</h3> <p>We&#x2019;ve continued to normalize the fields exported by Metricbeat to use a common format. This week the <a href="https://github.com/elastic/beats/pull/1700"><u>MySQL module</u></a> and the <a href="https://github.com/elastic/beats/pull/1754"><u>system module</u></a> field names were refactored. </p> <h3>Configuration files improvements</h3> <p>Continuing the theme from the last weeks, the configuration files were improved <a href="https://github.com/elastic/beats/pull/1690"><u>by adding Kafka and Redis outputs</u></a> to the &#x201C;full&#x201D; versions. Our system tests were <a href="https://github.com/elastic/beats/pull/1688"><u>updated</u></a> to use a similar structure for configuration as the files that we ship. Also, <a href="https://github.com/elastic/beats/pull/1709"><u>the docs were updated</u></a> to use the same style in configuration samples. </p> <h3>Packetbeat NFS fixes and refactoring</h3>Community contributor <a href="https://github.com/kofemann"><u>kofemann</u></a> has done a great job improving the NFS module in packetbeat with fixes and refactorings to make the code cleaner.</body></html>

Brewing in Beats: Getting Metricbeat ready for release

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <p>Last week we have released Beats 1.2.3 containing a few important <a href="https://www.elastic.co/blog/beats-1-2-3-released">bug fixes</a>. Besides that, we have worked on a set of improvements that we plan to release with 5.0.0-alpha3: </p> <h3>Conditionals for generic filtering</h3> <p>It is now <a href="https://github.com/elastic/beats/pull/1623">possible</a> to use conditionals when filtering out fields and events in libbeat. This makes it really easy to remove unneeded data even before sending it over the network. Here is an example filter that drops the HTTP header fields if the code is 200 and the status is OK: </p> <pre class="prettyprint">- drop_fields:
    fields: [&quot;http.request_headers&quot;, &quot;http.response_headers&quot;]
    equals:
      http.code: 200
      status: OK     </pre> <h3>Metricbeat gets an Nginx module</h3> <p>KS Chan, the person behind <a href="https://github.com/mrkschan/nginxbeat">Nginxbeat</a>, has contributed an <a href="https://github.com/elastic/beats/pull/1642">Nginx module</a> to Metricbeat. </p> <h3>Metricbeat progress</h3> <p>As anticipated last week, the work has started to normalize the metric names so that users get an uniform experience regardless of the system they monitor. For example, here is the <a href="https://github.com/elastic/beats/pull/1656">PR for Redis</a> and here is the one <a href="https://github.com/elastic/beats/pull/1683">for Apache</a>. </p> <p>Other Metricbeat improvements include: </p> <ul> <li>Add <a href="https://github.com/elastic/beats/pull/1632">connection timeout</a> to Redis Module.</li> <li>The Dockerfiles use for testing now have a <a href="https://github.com/elastic/beats/pull/1657">standard place</a> in each module folder.</li> <li>The list of imported modules is now <a href="https://github.com/elastic/beats/pull/1661">automatized</a>.</li> <li>The system module is now able to <a href="https://github.com/elastic/beats/pull/1665">filter processes by regexps</a>, similar with Topbeat.</li> <li>The metricset fields are <a href="https://github.com/elastic/beats/pull/1677">grouped together</a>.</li> </ul> <h3>Configuration refactoring</h3> <p>The work on reorganizing the options in our configuration files is almost <a href="https://github.com/elastic/beats/issues/1417">finished</a>. We now distribute two versions of the configuration files: the default one which contains only the most common options and a &#x201C;full&#x201D; one, that contains all non-deprecated options with longer comments. The full versions can be almost used as a reference and it is easy to copy and paste between them. </p> <p>Another configuration related improvement is that all <a href="https://github.com/elastic/beats/pull/1659">duration options</a> now accept the same specifiers (e.g 10s, 1ms). This change was done in a backwards compatible way. </p> <h3>Static builds</h3> <p>We now produce <a href="https://github.com/elastic/beats/pull/1673">statically linked binaries</a> for Filebeat and Winlogbeat (which don&apos;t use CGo) by using simple cross compilation, without Cgo enabled. The advantage of the statically linked binaries is that they tend to be more portable. The Beats that use CGo continue to be dynamically linked against libc and not much else. </p> <h3>Package names</h3> <p>We have updated our <a href="https://github.com/elastic/beats/pull/1666">artifact naming</a> to be consistent with the other Elastic stack projects. </p> <h3>Filebeat to Logstash performance</h3>The work on figuring out why the Filebeat to Logstash communication is slower than expected is now resumed. The current intention is to rewrite the beats input plugin in Logstash to use Netty. It&#x2019;s really early, but there are some very promising <a href="https://github.com/logstash-plugins/logstash-input-beats/issues/45#issuecomment-220094917">results</a> already.</body></html>

Brewing in Beats: Conditional Filtering

We’re happy to announce that the 1.2.3 bug fix release is now available for all the Elastic Beats: Filebeat, Packetbeat, Topbeat and Winlogbeat.

<html><head></head><body><p>We&#x2019;re happy to announce that the 1.2.3 bug fix release is now available for all the Elastic Beats: Filebeat, Packetbeat, Topbeat and Winlogbeat. You can download the new versions from the usual <a href="https://www.elastic.co/downloads/beats/">place</a>. You can find the full release notes <a href="https://www.elastic.co/guide/en/beats/libbeat/current/release-notes-1.2.3.html">here</a>. </p> <h3>Fix Filebeat sending duplicate logs of rotated files on restart</h3> <p>This fixes an <a href="https://github.com/elastic/beats/issues/1528">issue</a> where Filebeat didn&#x2019;t record the file offset on shutdown of rotated files. This could cause resending of already shipped files on restart. Note that while the bug is fixed with the 1.2.3 release, the duplicates can still happen one time only on the upgrade from 1.2.2. </p> <h3>Fix Topbeat high CPU usage on Windows</h3> <p>Before this release, Topbeat could cause <a href="https://github.com/elastic/beats/issues/1460">high CPU usage</a> on Windows when using a custom regular expression pattern to filter processes. The root cause was that the caching that we were using for command lines (getting the command line is fairly expensive on Windows) wasn&#x2019;t working when most of the processes were filtered out. </p> <h3>Fix Winlogbeat panic on large events</h3> <p>Messages larger than 32K could cause Winlogbeat to crash on Windows XP or Windows 2003. There are more details in this <a href="https://github.com/elastic/beats/pull/1498">pull request</a>. </p> <h3>Easy as 1, 2, 3</h3>Many thanks to our dear community members who tested the Beats, reported and helped with the troubleshooting of the above issues. If you find any issues with the new release or there are features that you miss, please interact with us on <a href="https://discuss.elastic.co/c/beats">Discuss</a> or <a href="https://github.com/elastic/beats">GitHub</a>.</body></html>

Count with us: Beats 1.2.3 released

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <p>Last week we released 5.0.0-alpha2, with several important <a href="https://www.elastic.co/blog/beats-5-0-0-alpha2-released">bug fixes and new features</a>. Apart from that, here is what we are busy with: </p> <p><strong></strong> </p> <h3>Per process stats in Metricbeat</h3> <p>We had a lot of back and forth on this <a href="https://github.com/elastic/beats/pull/1081">PR</a>, debating how we should best represent in Elasticsearch the metrics that are per &#x201C;dynamic entity&#x201D;. A good example of such entities are the processes in an operating system, but can also be mounted file systems or tables in an SQL database. After considering nested objects or creating dynamic fields, we went for the simpler solution of having individual objects for each entity. These objects are grouped in their own metricset. </p> <h3>Metricbeat API refactoring</h3> <p>In order to have a clean and stable API before adding more modules, Andrew did a lot of refactoring and documenting work on the <a href="https://github.com/elastic/beats/pull/1549">Metricbeat modules APIs</a>. </p> <h3>Configuration migration tool</h3> <p>Between the 1.x and 5.0 there are a few configuration file changes that break compatibility. We now have a <a href="https://github.com/elastic/beats/pull/1576">python script</a> that should help with migrating existing configuration files to 5.0 format. The script doesn&#x2019;t do real YAML parsing, meaning that it won&#x2019;t work on any possible configuration file, but it should do the job in most cases. It also requires no dependencies besides python itself and preserves the comments. </p> <h3>Zookeeper module in Metricbeat</h3> <p><a href="https://github.com/jeredding/zkbeat">Zkbeat</a>, created by Erik Redding, was <a href="https://github.com/elastic/beats/pull/1505">converted</a> into a Metricbeat module. It uses the Zookeeper <code>mntr</code> command to get simple stats. Thanks Erik! </p> <h3>Kafka output deadlock fix</h3> <p>Due to a bug in the Go library we use for outputting to Kafka, a <a href="https://github.com/elastic/beats/issues/1432">deadlock</a> is possible in case infinite retries are used. In time for alpha2, a <a href="https://github.com/elastic/beats/pull/1543">set</a> of <a href="https://github.com/elastic/beats/pull/1540">refactorings</a> were <a href="https://github.com/elastic/beats/pull/1551">done</a> to <a href="https://github.com/elastic/beats/pull/1550">workaround</a> this issue. We now avoid asking for infinite retries from the library and instead we simulate it in our code. </p> <h3>Topbeat: fix high CPU usage on windows</h3> <p>We had a couple of interesting <a href="https://github.com/elastic/beats/issues/1460">reports</a> that indicated that on Windows Topbeat uses more CPU when it is configured to monitor less processes. It turns out that the reason was failing to use the command line cache (getting the full command line is expensive on Windows) for the processes that were filtered out. The fixing PR <a href="https://github.com/elastic/beats/pull/1562">refactors</a> the logic so that the command line is not read at all if the process is filtered out. </p> <h3>Filebeat refactoring</h3> <p>Heavy <a href="https://github.com/elastic/beats/pull/1521">cleanup work</a> is <a href="https://github.com/elastic/beats/pull/1532">going on</a> in the Filebeat code, making the code more readable and easier to maintain. &#xA0;Previously the file state was loaded from disk every time a new file was found. The state from the registry file <a href="https://github.com/elastic/beats/pull/1558">is now only loaded</a> once during startup and from the one the prospector internal in memory state is used. This is more efficient and prevents race conditions. </p> <p><strong></strong> </p> <p>It also makes the Registrar and the Harversters <a href="https://github.com/elastic/beats/pull/1515">less coupled</a>. This refactoring work is crucial for us to be able to stay on top of possible races due to all the file rotation and file systems variations. </p> <h3>Normalize new line character after multiline</h3> <p>In order for the regular expressions to work in a consistent way on multiline events, this <a href="https://github.com/elastic/beats/pull/1552/files">PR</a> makes sure <code>\n</code> is used as a line separator after multiline stitching. </p> <span></span> <span></span></body></html>

Brewing in Beats: Metricbeat progress

Today we’re announcing the Beats 5.0.0-alpha2 release. It includes bug fixes and new features for Filebeat, Packetbeat, Topbeat, and Winlogbeat.

<html><head></head><body><p>Around a month ago we released the <a href="/blog/beats-5-0-0-alpha1-released">first Alpha version</a> of the next major release for the Elastic Stack. Today we&#x2019;re proceeding according to the plan and announce the 5.0.0-alpha2 release. It includes bug fixes and new features for the data collectors you know and love: Filebeat, Packetbeat, Topbeat, and Winlogbeat.
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats/">Download links</a></li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha2.html">Beats 5.0.0-alpha2</a> release notes</li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha1.html">Beats 5.0.0-alpha1</a> release notes</li>
</ul><h2>What&#x2019;s new?</h2><h3>Improved Redis output</h3><p>The Beats had a Redis output from the earliest days, but we neglected it for a while as we thought we will remove it to focus on the Logstash and Elasticsearch outputs. By popular demand we reverted that decision and, after introducing the new Kafka output in alpha1, we have now completely reworked the Redis output for alpha2.
</p><p>It now supports the guaranteed mode needed by Filebeat and Winlogbeat, meaning that events won&#x2019;t be lost in case of network unavailability. It also supports authentication, SOCKS5 proxies, and even <a href="https://www.elastic.co/guide/en/beats/filebeat/master/redis-output.html#_tls_3">encryption</a>. Redis itself doesn&#x2019;t natively support encryption, but you can make use of <a href="https://www.stunnel.org/index.html">stunnel</a> to secure the communication between the Beats and Redis.
</p><h3>Kibana dashboards per Beat</h3><p>Our <a href="https://github.com/elastic/beats-dashboards">sample Kibana dashboards</a> proved to be quite popular, so we worked on improving the experience around them. Part of this effort was to split them per Beat, so they are easier for us to maintain and more convenient for you to use. This means, for example, that if you only use Topbeat, you can load only the Topbeat dashboard along with all the required saved visualizations and searches.
</p><p>And since the dashboards are now per Beat, we include them in the <a href="https://www.elastic.co/downloads/beats/">downloadable packages</a>, so they are there when you need them.
</p><h3>New directory layout</h3><p>We also reorganized a bit where each Beat looks for and creates its files. This makes it easier to upgrade the Beats that store state and it makes it easier for us to create and maintain features like automatic template loading in a cross-platform way. &#xA0;The new <a href="https://www.elastic.co/guide/en/beats/filebeat/master/directory-layout.html">directory layout</a> will be familiar to you if you know how the Elasticsearch paths are organized.
</p><p>Another notable change is that the Beats now log by default to rotating files instead of syslog. This makes the experience more consistent across the various platforms that we support.
</p><h3>Bug fixes since Alpha1</h3><p>The change log also contains a few fixes for issues discovered since the 5.0.0-alpha1 release. These include bugs related to <a href="https://github.com/elastic/beats/issues/1378">JSON</a> <a href="https://github.com/elastic/beats/issues/1466">decoding</a>, to the automatic <a href="https://github.com/elastic/beats/issues/1321">template loading logic</a>, and to using <a href="https://github.com/elastic/beats/issues/1410">multiple outputs</a>.
</p><h2>Become a Pioneer</h2><p>A big Thank You to everyone that has tried the alpha1 release and has <a href="https://github.com/elastic/beats/issues/new">posted issues</a> or <a href="https://discuss.elastic.co/c/beats">provided feedback</a>. We&#x2019;d like to also remind you that if you post a valid non-duplicate bug report during the alpha/beta period against any of the Elastic stack projects, you are entitled to a <a href="https://www.elastic.co/blog/elastic-pioneer-program">special gift package</a>.
</p></body></html>

Beats 5.0.0-alpha2 released

How to write structured logs in JSON and how to ship them efficiently to Elasticsearch by using Filebeat.

<p>The idea behind structured logging is simple: instead of having applications write logs that need to be parsed via regular expressions into JSON objects that you index into Elasticsearch, you make the application write JSON objects directly. </p> <p>To exemplify, let’s say you are writing a Python web application, and you are using the standard library for logging. When a user signs in, you might use a logging statement that looks like this: </p> <p><span></span></p><pre class="prettyprint">logging.debug("User '{}' (id: {}) successfully logged in. Session id: {}"
              .format(user["name"], user["id"], session_id))</pre>  <p>And then when the user gets verified: </p> <p><span></span></p><pre class="prettyprint">logging.debug("User '{}' (id: {}) changed state to verified."
              .format(user["name"], user["id"]))</pre>  <p>This results in log lines like this: </p> <p><span></span></p><pre class="prettyprint">DEBUG:root:User 'arthur' (id: 42) successfully logged in. Session id: 91e5b9d
DEBUG:root:User 'arthur' (id: 42) changed state to verified.</pre>  <p>This way of logging is popular ever since the good old printf. Back then, you would typically have a single server, and you would tail and grep the log files and everything was great. </p> <p>Times have changed, however, and today it is more likely that you have tens, hundreds, or thousands of servers / virtual machines / containers creating massive amounts of logs, so you centralize them in Elasticsearch and use its magical (that’s just how it feels) search and aggregation features to navigate through them. </p> <p>This works best when the logs are pre-parsed in a structured object, so you can search and aggregate on individual fields. So before indexing, you would typically use Logstash’s amazing Grok filter to parse the application logs into a JSON object, but this means that you have to write and maintain Grok patterns and spend CPU cycles to do the parsing. </p> <p>Now let’s try the same example with structured logging. While usually not in the standard library, all major programming languages have libraries that make structured logging easy. James Turnbull created a list in this <a href="https://kartar.net/2015/12/structured-logging/">blog post</a>, which also goes into detail about how to do this for a Rails application. In Python, there’s the <a href="https://structlog.readthedocs.org/en/stable/">structlog</a> library, which we will use here: </p> <p><span></span></p><pre class="prettyprint">log = log.bind(user='arthur', id=42, verified=False)
log.msg('logged_in')
log.msg('changed_state', verified=True)</pre>  <p>Which results in log lines like this: </p> <p><span></span></p><pre class="prettyprint">verified=False user='arthur' session_id='91e5b9d' id=42 event='logged_in'
verified=True user='arthur' session_id='91e5b9d' id=42 event='changed_state'</pre>  <p>One thing to notice is that the code is less repetitive and it encourages the developer to include all the data rather than only what seems important while writing the code. Also note that in this format, the log lines are still reasonably easy to follow for a human during development. When moving to production, however, it makes more sense to use the JSON renderer: </p> <p><span></span></p><pre class="prettyprint">log = wrap_logger(PrintLogger(), processors=[JSONRenderer()])</pre>  <p>This results in log lines like this: </p> <p><span></span></p><pre class="prettyprint">{"verified": false, "user": "arthur", "session_id": "91e5b9d", "id": 42, "event": "logged_in"}
{"verified": true, "user": "arthur", "session_id": "91e5b9d", "id": 42, "event": "changed_state"}</pre>  <p>This is less readable to human eyes, but has the advantage that the data is already structured in the format that Elasticsearch likes. </p> <p><a href="https://www.elastic.co/beats/filebeat">Filebeat</a> is an open source log shipper, written in Go, that can send log lines to Logstash and Elasticsearch. It offers “at-least-once” guarantees, so you never lose a log line, and it uses a back-pressure sensitive protocol, so it won’t overload your pipeline. Basic filtering and multi-line correlation are also included. </p> <p>Starting with version 5.0 (currently in alpha, but you can give it a <a href="https://www.elastic.co/downloads/beats/filebeat">try</a>), Filebeat is able to also natively decode JSON objects if they are stored one per line like in the above example. </p> <p>Here is a sample configuration file that configures Filebeat to pick up the files and send the JSON objects to Elasticsearch: </p> <p><span></span></p><pre class="prettyprint">filebeat.prospectors:
- input_type: log
  paths: ["test/*"]
  json.message_key: event
  json.keys_under_root: true
  fields:
    planet: Magrathea
    service: ${SERVICE_NAME}
  tags: [
    "i",
    "heart",
    "json"
  ]

output.elasticsearch:
  hosts: ["192.168.99.100:9200"]
  template.name: filebeat
  template.path: filebeat.template.json</pre>  <p>This is one of the JSON objects that gets indexed this way: </p> <p><span></span></p><pre class="prettyprint">  {
    "@timestamp": "2016-04-22T22:09:09.631Z",
    "beat": {
      "hostname": "DeepThought",
      "name": "DeepThought"
    },
    "event": "logged_in",
    "fields": {
      "planet": "Magrathea",
      "service": "Answerer"
    },
    "id": 42,
    "input_type": "log",
    "offset": 0,
    "session_id": "91e5b9d",
    "source": "test/structured.log",
    "tags": [
      "i",
      "heart",
      "json"
    ],
    "type": "log",
    "user": "arthur",
    "verified": false
  }</pre>  <p>As you can see, Filebeat automatically adds a timestamp. Note that this is the time when the log line was read, which can be different from when the application wrote the log line. You can set up the structlog library to generate timestamps if you need better accuracy. </p> <p>Filebeat also automatically adds some metadata like the host name, and it makes it easy to add custom fields and tags via the configuration file. This means that the application doesn’t have to worry about adding metadata from the environment. </p>That’s about all you need. Simple things should be simple :-)

logs-woodpile-full.jpg

logs-woodpile.jpg

Structured logging with Filebeat

We’re happy to announce that the 1.2.2 bug fix release is now available for all the Elastic Beats: Filebeat, Packetbeat, Topbeat and Winlogbeat.

<html><head></head><body><p>We&#x2019;re happy to announce that the 1.2.2 bug fix release is now available for all the Elastic Beats: Filebeat, Packetbeat, Topbeat and Winlogbeat. You can download the new versions from the usual <a href="https://www.elastic.co/downloads/beats/">place</a>. </p> <h3>Bug fixes affecting all Beats</h3> <ul> <li>Fix race when multiple outputs access the same event with Logstash output manipulating event. <a href="https://github.com/elastic/beats/issues/1410">#1410</a></li> <li>Fix go-daemon (supervisor used in init scripts) hanging when executed over SSH. This was causing problems when running the init script via some of the deployment tools, notably Salt.&#xA0;<a href="https://github.com/elastic/beats/issues/1394">#1394</a></li> </ul> <h3>Bug fixes affecting Filebeat</h3> <ul> <li>Improvements in registrar dealing with file rotation. <a href="https://github.com/elastic/beats/issues/1281">#1281</a></li> </ul> <p><span></span>Many thanks to our dear community members who tested the Beats, reported and helped with the troubleshooting of the above issues. If you find any issues with the new release or there are features that you miss, please interact with us on <a href="https://discuss.elastic.co/c/beats">Discuss</a> or <a href="https://github.com/elastic/beats">GitHub</a>. </p></body></html>

Beats 1.2.2 Released

Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>New community Beat: JmxProxyBeat</h3> <p>Created by <a href="https://github.com/radoondas">Radovan Ondas</a>, who also wrote Apachebeat and Elasticbeat in the past, JmxProxyBeat gets JMX metrics from Tomcat via the JMX Proxy Servlet. Great companion for Metricbeat :-). </p> <h3>DNSSEC support in Packetbeat</h3> <p>DNS monitoring is already one of the most popular use cases for Packetbeat. Thanks to this <a href="https://github.com/elastic/beats/pull/1292">large community contribution</a>, Packetbeat is now also able to understand EDNS and DNSSEC resource records. </p> <h3>Clean shutdown procedure</h3> <p>It used to be that you couldn&#x2019;t stop the Beats if the output queue was not empty (for example, because Elasticsearch/Logstash was not available). After this fairly <a href="https://github.com/elastic/beats/pull/1402">complex change</a>, the Beats can signal to the publisher that they want to shutdown and that the outputs can drop the output queue if needed. Dropping the inflight messages is fine, because for the Beats that offer at-least-once-guarantees (Filebeat &amp; Winlogbeat), the registry makes sure they will be resent after the Beat restart. </p> <p>Besides solving the bug mention above, the clean shutdown semantics are one of the prerequisites for configuration reloading and running multiple Beats in a single binary. </p> <h3>Metricbeat, meet generic filtering</h3> <p>The new (since alpha1) libbeat functionality of generic filtering can now be used <a href="https://github.com/elastic/beats/pull/1384">at module level in Metricbeat</a>, making it possible to flexibly control which fields (metrics) are exported and which not. </p> <h3>Metricbeat system module</h3> <p>Metricbeat now borrows from Topbeat (every Beat is also a library!) the <a href="https://github.com/elastic/beats/pull/1416">system module</a>, so any host running it will report CPU/mem&#xA0;statistics. </p> <h3>Topbeat and Packetbeat now work on OpenBSD</h3> <p>We don&#x2019;t yet consider OpenBSD a supported platform for Beats, but thanks to the efforts of Jasper Lievisse Adriaanse, a <a href="https://github.com/jasperla" target="_blank">community member</a>, Topbeat and Packetbeat can now be compiled and executed on OpenBSD. Jasper went as far as fixing libpcap on OpenBSD and <a href="https://github.com/elastic/beats/pull/1456" target="_blank" rel="font-family: arial, sans-serif; font-size: 12.8px; line-height: 17.664px; background-color: rgb(255, 255, 255);">provide a vagrant box</a> to us so we can test OpenBSD more regularly. Filebeat was already working on BSDs via simple cross-compilation, because it&apos;s pure Go. </p> <h3>Load the Elasticsearch template on every connect</h3> <p>One of the known issues with 5.0.0-alpha1 was that the Beats only attempted to load the Elasticsearch template once, meaning that if they were started before Elasticsearch the template was not loaded. With this <a href="https://github.com/elastic/beats/pull/1381">change</a>, the template is loaded immediately after a successful connection is established. The error handling was also improved, so we can now make sure that we don&#x2019;t insert any documents before the template is loaded. </p> <h3>Filebeat&#x2019;s JSON decoder can now parse @timestamp fields</h3> <p>One of the bugs reported against alpha1 was that if a <code>@timestamp</code> field was supplied as JSON field, Filebeat would crash. This is because our output plugins expect the <code>@timestamp</code> field to have a particular type. Filebeat now <a href="https://github.com/elastic/beats/pull/1421">makes sure</a> that the timestamp and type (could also be affected by this) have the right types. Besides not crashing, a benefit of this is that the application can now provide its own timestamp&#xA0;to overwrite the one added by Filebeat. </p> <h3>Added path.logs option and log to files by default</h3> <p>The Beats used to log to syslog by default. With the introduction of the <a href="https://github.com/elastic/beats/issues/1371">directory layout</a>, and to be more similar with the other products in the stack, it makes more sense to log to files by default. So this <a href="https://github.com/elastic/beats/pull/1437">is now the default</a>. The files are automatically rotated and old files are removed. </p> <h3>When loading the default dashboards, you can now select the index name</h3> <p>Thanks to another contribution by <a href="https://github.com/radoondas">Radovan</a>, you can now rename the index names before loading the dashboards in Elasticsearch. </p> <h3>Other merges:</h3> <ul> <li><a href="https://github.com/elastic/beats/pull/1428">Fix crash</a> when using multiple outputs.</li> <li>Improved system testing for the <a href="https://github.com/elastic/beats/pull/1413">Metricbeat Apache module</a>.</li> <li>Added Packetbeat <a href="https://github.com/elastic/beats/pull/1424">NFS dashboards</a> to our sample dashboards.</li> <li>Metricbeat modules and metricsets must now be <a href="https://github.com/elastic/beats/pull/1426">unique</a>.</li> <li>The <a href="https://github.com/elastic/beats/pull/1435">confdir</a> option in Filebeat is now relative to the configuration path.</li> <li>Added a <a href="https://github.com/elastic/beats/pull/1446">beatname.sh</a> script to easily run the Beats when installed from DEB/RPM</li> <li><a href="https://github.com/elastic/beats/pull/1449">Upgraded</a> Go to 1.6.2</li> </ul></body></html>

Brewing in Beats: Tomcat JMX monitoring and DNSSEC monitoring

At almost precisely one year after the Packetbeat team joined Elastic, we’re excited to reveal the first alpha release of the next major version of the Beats.

<html><head></head><body><p>At almost precisely a year after the Packetbeat team has joined Elastic, we&#x2019;re excited to reveal the first alpha release of the Filebeat, Packetbeat, Topbeat, and Winlogbeat next major versions.
</p><ul>
	<li><a href="https://www.elastic.co/downloads/beats/">Download links</a> </li>
	<li><a href="https://www.elastic.co/guide/en/beats/libbeat/master/release-notes-5.0.0-alpha1.html">Release notes</a></li>
</ul><h2>One version to rule them all</h2><p>You might be wondering why we&#x2019;re jumping from version 1.2 directly to 5.0. To make our software suddenly more stable and to one-up our competition, of course. In seriousness, all the projects in the Elastic stack are doing releases in sync and will use the same version numbers from now on. As Kibana is currently at 4.5, we&#x2019;re all going with 5.0 as the next major.
</p><p>This is to avoid the support matrix from hell, for example now you need to know that Beats 1.2 were tested against Elasticsearch 2.3, Logstash 2.3 and Kibana 4.5. &#xA0;Starting with 5.0, you&#x2019;ll know that if Beats and Elasticsearch have the same version number, they were released at the same time and we have tested them together. It simplifies communication all around.
</p><h2>New Features</h2><p>Beats 5.0-alpha1 comes packed with new features and you can expect more of them to land in during the alpha and beta phases. Here are some of the highlights from Alpha 1:
</p><h3>Custom fields and generic filtering</h3><p>You now have more freedom over how the documents created by the Beats look like. On one hand, you can now add custom fields and tags per Beat and module. On the other hand, you can use the newly introduced <a href="https://www.elastic.co/guide/en/beats/packetbeat/master/filtering-data.html">generic filtering</a> to remove the fields that you don&#x2019;t want. These features are implemented at the libbeat level, meaning that all <a href="https://www.elastic.co/guide/en/beats/libbeat/master/community-beats.html">community Beats</a> automatically benefit from them as soon as they upgrade.
</p><h3>JSON support in Filebeat</h3><p>Filebeat can now natively <a href="https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#config-json">decode JSON </a>objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON. This can also be used as a convenient way of collecting logs from Docker hosts, because Docker uses JSON to wrap the log lines from the application.
</p><h3>Integration with Ingest Node</h3><p>The new <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html">Ingest Node</a> functionality, released with Elasticsearch 5.0.0-alpha1, is big news because it gives users processing capabilities similar with Logstash directly in Elasticsearch! This makes it really easy to get started with the Elastic stack. For simple logging usecases, for example, you only need Filebeat and Elasticsearch.
</p><p>All Beats can work with the Ingest Node, simply set the <a href="https://www.elastic.co/guide/en/beats/filebeat/master/configuring-ingest-node.html">pipeline parameter</a> in the Elasticsearch output configuration.
</p><h3>Packetbeat IP/TCP flows</h3><p>So far Packetbeat was focused on the application layer protocols, giving you visibility into the business transactions as seen in the network. Packetbeat now also reports statistics like packet count and byte count about IP and TCP flows, regardless of the upper layer protocols. This opens Packetbeat to a new set of use cases, giving insights into how the traffic is flowing through the network.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4d509985d58976378ac5/download?uid=blt801ffa0376df380c?uid=blt801ffa0376df380c" data-sys-asset-uid="bltf5249a33c94ec408" alt="Packetbeat Flows in Kibana 5" style="max-width:100%;">
</p><h3>Kafka output</h3><p>We listened to your feedback and we&#x2019;ve added Kafka output support in Beats, at the same time removing the deprecation mark for the Redis output. This means that if you are passing all messages through a Kafka queue anyway, you won&#x2019;t need a Logstash instance to convert between Beats and Kafka.
</p><h3>Winlogbeat improvements</h3><p>Winlogbeat now extracts all the fields from Windows event log records including the EventData and UserData fields and includes them in the documents it indexes. In addition, now it is possible to select events by event ID, level, and provider. Winlogbeat efficiently implements this event selection by using a query with Windows APIs so that only the requested events are returned.
</p><h2>Don&#x2019;t fear the alpha</h2>Getting your feedback early is key for us to make the necessary adjustments in time for the 5.0 GA release, so please test early and often.&#xA0;You can find us on <a href="https://discuss.elastic.co/c/beats/">discuss</a> for question and discussion and on <a href="https://github.com/elastic/beats">Github</a> for issues and enhancement requests.</body></html>

Screen Shot 2016-04-05 at 15.04.14.png

Beats 5.0.0-alpha1 released

​Welcome to Brewing in Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to

<html><head></head><body><p>Welcome to Brewing in Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>JSON support in Filebeat</h3> <p>Since merging this <a href="https://github.com/elastic/beats/pull/1143">PR</a> last week, Filebeat can natively decode JSON objects from log lines. This is useful for structured logging, where the logging library writes the metadata directly formatted as JSON. Of course, this was already possible with Logstash, but &#xA0;this enables people to take the direct Filebeat -&gt; Elasticsearch path when they have their logs in JSON already. </p> <p>Another interesting use case for the JSON decoding is that it can be used to ship the logs from a Docker host. When writing the logs to files, Docker wraps the log lines of the application in JSON to add some meta-data. Because Filebeat decodes the JSON before applying line filtering and multiline rules, it is able to unwrap the JSON and then apply these rules, so these features combine well in the Docker and other similar use cases. </p> <h3>Apachebeat and Redisbeat merged into Metricbeat</h3> <p>Radovan Ondas, the creator of <a href="https://github.com/radoondas/apachebeat">Apachebeat</a>, and Chris Black, the creator of <a href="https://github.com/chrsblck/redisbeat">Redisbeat</a>, have contributed modules for Apache and Redis to Metricbeat. It&#x2019;s a great sign for Metricbeat that the Beats community devs are embracing it. </p> <h3>Metricbeat progress</h3> <p>Speaking of Metricbeat, Nicolas continued to shape it over the last few weeks, adding a <a href="https://github.com/elastic/beats/pull/1108">developer guide</a>, a common way of handling <a href="https://github.com/elastic/beats/pull/1175">timeouts</a>, adding <a href="https://github.com/elastic/beats/pull/1185">host</a> metadata, <a href="https://github.com/elastic/beats/pull/1172">restructuring Topbeat</a> so its functionality can be shared in Metricbeat, adding <a href="https://github.com/elastic/beats/pull/1187">MySQL authentication</a>, and many others. </p> <h3>New 5.0 Elasticsearch templates</h3> <p>After making sure that Elasticsearch 5.0 is able to upgrade automatically the mapping templates used by the Beats, Adrien opened a <a href="https://github.com/elastic/beats/pull/1221">PR</a> to upgrade our templates with the the options accepted by 5.0. These are the templates that&#xA0;we will ship with&#xA0;Beats 5.0-alpha1. </p> <h3>Winlogbeat - select events by level, event_id, and provider</h3> <p>Winlogbeat is now able to <a href="https://github.com/elastic/beats/pull/1218">filter</a> <a href="https://github.com/elastic/beats/pull/1218">the events</a> by these key fields. It does this efficiently by adjusting its Windows API query to only return the events needed. The details are&#xA0;in the <a href="https://github.com/elastic/beats/pull/1218">PR</a>. </p> <h3>Packetbeat - Split real_ip_header to only have one value</h3>This fixes an <a href="https://github.com/elastic/beats/issues/1236">issue</a>, where Packetbeat&#x2019;s Geoip resolving didn&#x2019;t work if the X-Forwarder-For (or similar) header had multiple IP addresses inside. With this&#xA0;<a href="https://github.com/elastic/beats/pull/1241">fix</a>, Packetbeat takes the first IP address when there are multiple defined in the header. &#xA0;The &#xA0;fix will be available in 1.2.</body></html>

Brewing in Beats: JSON support in Filebeat

Welcome to Weekly Beats! With this series, we're keeping you up to date with all that's new in Beats, from the details of work in progress pull requests to rele

<html><head></head><body><p>Welcome to Weekly Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p><p>Merged this week:
</p><h3>DNS eTLD+1 support in Packetbeat</h3><p>As Andrew <a href="https://github.com/elastic/beats/pull/1107">wrote</a>, &#x201C;the effective top level domain plus one more label is really useful for clustering DNS requests (hostnames). For example, the eTLD+1 for <code>foo.bar.golang.org.</code>&#xA0;is <code>golang.org.</code>.&#xA0;This was the basis for the aggregations used in <a href="https://www.elastic.co/blog/detecting_dns_tunnels_with_packetbeat_and_watcher">Detecting DNS Tunnels with Packetbeat and Watcher</a>.&quot; The <a href="https://github.com/elastic/beats/pull/1107">PR</a> adds a <code style="line-height: 1.6em;">dns.question.etld_plus_one</code>field to the Packetbeat DNS transactions.</p><h3>SOCKS5 proxy for the connection to Logstash</h3><p><a href="https://github.com/elastic/beats/pull/1106">This</a> makes it possible for the Beats to connect to Logstash via SOCKS5 proxy. This can useful when sending the data between different Internet domains. It also opens the possibility of using username/password for authenticating to Logstash.
</p><h3>Cleaner separation between the Go unit tests and integration tests</h3><p>Our Go <code>_test.go</code>&#xA0;files contain both unit and integration tests (defined as tests that require other services). We used to separate them via the &#x201C;short&#x201D; flag on a test by test basis. This also meant that in order to run only the unit tests, one had to call <code>go test -short</code>. With this <a href="https://github.com/elastic/beats/pull/1073">change</a>, the unit and integration tests are strictly separated in different files and we use build flags to select between them. From now, to run only the unit tests you simply write <code>go test</code>. To execute the integration tests, you now need <code>go test -tags=integration</code>.
</p><h3>Fix Topbeat CPU time computation on Windows</h3><p><a href="https://github.com/elastic/gosigar/pull/19">This</a> avoids some floating point arithmetic when converting from a Windows structure to a 64 bits value. We&#x2019;re hoping that this fixes a <a href="https://github.com/elastic/beats/pull/1128">bug</a> where huge numbers were reported for CPU usage.
</p><h3>Track unavailability in Metricbeat</h3><p>If a monitor system is not available at all, an error document is <a href="https://github.com/elastic/beats/pull/1133">sent</a> to Elasticsearch so it is easier to track downtime.
</p><p><strong></strong>
</p><p>And from the currently in progress pull requests:
</p><h3>Generic event representation</h3><p>The way we currently represent internally an event in Beats is in the form of <code>map[string]interface{}</code>. Translating from Go, that means a map from string to anything (because anything implements the empty interface). The only requirement is that all types inside are JSON serializable, but then again, pretty much everything in Go is. This makes it quite easy to write new Beats (just throw anything have into a map), but it means that we have to rely on reflection when working with the event in libbeat, for example for the <a href="https://github.com/elastic/beats/pull/1120">generic filtering</a>. Reflection code is slow and error prone. Benchmarking also showed that the JSON serialization often dominates the performance of the Beats, and it is slow because it has to do reflection.
</p><p><strong></strong>
</p><p>So we want to move away from this &#x201C;anything goes&#x201D; events. For the transition phase (we don&#x2019;t have control over the community Beats) and for convenience later, we&#x2019;ve written <a href="https://github.com/elastic/beats/pull/1120/files#diff-689e10c72968be39c1f459321bc9a1eeR28">some code</a> that takes a <code>map[string]interface{}</code>&#xA0;event and uses reflection (no way around that) to transform it in a nested map that only contains a few accepted types. This makes the rest of the libbeat code easier, especially the filtering code.
</p><h3>JSON support in Filebeat</h3><p>We now have a <a href="https://github.com/elastic/beats/pull/1143">second proposa</a>l for implementing JSON support in Filebeat. The advantage of this one over the one I <a href="https://github.com/elastic/beats/pull/1069">linked</a> in last week&#x2019;s update is that you can combine it with multiline and line filtering in a more meaningful way. This makes it a good way from shipping logs from a Docker host, for example, while still being able to use multiline on the application logs.
</p><h3>Dashboards per Beat &amp; module</h3><p>We <a href="https://github.com/elastic/beats/pull/1109"><u>started</u></a> to move the Kibana configuration for the <a href="https://github.com/elastic/beats-dashboards"><u>sample dashboards</u></a> inside the main repo and organize it per Beat and even per module (in Metricbeat). Nicolas wrote Python scripts to split the dashboards into &#x201C;snippets&#x201D; and combine them back.
</p></body></html>

Brewing in Beats: SOCKS5 support when sending to Logstash

<html><head></head><body><p>Welcome to Weekly Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <p>Lots of positive energy in the team after Elastic{ON}, thank you everyone that attended and provided us with feedback. </p> <h3>Custom fields for all Beats</h3> <p>A common request was to add Filebeat-like custom fields to all other Beats. The tags we had in the common section of the configuration files just weren&#x2019;t enough. So <a href="https://github.com/elastic/beats/pull/1092">now</a> all Beats have fields <strong>and</strong>&#xA0;tags in the common section and it is also easy to add them per Beat-specific element (e.g. prospector in Filebeat, protocol in Packetbeat, etc.). </p> <h3>New Community Beats</h3> <p>More goodies from the community: </p> <ul> <li><a href="https://github.com/FStelzer/flowbeat">Flowbeat</a>: For collecting <a href="https://en.wikipedia.org/wiki/SFlow">sflow</a> data. </li> <li><a href="https://github.com/gravitational/udpbeat">Udpbeat</a>: Receives structured logs over UDP and sends them to Elasticsearch. The author wrote a <a href="http://blog.gravitational.com/experiment-with-elastic-beats-go-and-structured-logging-with-elasticsearch/">blog post</a> about how to use it to get trace data for Go errors.</li> <li><a href="https://github.com/colings86/batterybeat">Batterybeat</a>: Colin wrote a Beat to poll information about his Mac battery.</li> <li><a href="https://github.com/buehler/go-elastic-twitterbeat">Twitterbeat</a>: Polls the tweets of a preconfigured list of screen names and inserts them into Elasticsearch.</li> </ul> <h3>Better configuration file handling</h3> <p>Up to now we were using a generic YAML parser for reading our configuration files. This is very clean and convenient from the code perspective, but there are several things about it that annoyed us: </p> <ul> <li>Because YAML is context sensitive, it&#x2019;s difficult to delegate the parsing to another function. This makes it hard to have our modules self contained from the configuration point of view.</li> <li>YAML is very permissive, with almost any file being valid. Because we have no visibility into the unmatched values, it&#x2019;s hard for us to provide good error messages in case of configuration issues.</li> <li>Whitespace is just problematic when it comes to configuration files. People copy paste configuration examples, and it&#x2019;s enough to forget one blank character and nothing works. Add to that the point above about us not being able to provide good error message, and you can guess what&#x2019;s our number one source of Discuss and Github tickets.</li> </ul> <p>Those were the bad news. The good news is that Steffen is <a href="https://github.com/elastic/beats/pull/1042">working</a> on fixing them, and not only for the Beats but for other Go programs that use YAML as well. He is writing ucfg (universal configuration) that adds a layer above the YAML parser (or any other parser) that will eventually be able to do configuration validation, accepts dots in field names (so we don&#x2019;t have to always rely on white space) and make the configuration definitions pluggable. </p>  <p>This is already used for <a href="https://github.com/elastic/beats/pull/1078">making</a> the output modules in libbeat as well as the &#xA0;<a href="https://github.com/elastic/beats/pull/1101">Packetbeat modules</a> truly self contained. </p> <h3>Topbeat performance improvement</h3> <p>When monitoring a large number of processes, the cost of getting their command line on each poll was significant, especially on Windows. This PR fixes it by <a href="https://github.com/elastic/beats/pull/1052">caching</a> the command line strings. Telling benchmarks for Windows and OS X are in the PR description. </p> <h3>Improved shutdown logic in libbeat</h3> <p>Shutting down is never easy when dealing with lots of channels. McStork, a community contributor, stepped up and helped i<a href="https://github.com/elastic/beats/pull/1075">mproving the logic</a> we have for shutting down the publisher. </p> <h3>Metricbeat progress</h3> <p>Nicolas is making good progress on getting the Metricbeat infrastructure ready before adding more modules. This includes a reworking of the <a href="https://github.com/elastic/beats/pull/1049">configuration</a> handling and getting more metricsets in the existing <a href="https://github.com/elastic/beats/pull/1063">modules</a>, testing if <a href="https://github.com/elastic/beats/pull/1081">Topbeat could be included in Metricbeat</a> and starting the <a href="https://github.com/elastic/beats/pull/1108">contributor guide</a> to new Metricbeat modules. </p> <h3>Filebeat fix for duplicates on restarting</h3> <p>We discovered a <a href="https://github.com/elastic/beats/issues/1010">bug</a> that could cause Filebeat to re-read a complete file if it was restarted at the wrong time. It is now fixed and we&#x2019;re preparing a 1.1.2 release for next week to include this. </p> <h3>Other notable merges since the last update</h3> <ul> <li><a href="https://github.com/elastic/beats/pull/768">AMQP</a> merged in Packetbeat.</li> <li><a href="https://github.com/elastic/beats/pull/756">Flows</a> support merged in Packetbeat.</li> <li><a href="https://github.com/elastic/beats/pull/1008">Un-deprecate</a> Redis output.</li> <li>Added <a href="https://github.com/elastic/beats/pull/1051">FAQ pages</a> for each Beat. Here is an example such <a href="https://www.elastic.co/guide/en/beats/packetbeat/master/faq.html">page</a> for Packetbeat.</li> <li><a href="https://github.com/elastic/beats/pull/1065">Adjusted</a> the default Filebeat config to avoid a common configuration mistake. Details <a href="https://github.com/elastic/beats/issues/1033">here</a>.</li> <li>The <a href="https://github.com/elastic/beats/pull/1068">close_older</a> setting backported to the 1.2 branch.</li> <li>Dockerized <a href="https://github.com/elastic/beats/pull/1105">Kafka integration tests</a>.</li> </ul> <h3>And from the newly opened / in progress / discussion PR</h3> <ul> <li><a href="https://github.com/elastic/beats/pull/1069">JSON support </a>in Filebeat.</li> <li><a href="https://github.com/elastic/beats/pull/1106">SOCKS5 support </a>for the connection to Logstash.</li> <li><a href="https://github.com/elastic/beats/pull/1118">DRDA</a> protocol in Packetbeat (used by DB2 and Apache Derby).</li> <li><a href="https://github.com/elastic/beats/pull/959">FCGI</a> protocol in Packetbeat.</li> </ul></body></html>

Brewing in Beats: New community Beats, custom fields, performance improvements

<html><head></head><body><p>Welcome to Weekly Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <p>Quite a few things going on while we&#x2019;re working on the next major and preparing for Elastic{ON}. This week also brings us a new community Beat. </p> <h3>Beat generator: template for creating a new Beat</h3> <p>We started <a href="https://github.com/elastic/beat-generator">beat-generator</a>, a new project that makes it a lot easier to start a new Beat. One command and you get all the boiler code required, the Makefile to build it, a correct Go dependency setup and a way to keep up to date with the libbeat changes (<code>make update</code>;). We hope to not only make it easier to start new Beats, but also to have a more unified dev process around the community Beats. &#xA0; </p> <p><strong></strong> </p> <p>If you plan to create a new Beat, we recommend using it already. </p> <h3>New community Beat: Elasticbeat</h3> <p>It was bound to happen, a community Beat to monitor Elasticsearch: <a href="https://github.com/radoondas/elasticbeat">Elasticbeat</a>. We&#x2019;ll be following this one for sure. </p> <h3>Improved system testing</h3> <p>Also on the theme of making it easier for the community Beats creators, the system tests were refactored to avoid duplication and to be importable from new Beats. In addition, they were improved to automatically fail on panics in the logs or on non-zero exit code from the Beat. Special thanks go to community contributor <a href="https://github.com/cyrilleverrier">Cyrille Verrier</a>, who proposed several improvements and helped with implementing them. </p> <h3>Filebeat: introduced close_older setting</h3> <p>The ignore_older setting in Filebeat used to do <a href="https://github.com/elastic/filebeat/issues/181">two things</a>, which used to cause issues because sometimes there was no single value convenient for both. This <a href="https://github.com/elastic/beats/pull/718">PR</a> splits them and also changes the defaults. ignore_older is now disabled by default and close_older is set to one hour.  </p> <h3>Topbeat: added support for capturing the full command line</h3> <p>Topbeat used to only collect the process name, now it captures the full <a href="https://github.com/elastic/beats/pull/749">command line</a>, which was an often requested feature request. This also works on Windows! </p> <h3>Performance improvements in the publisher pipeline</h3> <p>We&#x2019;ve talked in the past about the performance issues we&#x2019;re having on the &#x201C;at-least-once&#x201D; communication between Filebeat &#xA0;and Logstash. It&#x2019;s a fairly difficult issue because the Beat needs to balance between sending large batches fast to increase throughput and being able to reduce the output when Logstash is busy. We refactored the code and <a href="https://github.com/elastic/beats/pull/782">made it possible for Filebeat to process the next batch</a> while waiting for the current one to be ACKed.  </p> <p><strong></strong> </p> <p>This helps with the overall throughput by enabling load balancing between multiple output threads, at the cost of memory usage. With the right settings and enough memory and CPU power, we&#x2019;ve seen Filebeat pushing around 45K events/s, compared to around 18K before this change. </p> <h3>Freebsd and Solaris are now part of our CI</h3> <p>Jenkins now runs the tests on these two new platforms, which is the first step in supporting them. All tests from Filebeat and Packetbeat are passing already. Topbeat would require more work. </p> <h3>Work in progress on generic filtering in libbeat</h3> <p>Monica is making <a href="https://github.com/elastic/beats/pull/830">progress</a> on the generic filtering feature in libbeat, from which all Beats are expected to benefit. For the moment it is possible to filter fields from a generic event, which already covers a lot of the feature requests that we received. </p> <h3>Work in progress on Packetbeat flows</h3> <p>Steffen <a href="https://github.com/elastic/beats/issues/670">started working</a> on a very promising Packetbeat feature, ability to extract information about TCP/UDP/TLS flows for which we don&#x2019;t understand the upper layers. This should open a new set of possible use cases for Packetbeat. </p></body></html>

Brewing in Beats: Beat generator

<html><head></head><body><p>Welcome to Weekly Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources. </p> <h3>Unifiedbeat - Beat for IDS/IPS event logs </h3> <p><a href="https://github.com/cleesmith/unifiedbeat">Unifiedbeat</a> reads records from <a href="http://manual.snort.org/node44.html">Unified2</a> binary files generated by network intrusion detection software like Snort or Suricata, and indexes the records in Elasticsearch. Its author wrote a <a href="https://medium.com/@cleesmith/unifiedbeat-update-43fc00dda4ad#.qg0wrxyw9">blog post</a> in which he explains why Elasticsearch is a great option for storing these logs. We agree and think this is a great use of the Beats framework. </p> <p><strong></strong> </p> <h3>Factbeat - Beat for Puppet Facter info  </h3><p><a href="https://github.com/jarpy">Factbeat</a> is a new community Beat, created by <a href="https://github.com/jarpy">Toby McLaughlin</a> from Elastic. It runs Facter periodically and sends the results to Elasticsearch. Having all the facts about your servers available in&#xA0;Elasticsearch makes it easy to query and visualize your infrastructure in new and interesting ways. </p> <p><strong></strong> </p> <h3>Expand env variables in configuration files  </h3><p>A commonly requested feature was to be able to use environment variables for configuring the Beats. Andrew came up with an elegant <a href="https://github.com/elastic/beats/pull/715">solution</a> that allows you to use environment variables in the configuration files, while allowing for default values. This works automatically for all Beats. </p> <p><strong></strong> </p> <h3>Winlogbeat new field names  </h3><p>In time for the first release of Winlogbeat, we&#x2019;ve <a href="https://github.com/elastic/beats/pull/689">renamed</a> the fields exported by Winlogbeat to be more uniform with the ones exported by the other Beats. </p> <p><strong></strong> </p> <h3>Filebeat refactoring  </h3><p>We&#x2019;ve made more <a href="https://github.com/elastic/beats/pull/720">cleanups</a> to the Filebeat code, making it possible to cleanly shutdown Filebeat in all the corner cases. </p> <p><strong></strong> </p> <h3>PowerShell script for loading the dashboards  </h3><p>Thanks to a <a href="https://github.com/dovicn">community contributor</a>, our next release will include a <a href="https://github.com/elastic/beats-dashboards/pull/68">PowerShell script</a> that makes it easy to load our sample dashboards on Windows. </p> <p><strong></strong> </p> <h3>Testing environment  </h3><p>We now have a dockerized <a href="https://github.com/elastic/beats/pull/682"><u>test environment</u></a> for the manual QA phase. This makes it very easy to test against specific versions of Elasticsearch, Logstash, and Kibana. </p></body></html>

Brewing in Beats: More community Beats

There’s a new community Beat, written by Jay Greenberg, and it’s really interesting. Nagioscheckbeat runs Nagios checks and sends the results to Elasticsearch.

<html><head></head><body><p>Welcome to <em>Weekly Beats</em>! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from the details of work in progress pull requests to releases and learning resources.
</p><h3>New community Beat: Nagioscheckbeat</h3><p>There&#x2019;s a new community Beat, written by Elastic&#x2019;s <a href="https://github.com/PhaedrusTheGreek">Jay Greenberg</a>, and it&#x2019;s really interesting. <a href="https://github.com/PhaedrusTheGreek/nagioscheckbeat">Nagioscheckbeat</a> runs Nagios checks and sends the results to Elasticsearch. I don&#x2019;t think it was ever easier than this to scale and get analytics insights from all those Nagios scripts. &#xA0;Huge potential here, if you ask me. Thank you, Jay :-)
</p><h3>New community Beat: hsbeat (HotSpot VM metrics)</h3><p><a href="https://github.com/YaSuenag/hsbeat">Hsbeat</a> is another very interesting and promising community Beat. It collects performance data from the Java HotSpot VM by reading and decoding the binary performance logs created by the VM.
</p><h3>Filebeat to Logstash throughput improvements</h3><p>Steffen continued to investigate the <a href="https://github.com/elastic/beats/issues/587">issue</a> about low throughput performance when sending synchronously to Logstash, and <a href="https://github.com/elastic/beats/pull/628">adjusted the defaults</a> for a significant (9x) increase in the Filebeat -&gt; LS throughput.
</p><p><strong></strong>
</p><p>These new libbeat defaults are at odds with the maximum memory usage of Packetbeat due to the multiplication of buffers (buffer size * bulk size), so there is another <a href="https://github.com/elastic/beats/pull/649">change</a> that solves this issue by making the publisher know if the&#xA0;message contains one event or multiple (possibly thousands). This also improves the configurability of these buffers from the configuration file.
</p><p><strong></strong>
</p><p>With these changes, Filebeat is at around 60% of the throughput achieved by the Logstash Forwarder. The difference seems to come from the JSON serialization, which we need for creating new features easier, and from making the protocol safer for&#xA0;back pressure situations. Steffen has ideas for closing and even reversing this gap, but they require larger code changes.
</p><h3>Metricbeat (name not final) design started</h3><p>Nicolas put together a <a href="https://github.com/elastic/beats/pull/611">prototype-proposal</a>, which stirred up very pragmatic conversations about what we want to achieve with Metricbeat, how it relates to other Beats (especially the existing community beats), and even how the implementation will look like. If you want to join the discussion, you can do so in this <a href="https://github.com/elastic/beats/issues/619">ticket</a>, which also contains our current vision for Metricbeat.
</p><h3>Unified release process</h3><p>To improve the way all Elastic projects are being built, tested, and released, a common interface needs to be implemented by all of them. We started <a href="https://github.com/elastic/beats/issues/633">executing</a> on this interface, which included moving the <a href="https://github.com/elastic/beats-packer">beats-packer</a> code into the main beats repository.
</p><h3>Topbeat exports total CPU times instead of user CPU times</h3><p>Topbeat used to export user times&#xA0;for each process, but the Linux top command shows total times, which are arguably more important. So we <a href="https://github.com/elastic/beats/pull/641">adjusted</a> topbeat to export total times instead.
</p> <span></span></body></html>

Brewing in Beats: new community Beat for Nagios checks

<html><head></head><body><p>Today we are releasing a patch release for the Beats, fixing some of the issues reported since the 1.0.0 release.
</p><p><a href="https://www.elastic.co/downloads/beats/">Download links for Topbeat, Packetbeat and Filebeat</a>
</p><p><a href="https://www.elastic.co/guide/en/beats/libbeat/current/release-notes.html#release-notes-1.0.1">Release notes</a>
</p><h3>Changed the default bulk size for the communication with Logstash</h3><p>The previous default was quite high (10000), which worked good in normal operations but could cause a significant increase in memory usage when Logstash was temporary unavailable. We have reduced the default to 200, which worked good in the real world installations that we tried it on. The setting can be changed from the <code>bulk_max_size</code> option from the logstash setting in the configuration of any of the Beats.
</p><h3>Improved the force_close_files configuration setting</h3><p>The <code>force_close_files</code> settings exists in Filebeat to deal with the way Windows still shows deleted files in the file system until they are closed by all the processes that have them open. We tweaked its behavior a little to better detect when files are rotated on Windows. See this <a href="https://github.com/elastic/filebeat/issues/93">ticket</a> for the discussion.
</p><h3>Fixes and performance improvements for the Redis and MongoDB parsers</h3>Several performance issues and crashes in the Redis and MongoDB protocol implementation in Packetbeat have been solved. Please see the <a href="https://www.elastic.co/guide/en/beats/libbeat/current/release-notes.html#release-notes-1.0.1">release notes</a> for the links to the relevant tickets and pull requests.</body></html>

Beats 1.0.1 released

Why and how we moved the Beats open source projects into a single Git repository.

<html><head></head><body><p><img src="undefined" data-sys-asset-uid="blt71d918e8c11b4288" alt="keep-calm-we-are-only-moving-in-together-2.png" style="float: right; width: 350px; margin:0 0 20px 20px;">
</p>
<p>A time comes in the life of many development teams when they consider switching from multiple source code repositories to one (or the other way around). For the team behind the Beats open source projects, this time was a few weeks ago, and it resulted in us merging <a href="https://github.com/elastic/libbeat">libbeat</a>, <a href="https://github.com/elastic/filebeat">Filebeat</a>, <a href="https://github.com/elastic/topbeat">Topbeat</a>, <a href="https://github.com/elastic/winlogbeat">Winlogbeat</a> and <a href="https://github.com/elastic/beats">Packetbeat</a> into a single git repository last week.
</p>
<h2>The Why</h2>
<p>We were all feeling for some time that having to constantly keep our eight or so repositories in sync was time consuming, but I personally thought about it more as a necessary annoyance that we can live with, rather than something we must fix in the short term.
</p>
<p>So when someone casually brought up the idea during an internal chat, I didn&#x2019;t think the cost of switching would be worth it. For one thing, with the Beats being open source, the way we organize the repositories is not just our internal business, but it is part of the way we communicate with the wider developer community around the projects. I was worried that a single repo would make the individual Beats feel less independent and might discourage folks from creating their own Beat under their personal Github account.
</p>
<p>Nevertheless, we started to gather in a document the pros and cons and we payed more attention to all the tasks that we have to do because of the multiple repos. Small things like bumping the version number in the docs before releasing, creating new Github labels, closing and merging the changelog files, &#xA0;publishing the Github release, or just checking the open issues across repositories take a significant amount of time when you have to do them four times.
</p>
<p>Yes, most of this stuff can be automated, and indeed we had various scripts and tools that helped us a lot. But even then, there was a lot of time wasted, and creating these tools also didn&#x2019;t come for free. We do everything via pull requests and we review every single one, no matter how trivial. We also don&#x2019;t merge them until the continuous integration systems give the green light. So, in the last few days before a release all of the tiny changes added up and were taking forever.
</p>
<p>Then there were the tasks that are not so easy to automate, like backporting features or bug fixes between release branches. If the feature or bug affected more than one repository, the tedious and error-prone task of solving rebase conflicts had to be done more than once.
</p>
<p>Another important aspect for us was how friendly we were to the occasional external contributor. If someone wanted to fix a bug in Packetbeat but the code was actually in libbeat, they would first have to find the code in a different repository. Then after fixing it, figure out how to update the libbeat code into Packetbeat (godep), test it, and open pull requests in both libbeat and Packetbeat. A libbeat change can potentially break the tests in another Beat as well, and we could not expect an occasional contributor to clone three more repositories and run the tests before submitting the PR.
</p>
<p>After debating these points, we all agreed that a single repository would help us move faster and with fewer errors.
</p>
<h2>The How</h2>
<p>We thought that switching would be easy, because except for Packetbeat, all the other repositories are only one to six months old. But even for our young projects, there were quite a few things to consider.
</p>
<h3>Timing</h3>
<p>The more in-progress work you have at the time of the migration, the bigger the disruption is, so we chose to do the migration in the week after a major release, when most new projects were still in the design phase.
</p>
<h3>Git history</h3>
<p>We felt keeping the Git history was important, both to credit our external contributors and for us to be able to track down changes. Luckily the <a href="http://stackoverflow.com/a/14992078/15685">git subtree add</a> command makes this extremely easy. We imported the code from each Beat into a subdirectory in the final repository.
</p>
<p>One special requirement we had here was to have both the master and our latest release branch 1.0.0 available, so we actually did the subtree import twice, once for master and once for the 1.0.0 branch.
</p>
<p>The resulting graph is messy for sure, but you can clearly see the four repositories merged into one and the 1.0.0 branch being spawned just before the merge.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cf343e9adc538714bba/download?uid=blt703fc8cb5aa43308?uid=blt703fc8cb5aa43308" data-sys-asset-uid="blt18fd91bf7b5cb98c" alt="Screen Shot 2015-12-10 at 23.49.52.png" style="max-width:100%;">
</p>
<h3>Issues and links</h3>
<p>We also had to keep in mind that GitHub issues cannot be moved cleanly between repositories. There are scrpts out there that can close the original issues and create copies with the same content in the new repository, but the new issues will all be created by the same username. Links between issues, as well as links from commit messages to issues, get broken.
</p>
<p>For these reasons, we decided to keep the old repos around and only replace their README files with a move notice. We didn&#x2019;t use a script to migrate the issues. Rather, because there weren&apos;t many issues, we moved the open tickets one by one by hand. This gave us an opportunity to clean house.
</p>
<p>To minimize the negative effects of the migration, we kept and renamed the repository that had the most history (Packetbeat) and merged the other repositories into it.
</p>
<h3>Open Pull Requests</h3>
<p>You can&#x2019;t take over pull requests from one repository to another, so before the cutoff moment, we went through all open pull requests and either worked with the author to get them merged or closed them. Thank you everyone who helped with this effort.
</p>
<h3>Dependencies</h3>
<p>This one is fairly specific to Go. We vendor our dependencies in the source tree, which means that before the migration we had a dependencies folder (Godep) in each of the repositories. Because taking over that structure would negate many of the benefits of the single repository, we decided to switch at the same time to use the <a href="https://docs.google.com/document/d/1Bz5-UB7g2uPBdOx-rw5t9MxJwkfpx90cqG9AFL0JAYo/edit">Go 1.5 vendor experiment</a> and put all our dependencies in a single top-level vendor folder.
</p>
<p>We started using <a href="https://github.com/Masterminds/glide">glide</a> to manage this repository, and the transition from godeps was fairly simple. We prepared the glide.yml file in advance in an experimental merged repository, so at the cutoff time, we only had to copy the glide.yml file and use glide to create the vendor folder.
</p>
<h3>Integration tests and build systems</h3>
<p>These systems depend on the repository structure, so they needed configuration adjustments. In our environment, we are using <a href="http://travis-ci.org">Travis CI</a>, <a href="http://www.appveyor.com/">Appveyor</a> and our own <a href="http://build.elastic.co/">Jenkins</a> instance to provide test coverage over the different platforms we support. Preparation was key here again. In our experimental merged repository, we prepared new travis.yml and appveyor.yml files that we could simply copy over at the cutoff time.
</p>
<p>One negative side effect (and the biggest so far) of the migration is that these systems now have to run the tests from all the Beats on every change, making them slower in giving us feedback. This is especially a problem with Travis, which immediately after the cutover had to run <a href="https://travis-ci.org/elastic/beats/builds/94352929">11 different targets</a>, some of them taking quite long. Because some of the tests use Docker, it means all the tests have to run on a non-containerised system, further adding to the wait time.
</p>
<p>To mitigate this, we moved the most time-consuming Travis targets to be executed by Jenkins only, and we&#x2019;ll continue to work on making our tests faster.
</p>
<h2>What does this mean for you?</h2>
<p>If you are using one of the Beats, not much changes for you. Our next release will be built from the merged repository, but hopefully this won&#x2019;t be noticeable to you at all. Please post new issues to the new <a href="https://github.com/elastic/beats">Github repository</a>, and if you have open issues on the old repositories, please help us by moving them to the new one.
</p>
<p>If you are maintaining a Beat or other project that depends on libbeat, we left the old libbeat repository alive, so we don&#x2019;t break any existing code. However, we recommend you update your import paths to use the new libbeat path: github.com/elastic/beats/libbeat. Other than that, nothing really changes. You should continue to maintain your Beat under your GitHub account and we&#x2019;ll continue to support you in maintaining it to the best of our abilities.
</p>We apologize if this caused / will cause any issues to you. It helps us move the Beats platform forward faster and safer.</body></html>

keep-calm-we-are-only.png

The Beats are Moving Together into a Single Git Repository

<html><head></head><body><p><em>Welcome to Weekly Beats! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a></em><em>, from the details of work in progress pull requests to releases and learning resources.</em> </p> <p>The past week we did a reorg of our Github repositories and worked on the design for a few key features we&#x2019;re planning for the next future releases. </p>  <h3>Repository merging</h3> <p>We merged libbeat and the four officially supported beats into a single repository: <a href="https://github.com/elastic/beats"></a><a href="https://github.com/elastic/beats">https://github.com/elastic/beats</a>. We&#x2019;ll have a blog post detailing the reasons, but we essentially found a single repo to result in a lot less work for us but also be more efficient in terms of communication with the rest of the community. </p> <h3>Filebeat filtering</h3> <p>One of the most often requested features for Filebeat is to be able to do basic line filtering before sending to Logstash. Think of it as doing a &#x201C;egrep -v&#x201D; to filter out, for example, debug messages before sending the logs out. If you were previously dropping those lines in Logstash, there are now two handy options in Filebeat that can save you network traffic and CPU load: include_lines and exclude_lines. </p> <p>All the details are in the <a href="https://github.com/elastic/filebeat/issues/78">Github issue</a> and the <a href="https://github.com/elastic/beats/pull/430">Pull Request</a>. </p> <h3>Design for generic filtering</h3> <p>In addition to the simple filtering described above, we started thinking also about a more generic way to filter events or fields, from which all Beats will benefit. The discussion is happening in this <a href="https://github.com/elastic/beats/issues/451">issue</a> and you are invited! </p> <h3>Design for multiline</h3> <p>Another Frequently Requested Feature for Filebeat is about being able to merge related lines (like the ones from an exception, for example) into a single event. This can be done in Logstash already with the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html">multiline codec</a>, but doing this closer to where the files are read might make it more reliable for some people and easier to configure and maintain for others. This is also in the design phase, see this <a href="https://github.com/elastic/filebeat/issues/301">GitHub issue</a>. Once again, your input here would be valuable. </p> <h3>Design for Packetbeat&#x2019;s HTTP body handling</h3> <p>Another interesting <a href="https://github.com/elastic/beats/issues/404">issue</a> on which design is discussed is about making Packetbeat be able to understand the application specific HTTP payload. For example, if we know that the body is JSON created by an Elasticsearch client or server, we could extract some more interesting information, like the index name, the search query, the number of results, etc. The same can be done, and with relatively little effort, for any protocol that works over http, like CouchDB, Docker&#x2019;s protocol, XML-RPC, JSON-RPC, etc. </p> <h3>Apachebeat</h3> <p>We are very happy to see another Beat created by the community: <a href="https://github.com/radoondas/apachebeat">Apachebeat</a> created by <a href="https://github.com/radoondas">Radovan Ondas</a>. Similar to the existing <a href="https://github.com/mrkschan/nginxbeat">Nginxbeat</a>, Apachebeat can be used to insert the key Apache metrics into Elasticsearch. </p> <h3>Connect to Elasticsearch over proxies</h3> <p>It&#x2019;s now <a href="https://github.com/elastic/libbeat/pull/342">possible</a> for all Beats to also use an HTTP proxy when connecting to Elasticsearch. </p> <h3>Packetbeat performance improvements</h3> <p>Steffen has been profiling and improving the performance of the <a href="https://github.com/elastic/beats/pull/442">Redis</a> and <a href="https://github.com/elastic/beats/pull/453">Http</a> parsers in Packetbeat. These are still in progress and being discussed. </p> <h3>Winlogbeat</h3> <p>Winlogbeat is slowly getting closer to being releasable, and now it also has his own <a href="https://github.com/elastic/winlogbeat/pull/15">docs</a>. </p></body></html>

Brewing in Beats: The Git repository merge

<html><head></head><body><p>Welcome to <i>Weekly Beats</i>! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats"><span class="s1">Beats</span></a>, from the details of work in progress pull requests to releases and learning resources.</p><p class="p3">In the past week we released 1.0.0-rc2 and we are now ready for our first generally available&#xA0;release!</p><p class="p3"><strong>Filebeat fixes</strong></p><p class="p3">Two more important bugs were reported, tracked down, cornered, and fixed.</p><p class="p3"><span class="s1"><a href="https://github.com/elastic/filebeat/issues/258">One</a></span> was about lines above 1023 character potentially getting truncated due to a buffer that was <a href="https://github.com/elastic/filebeat/pull/259"><span class="s1">unintentionally reused</span></a>.</p><p class="p3">The <a href="https://discuss.elastic.co/t/err-file-reading-error-stopping-harvester-error-eof/34764/2"><span class="s1">other</span></a> could cause Filebeat to get stuck in case of quickly rotating files and we&#x2019;ve got reports for it <a href="https://discuss.elastic.co/t/filebeat-do-not-see-file-updates-after-a-while/34914/7"><span class="s1">several</span></a> <a href="https://github.com/elastic/filebeat/issues/270"><span class="s1">times</span></a>, so we are quite happy that we could <a href="https://github.com/elastic/filebeat/pull/261"><span class="s1">reproduce it and fix it</span></a> before the GA.</p><p class="p3"><strong>Fixed error handling in Elasticsearch output</strong></p><p class="p3">We found an <a href="https://github.com/elastic/libbeat/pull/309"><span class="s1">issue</span></a> with the error handling code of the response from Elasticsearch. The code assumed the &#x201C;error&#x201D; key to be of type string, which is no longer true in Elasticsearch 2.0. The <a href="https://github.com/elastic/libbeat/pull/310"><span class="s1">fix</span></a> came just in time for RC2.</p><p class="p3"><strong>Copy beat.hostname into host in the Logstash beats plugin</strong></p><p class="p3">The beats use the &#x201C;beat.hostname&#x201D; for the hostname where the Beat is running, but other Logstash plugins typically use the &#x201C;host&#x201D; field for the same thing. This caused annoyance to users, so we listened and found the compromise of having the Logstash Beats plugin automatically <a href="https://github.com/logstash-plugins/logstash-input-beats/pull/28"><span class="s1">copy &#x201C;beat.hostname&#x201D; into &#x201C;host&#x201D;</span></a>.</p><p class="p3"><strong>System tests for Winlogbeat</strong></p><p class="p3">The Winlogbeat (ICYMI, the Beat for Windows events logs) got <a href="https://github.com/elastic/winlogbeat/pull/6"><span class="s1">python system tests</span></a> and i<a href="https://github.com/elastic/winlogbeat/pull/8"><span class="s1">ntegration tests</span></a> to mirror the automated tests for our other officially supported Beats.</p><p class="p3"><strong>Topbeat on RaspberryPi</strong></p><p class="p3">Our own Robin Clarke played with cross-compiling Topbeat for ARM and tried it on his RaspberryPi. It just worked :-). Not quite an official supported platform for the Beats yet, but it&#x2019;s good to know that&#x2019;s well within reach.</p><p class="p3"><strong>Release fever</strong></p><p class="p3">We&#x2019;re doing the last bits of work in preparation for the GA release. Stay tuned!</p></body></html>

Brewing in Beats: Last step before the GA

<html><head></head><body><p><span style="line-height: 22.4px;">Welcome to&#xA0;</span><em style="line-height: 22.4px;">Weekly Beats</em><span style="line-height: 22.4px;">! With this series, we&apos;re keeping you up to date with all that&apos;s new in&#xA0;</span><a href="https://www.elastic.co/products/beats" style="line-height: 22.4px; background-color: rgb(255, 255, 255);">Beats</a><span style="line-height: 22.4px;">, from&#xA0;the details of work in progress pull requests to&#xA0;releases&#xA0;and learning resources.</span><br>
</p><blockquote class="twitter-tweet" lang="en">Beats 1.0.0-beta4 release brings you Filebeat: the lightweight log shipper successor to Logstash-Forwarder: <a href="https://t.co/sHipvzHREb">https://t.co/sHipvzHREb</a><br>&#x2014; elastic (@elastic) <a href="https://twitter.com/elastic/status/657243140689166336">October 22, 2015</a></blockquote><script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script><p>ICYMI, we <a href="https://www.elastic.co/blog/beats-beta4-filebeat-lightweight-log-forwarding"><span class="s1">released 1.0.0-beta4</span></a> on Thursday.&#xA0; The clear highlight of the release is the first version of FIlebeat, the lightweight log shipper successor of the Logstash Forwarder. Thank you everyone that helped testing the internal candidate, we caught at least one pretty bad issue and we got a ton of input on how we can improve the docs.
</p><p class="p2">Most of the work this week went into preparing and testing the&#xA0;release.<br>
</p><p class="p2">Other noteworthy things:
</p><p class="p1"><strong>Configurable file encoding in Filebeat</strong>
</p><p class="p1">Filebeat can <a href="https://github.com/elastic/filebeat/pull/101"><span class="s1">now be configured</span></a> on a per-glob basis to expect a particular encoding. This wasn&#x2019;t a problem for the Logstash Forwarder because its serialization protocol was encoding agnostic and the user could configure a codec in Logstash. It was a problem for Filebeat because (1) it uses JSON for communicating with Logstash and (2) it can send the data directly to Elasticsearch. With Windows using UCS-2 by default, it was important for us to have a solution for the beta4.
</p><p class="p1">Even if this means an extra configuration parameter for Filebeat, we expect that configuring the encoding close to where the files are is better for users, so it is likely a plus overall.
</p><p class="p1"><strong>Redis output deprecated</strong>
</p><p class="p1">With the beta4 release, which makes it possible for all Beats to send data to Logstash without any intermediaries, we <a href="https://github.com/elastic/libbeat/pull/199"><span class="s1">deprecated</span></a> the Redis output. See this <a href="https://github.com/elastic/filebeat/issues/132"><span class="s1">issue</span></a>, where the rationale behind this decision is further explained.
</p><p class="p1"><strong>Lumberjack code copied in the Beats input (Logstash)</strong>
</p><p class="p1">The Lumberjack code -- used by both the new Beats input plugin and the Lumberjack input plugins in Logstash -- used to live in its own library, even though fairly different code paths were taken depending on the plugin in use. This caused some dependency conflicts at install time. To fix this, and also to gives the beats-input plugin more freedom in changes to the protocol, the protocol code <a href="https://github.com/logstash-plugins/logstash-input-beats/pull/2"><span class="s1">was moved</span></a> in the plugin repo.&#xA0;
</p><p class="p1">Thanks to this change, installing the beats input is now as easy as `./bin/plugin install logstash-input-beats`.&#xA0;
</p><p class="p1">With the next Logstash release, this step won&#x2019;t be required at all as the plugin will be bundled in the default distribution.
</p><p class="p1"><strong>Documentation effort</strong>
</p><p class="p1">We continued our push for improving the docs, with everyone in the team contributing. We now have a professional editor&#xA0;helping with this, so we expect the docs quality to be rapidly improved.
</p><p class="p1">Related to the documentation effort, we have updated the sample configuration files to contain a lot more comments describing the available options.
</p><p class="p1"><strong>Always use UTC</strong>
</p><p class="p1">We found and fixed <a href="https://github.com/elastic/libbeat/issues/81"><span class="s1">a case</span></a> where the index names that we create (e.g. [packebeat-]YYYY.MM.DD) was not using the UTC timezone.
</p><p class="p1"><strong>TLS integration tests</strong>
</p><p class="p1">We now have <a href="https://github.com/elastic/libbeat/pull/195"><span class="s1">automatic tests</span></a> for the Filebeat -&gt; Logstash communication also for the TLS enabled case, including server certificate validation. Yay for one less thing to manually check and worry about.
</p><p class="p1"><strong>Nightlies and automatizing the packaging tests</strong>
</p><p class="p1">The listing on the S3 bucket serving the <a href="https://beats-nightlies.s3.amazonaws.com/index.html"><span class="s1">beats nightlies</span></a> is now easier to browse thanks to a new JS listing snippet. More importantly, the last built artifacts <a href="https://github.com/elastic/beats-packer/pull/12"><span class="s1">now use &#x201C;latest&#x201D;</span></a> as their version and show up first on each page, so it is now easier to download the most recent build.
</p><p class="p1">We have an ongoing effort to execute the packaging tests daily on a bare metal machine (has to be bare metal because it makes use of Virtualbox machines), so having predictable download URLs was needed.
</p></body></html>

Brewing in Beats: First Filebeat release

We just released Beats version 1.0.0-beta4. It contains improvements for Packetbeat and Topbeat but also something new: the first version of Filebeat.

<p><img src="https://api.contentstack.io/v2/assets/575e4cc643e9adc53871445b/download?uid=bltd65877eccb1d7d89?uid=bltd65877eccb1d7d89" data-sys-asset-uid="bltb99fdc35433e352a" alt="8777550156_b978fefa34_z.jpg" style="float: right; padding: 0 0 10px 10px;">We just released Beats version 1.0.0-beta4. It contains improvements for <a href="https://www.elastic.co/downloads/beats/packetbeat">Packetbeat</a> and <a href="https://www.elastic.co/downloads/beats/topbeat">Topbeat</a> but also something new: the first version of Filebeat, our new open source log forwarder that ships to Logstash or Elasticsearch.
</p><p>The goal of Filebeat is to tail logs and to ship them off servers to a central location for further processing. It is very lightweight in terms of consumed resources and has no dependencies or plugins to manage.<span></span>
</p><h2>Based on the Logstash Forwarder code base</h2><p>While Filebeat is a new project, significant parts of its code base are based on the <a href="https://github.com/elastic/logstash-forwarder">Logstash Forwarder</a> project, which has been&nbsp;used in production by many companies for years.
</p><p>Because Logstash Forwarder and Logstash don’t share any code and they are written in different programming languages (Go and Ruby), Logstash Forwarder was unfortunately neglected for too long by the dev community around Logstash and tended to lag behind in terms of improvements and bug fixes. A few community forks have been created, for example the <a href="https://github.com/elastic/logstash-forwarder">log-courier</a> and this <a href="https://github.com/etsy/logstash-forwarder">one</a> by Etsy.
</p><p>By making Logstash Forwarder a <a href="https://github.com/elastic/beats">Beats</a> project, which has a dedicated team of Go developers inside Elastic, we make sure the Elastic stack has a well maintained log forwarder.
</p><p>Over the&nbsp;past couple of months, we took the Logstash Forwarder code, split it into parts, replaced the more rusty bits, added tons of unit, integration and system tests and then put it back together in a Beat. Filebeat now shares a lot of code with our other Beats via <a href="https://github.com/elastic/beats/tree/master/libbeat">libbeat</a>, making the projects benefit from each other’s improvements.
</p><h3></h3><h2>The logs are the queue</h2><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">One of the core features of the Logstash Forwarder, which we have inherited in Filebeat is that after sending the log lines to Logstash it waits for a confirmation that the message was received. Only after this ack is received the persistent registry where it remembers how far it advanced in each file.
</p><p>This means that if there is a network partition between Filebeat and Logstash and the connection is temporarily lost, Filebeat simply waits for the connection to be re-established before advancing its pointer in the log files. Unless the connection cannot be established for a very long time and the logs are rotated, no log lines are lost. Similarly, if you restart Logstash, no log lines are lost. If you restart Filebeat, no log lines are lost. If there’s network congestion on the path, no log lines are lost. You get the idea.
</p><p>Note that there is also drawback to this “at-least-once” approach. Because we retransmit unacknowledged messages, log lines might be duplicated. For most people duplicate log messages are better than lost log messages, but we’ve been working on reducing this effect.
</p><h2>I'm a lumberjack and I'm ok</h2><p><img src="https://api.contentstack.io/v2/assets/575e4cc69e7a83165490ca8e/download?uid=bltc4a127040f7c0b0c?uid=bltc4a127040f7c0b0c" data-sys-asset-uid="blt184976bd834d7107" alt="5081167157_c1b85d4d96_m (2).jpg" style="float: right; padding: 0 0 10px 10px;">
</p><p>The protocol used between the Logstash Forwarder and Logstash already supports encryption, message batching and acknowledgements. The current version of Filebeat uses the same protocol but with a few improvements. Notably, for minimizing congestion situations that can result in duplicates, a slow start mechanism is used for the batch size, similar with the approach taken by TCP to avoid congestion. Partial ACKs were also introduced to help in this area.&nbsp;To allow for a simple migration path from the Logstash Forwarder, we have created a new input plugin for Logstash, called <a href="https://github.com/logstash-plugins/logstash-input-beats">logstash-input-beats</a>, and we made sure it can be used in parallel with the existing input plugin used by the Forwarder.
</p><p>Another change is making the certificate-based authentication optional so that it’s easier to use Filebeat while testing or developing.
</p><h2>Yes, Filebeat works on Windows</h2><p>I’m glad you asked. Just like Packetbeat and Topbeat, the other Beats projects that we have so far, Filebeat supports Windows. The Logstash Forwarder already worked on Windows but there were&nbsp;a few issues that severely crippled its functionality on that platform. We fixed those and now we run the same tests that we have for Linux and OS X on Windows as well, so you don’t have to worry about differences in behaviour when you have a mixed environment.
</p><p>The Beats packaging framework also makes it a bit easier to run Filebeat as a Windows service.
</p><h2>Migrate from the Logstash Forwarder</h2><p><img src="https://api.contentstack.io/v2/assets/575e4cc658208ba076e289fe/download?uid=bltb16461533bff7396" data-sys-asset-uid="bltb16461533bff7396" alt="7095753479_7f736695a0_m.jpg" style="float: left; padding: 0 10px 10px 0;">
</p><p>To be more like the other Beats, Filebeat uses YAML for its configuration file, rather than the JSON+comments language used by Logstash Forwarder. Since we started a new project and re-configuration is needed anyway, we took the opportunity to clean up some configuration options. For example, many of the options that Logstash Forwarder received as CLI options are now configuration file options.
</p><p>Don’t worry though, it’s all quite simple, and we’ve written a fairly extensive <a href="https://www.elastic.co/guide/en/beats/filebeat/current/_logtash_forwarder_to_filebeat_migration.html">migration guide</a> to help you on the process.
</p><h2>Packetbeat and Topbeat</h2><p>While the main focus during this release cycle was Filebeat, we continued to improve Packetbeat and Topbeat. You can find the respective change logs <a href="https://www.elastic.co/downloads/past-releases/packetbeat-1-0-0-beta4">here</a> and <a href="https://www.elastic.co/downloads/past-releases/topbeat-1-0-0-beta4">here</a>. Notably they now both support sending the data to Logstash via the same protocol that Filebeat uses, so Redis is no longer needed for the integration with Logstash. This also means that you can now easily use Logstash as a gateway to the many outputs supported by Logstash.
</p><h2>The (near) future</h2><p>We intend to have the first GA release for all the Beats in a matter of weeks. Shortly after removing the beta label, we will add to Filebeat the three most commonly requested features: multi-line support, Windows event log support, and simple grep-like filtering based on the log message. The three new planned enhancements are tracked in this <a href="https://github.com/elastic/filebeat/labels/v1.1.0">list</a>.
</p><p>We always love hearing from you, so why not give <a href="https://www.elastic.co/downloads/beats/filebeat">Filebeat</a>, <a href="https://www.elastic.co/downloads/beats/packetbeat">Packetbeat</a> or <a href="https://www.elastic.co/downloads/beats/topbeat">Topbeat</a> a try and let us know what you think on the <a href="https://discuss.elastic.co/c/beats">forums</a>?
</p><p>Image credits: <a href="https://www.flickr.com/photos/sbeebe/8777550156/">port</a>, <a href="https://www.flickr.com/photos/pasukaru76/5081167157">lumberjack</a>,&nbsp;<a href="https://www.flickr.com/photos/newtown_grafitti/7095753479">logs</a>
</p>

8777550156_5a01a3b307_k (1).jpg

Beats 1.0.0-beta4: lightweight log forwarding with Filebeat

<html><head></head><body><p>Welcome to <em>Weekly Beats</em>! With this series, we&apos;re keeping you up to date with all that&apos;s new in <a href="https://www.elastic.co/products/beats">Beats</a>, from&#xA0;the details of work in progress pull requests to&#xA0;releases&#xA0;and learning resources.</p><p>Here is what we worked on over the past week.
</p><h3>Preparing for the first Filebeat release</h3><p>We&#x2019;re very close to releasing Filebeat (the Logstash-Forwarder successor) and we are&#xA0;excited to share it with the rest of the world.
</p><span id="docs-internal-guid-21c314f9-8133-eee6-3a3e-e2f8dc6695cb"><p>We have a new <a href="https://github.com/logstash-plugins/logstash-input-beats/pull/1">Logstash plugin</a> that speaks the Beats flavor of the Lumberjack protocol. The initial plan was to enhance the Lumberjack plugin for Filebeat &#xA0;in a backwards compatible way but we changed course when we realised that there will be at least one backwards incompatible change (TLS off by default) and that having a separate plugin would make the LSF -&gt; Filebeat migration path easier.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">The new plugin name also makes our intentions of using it for all existing and future Beats more clear to our users. In a similar change, the Lumberjack output in Filebeat is renamed to &#x201C;Logstash&#x201D;.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">The new plugin compatible with both Logstash 1.5.x and 2.x branches and it&#xA0;makes&#xA0;it possible to load the &#x201C;beats&#x201D; and the &#x201C;lumberjack&#x201D; plugins at the same time.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><h3>Documentation effort
</h3><p>We had a team effort on pushing forward the <a href="https://github.com/elastic/filebeat/pull/46">documentation</a> for Filebeat as well as incrementally improving the docs for the other Beats as well, for example by documenting the <a href="https://github.com/elastic/libbeat/pull/147">CLI flags</a> and the <a href="https://github.com/elastic/libbeat/pull/139">TLS config options</a>. The developer guides for adding a new protocol to Packetbeat were also <a href="https://github.com/elastic/packetbeat/pull/307">updated</a>.<br>
</p><p>We&#x2019;re also working on a <a href="https://github.com/tsg/filebeat/blob/migration_guide/docs/migration.asciidoc">migration guide</a> from LSF.<br>
</p><h3>Direction field in Packetbeat
</h3><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Coming from a community contributor, this <a href="https://github.com/elastic/libbeat/pull/150">PR</a> adds a &#x201C;direction&#x201D; field to the transactions captured by Packetbeat. If the request is received by the machine on which Packetbeat runs, the value will be &#x201C;in&#x201D;. If the request is sent by the machine on which Packetbeat runs, the value for it will be &#x201C;out&#x201D;. This is useful for differentiating between the client and server side of the same transactions, so one can measure the response time as seen by the client, the response time as seen by the server, and then calculate the&#xA0;network time by subtracting the two.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><h3>Platform dependent location for the registry db in Filebeat
</h3><p>Filebeat persists the read offsets in a json file on disk, so no log lines are lost in case of Fiebeat restart. The JSON file is by default named &#x201C;.filebeat&#x201D; and placed in the current working directory. This is <a href="https://github.com/elastic/filebeat/issues/82">not very convenient</a> when installing from packages or running as a service on Windows, so we made the location and name <a href="https://github.com/elastic/filebeat/pull/91">depend on the way Filebeat is installed</a>. It&#x2019;s now `/var/lib/filebeat/registry` for deb/rpm and `C:\ProgramData\filebeat\registry` on Windows.
</p><h3>Gofuzz on the protocol parsers
</h3><p>Another community contributor, <a href="https://github.com/darxriggs">darxriggs</a>, tried a cool <a href="https://github.com/dvyukov/go-fuzz">coverage-guided fuzzing tool</a> on the Packetbeat parsers and found <a href="https://github.com/elastic/packetbeat/issues/315">two</a> <a href="https://github.com/elastic/packetbeat/issues/316">bugs</a> in the mongodb parser. Great way of contributing! Small side note: a nice article about <a href="https://blog.cloudflare.com/dns-parser-meet-go-fuzzer/">running go-fuzz on a DNS parser</a>.
</p><h3>Configurable transaction timeout in Packetbeat
</h3><p>The time after we removed a transaction from memory if the response was not received was hardcoded to 10 seconds in Packetbeat. This essentially meant that we couldn&#x2019;t correlate transactions that take longer than that. The timeout is now <a href="https://github.com/elastic/packetbeat/pull/308">configurable</a> on a per-protocol basis.
</p><h3>Keep a change log
</h3><p>We started to to keep change&#xA0;logs in all our projects by using the <a href="http://keepachangelog.com/">keepachangelog.com</a> standard. Here is the <a href="https://github.com/elastic/packetbeat/blob/master/CHANGELOG.md">current one for Packetbeat</a>. This will save time for us at the release time and because it&#x2019;s part of the review process, it&#x2019;s less likely to be incomplete, so it will also save time for the users.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">One gotcha with keeping the change&#xA0;log in a committed file is that it tends to result in lots of merge conflicts. We had this problem a lot until Steffen found a way to tell Git how to always <a href="https://github.com/elastic/packetbeat/pull/314">union-merge the file</a>.
</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br>
</p><h3>Better user experience with -devices
</h3><p>The `-devices` CLI flag of Packetbeat prints all the network devices on which it can sniff. If you run it without `sudo`, you cannot sniff so it used to print absolutely nothing. It now prints a <a href="https://github.com/elastic/packetbeat/pull/302">nice message</a> and on non-windows platforms it suggest using sudo.
</p></span></body></html>

Brewing in Beats: Preparing the first Filebeat release

Announcing the release of Elastic Beats 1.0.0-beta3, with Packetbeat and Topbeat, which can monitor your systems’ resources and processes.

<html><head></head><body><p>We are excited to announce a new release for the Beats, <a href="https://www.elastic.co/downloads/beats/packetbeat">Packetbeat</a> and <a href="https://www.elastic.co/downloads/beats/topbeat">Topbeat</a>, our trusted shippers of operational data. It&#x2019;s not just Packetbeat anymore, now we&#x2019;ve added Topbeat, which you can use to monitor your systems&#x2019; resources and processes. And don&#x2019;t worry, we&#x2019;ve continued to improve Packetbeat as well. In fact, thanks to you -- our community -- we have increased the pace in improving Packetbeat.
</p>
<p>Here are the highlights of the new release.
</p>
<h2>Topbeat</h2>
<p>It&#x2019;s like the top command that you know from the Linux/Unix shell, but sends the data periodically to Elasticsearch. It captures system wide data like the system load, free/used memory or disk stats as well as per process stats. You can monitor all the processes running on your system or just a subset. Oh, and it&#x2019;s not just for Linux, it works on Windows and OS X too.
</p>
<p>Guess what, using Kibana to monitor your operating system metrics is a lot of fun!
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4caed8edd48f76936355/download?uid=blt296810a0ac8b87cb?uid=blt296810a0ac8b87cb" data-sys-asset-uid="bltd6fe5f308987bdc2" style="max-width:100%">
</p>
<h2>DNS support in Packetbeat</h2>
<p>DNS support is probably our most requested protocol for Packetbeat. <a href="https://github.com/andrewkroh">Andrew Kroh</a> stepped up and added generic support for UDP protocols to Packetbeat and then DNS support on top of it. Check out the screenshot for some of the interesting facts Packetbeat reveals.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4cae43e9adc538714072/download?uid=blt84bb8fa8171e868b?uid=blt84bb8fa8171e868b" data-sys-asset-uid="blt8556214db03c4435" style="max-width:100%">
</p>
<h2>Memcache support in Packetbeat</h2>
<p>Memcache is another commonly requested feature. With this release Packetbeat fully supports it, including the binary and text protocols, over UDP and over TCP. Here is a screenshot showing statistics about the Memcache errors that Packetbeat found:
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4caed8edd48f7693635a/download?uid=bltd4cd97bb5187ded6?uid=bltd4cd97bb5187ded6" data-sys-asset-uid="blt129bd0555386a705" style="max-width:100%">
</p>
<h2>Windows support</h2>
<p>We now officially support Windows. It used to be possible to run Packetbeat on Windows but it wasn&#x2019;t an easy task. With this release we took several steps to make it simpler. This includes running as a native Windows service and logging to files. It&#x2019;s also easier now, with Packetbeat, to <a href="/support/matrix">list the available network interfaces</a> and pick the one you want to use for network traffic capturing.
</p>
<p>We plan to go even further with improving the Windows support, including adding a GUI installer to makes things even easier. Stay tuned.
</p>
<h2>Wrong libpcap version no more</h2>
<p>We&#x2019;ve completely reworked our build system and now we can statically compile against C libraries. This means libpcap is no longer a runtime dependency for Packetbeat on Linux. This means Packetbeat and Topbeat binaries work not only on the officially supported Linux distributions but also on any other Linux distribution as long as glibc is newer than 2.11.
</p>
<h2>Bonus: Developer Guides!</h2>
<p>Lots of you asked for developer guides for how to create your own Beats or for how to add a new protocol to Packetbeat. We have now published two fairly comprehensive guides: one for <a href="/guide/en/beats/libbeat/current/index.html">creating a new Beat</a> and one for <a href="/guide/en/beats/packetbeat/current/_developer_guide_adding_a_new_protocol.html">adding a new protocol</a> to Packetbeat.
</p>
<p>If you have an idea for a new Beat or have a question about any of the above, please open a topic on the <a href="https://discuss.elastic.co/c/beats">discuss forums</a>. Here are the download links for <a href="/downloads/beats/packetbeat">Packetbeat</a> and <a href="/downloads/beats/topbeat">Topbeat</a>.
</p></body></html>

packet-beat-fish-packetbeat-beats-thumbnail.png

Beyond packets: Elastic Beats 1.0.0-beta3 released

<html><head></head><body><p><img src="https://api.contentstack.io/v2/assets/575e4c8f9e7a83165490c1d0/download?uid=blt0a3be37aa959b31f?uid=blt0a3be37aa959b31f" data-sys-asset-uid="blt140adad6df1b0753" alt="" style="float: right; margin: 0 0 0 20px;">Since the recently released 1.0.0-beta2 version, Packetbeat can understand the MongoDB wire protocol. This means you can now use Packetbeat and Elasticsearch to monitor the performance of your MongoDB servers. You can play with a live demo of the associated Kibana dashboard <a href="http://demo.elastic.co/packetbeat/#/dashboard/MongoDB-performance" target="_blank">here</a>.
</p><p>The way Packetbeat works is that it captures the network traffic, correlates the request with the response, and inserts a document into Elasticsearch describing each MongoDB query or operation seen on the wire. You can then use Kibana to discover errors or slow queries and to visualize things like &#x201C;Top slowest MongoDB queries&#x201D;, &#x201C;Response time percentiles for a particular collection&#x201D;, or the &#x201C;Number of writes with unacknowledged write concern&#x201D;. But more about these visualizations later, let&#x2019;s start by giving credit where credit is due.
</p><h2>Based on a community contribution</h2><p>The beauty of having a system like Packetbeat be open source is that anyone can add support for the protocols they need. By making it easy to add new protocols and encouraging community contributions, we&#x2019;re not only extending Packetbeat&#x2019;s list of supported protocols faster, but we benefit from having experts in each technology providing insights. The open source nature also means that folks can implement support for niche or proprietary protocols.
</p><p>While we hoped that people will start creating their own protocol modules for Packetbeat, we didn&#x2019;t expect it to happen this quickly. We don&#x2019;t have yet a proper developer guide for new protocols and the APIs are scarcely documented and may be inconsistent here and there. But all of&#xA0;this didn&#x2019;t stop <a href="https://github.com/albanm">Alban Mouton</a> to submit the pull request for adding MongoDB support just a few days after we announced that <a href="https://www.elastic.co/blog/welcome-packetbeat-tudor-monica">Packetbeat has joined Elastic</a>. His pull request was well documented, contained unit and integration tests and even a simple <a href="https://github.com/elastic/packetbeat/blob/8f42d0fe0c20d65c8f6965fef695fd807ef5a6c1/SUPPORT_PROTOCOL.md">guide</a> to future protocol module writers.
</p><p>As if he didn&#x2019;t do enough already, we asked Alban to contribute to this blog post as well. He sent us the following quote:
</p><p style="margin-left: 20px;"><em>&quot;I am a enthusiastic user of Packetbeat, along with the ELK stack. The company I work for develops and integrates multiple services communicating mostly over HTTP. Tracing activity and performance at the network level is great because it separates us from the varying implementations of these services and makes system-wide analysis easy. We are at the beginning of our experience with Packetbeat but I have high hopes for the future.&quot;</em>
</p><p style="margin-left: 20px;"><em>&quot;Recently our main database has been MongoDB and I was looking for an opportunity to take a peek at the Go language, so here we are ! Thanks a lot to the Packetbeat team, Tudor Golubenco in particular, for playing the Open Source game so well and for bringing this rough contribution to release level.&quot;</em>
</p><p>Thank you Alban for your excellent work and for leading the way to more community contributions!
</p><h2>Mongo&#x2019;s wire protocol</h2><p>Let&#x2019;s do a quick dive in the details of the MongoDB wire protocol, so you get a better understanding of what data Packetbeat captures.
</p><p>It is a fairly friendly protocol to a passive decoder like Packetbeat. Messages and individual fields have lengths, so it is easy to jump over the fields that we don&#x2019;t understood or we don&#x2019;t need. There is little contextual state so it is easy to understand what is happening even when capturing only part of the conversion. Also, because a single serialization protocol is used in all messages (<a href="http://bsonspec.org/">BSON</a> - binary JSON), it was easy for us to leverage an existing Go library to decode them. Namely, we use the BSON implementation from the <a href="http://labix.org/mgo">mgo </a>driver, written by Gustavo Niemeyer.<br>
</p><p>One thing to note is that prior to version 2.6, the write operations didn&#x2019;t actually require a response from the server. By default, however, the drivers called the GetLastError method to acknowledge that the write was successful.&#xA0;
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8fd8edd48f76935e07/download?uid=bltc7e6aa0b9b7cb888?uid=bltc7e6aa0b9b7cb888" data-sys-asset-uid="bltc0cf17784a88a8cc" style="max-width:100%;">
</p><p>This was <a href="http://docs.mongodb.org/manual/release-notes/2.6/#rel-notes-write-operations">changed in version 2.6</a>, which introduced a new wire protocol for insert, update and delete commands so that a response is always sent from the server and it doesn&#x2019;t require a second command. Currently Packetbeat only supports the newer version of the protocol, so write commands from versions prior to 2.6 are not captured. We (or you!) can implement this in the future if there is demand.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8e12798d8e381fd6a9/download?uid=blt3aa2e7c3b65a7a11?uid=blt3aa2e7c3b65a7a11" data-sys-asset-uid="blt119ef9b785d07514" alt="" style="float: right; margin: 0px;">
</p><p>Another interesting aspect is that in a MongoDB cluster the same wire protocol is used between the three server types: mongos, config server and data shards servers. The same protocol is also used for replication between shard nodes. This means you can use Packetbeat for advanced troubleshooting of what is going on inside the cluster!
</p><p>To capture the traffic inside the cluster, you need to install Packetbeat not only on the routing nodes but also on the shard nodes. You likely also need to add port 27019 to the list of MongoDB ports, because this is used by the configuration nodes.
</p><h2>Kibana dashboards</h2><p>As usual with the protocols we support, we also provide a sample Kibana <a href="http://demo.elastic.co/packetbeat/#/dashboard/MongoDB-performance" target="_blank">dashboard</a> to highlight some of the visualizations you can build based on the data Packetbeat collects and to give you a good starting point for creating your own dashboard. Let&#x2019;s see some of the visualizations made possible by this data.
</p><h3>MongoDB errors</h3><p>Packetbeat inspects automatically the response payload to check if it contains error. If it finds any, it sets the status field &#xA0;to error Error. This example shows the number of MongoDB errors evolution sliced by both the collection name and operation type.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8f9e7a83165490c1ad/download?uid=bltc9525251c2a385a3?uid=bltc9525251c2a385a3" data-sys-asset-uid="blt87269016986f5928" style="max-width:100%;">
</p><h3>Total time spent per collection</h3><p>This example shows the total amount of time spent while querying each collection. This view adds to, for example, a 99th percentile visualization of response times because a high total count could be reached if there are slow queries but also if there are many small queries.<br>
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8f3dc542cb38c082c8/download?uid=blt5c6b1c3e14b78a0f?uid=blt5c6b1c3e14b78a0f" data-sys-asset-uid="blte488011987bacbf7" style="max-width:100%;">
</p><h3>Response times and count per collection</h3><p>This one plots the 99th response time percentile on the Y axis and the dot size represents the number of queries in the particular time bucket. Different colors represent different collections. Just another way of visualizing performance data which can be helpful in spotting trends. In this screenshot, for example, you can notice the cyclicality of the response times.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8fd8edd48f76935deb/download?uid=blte0f1674c1eb8f622?uid=blte0f1674c1eb8f622" data-sys-asset-uid="bltc3a043c0784f87ae" style="max-width:100%;">
</p><h3>Number of writes with unacknowledged write concern</h3><p>By inspecting the request payload and looking for the writeConcern parameter of write operations, we can plot the number of them that have an unacknowledged write concern. As always with Packetbeat and Kibana, you can drill down from this graph and find these exact transactions and who creates them.<br>
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8f0342dfd738263158/download?uid=blt008d9a2e43355a76?uid=blt008d9a2e43355a76" data-sys-asset-uid="blt78196b2628f62bd3" style="max-width:100%;">
</p><h3 dir="ltr">Top slowest MongoDB queries</h3><p>Let&#x2019;s end with a fairly obvious but very useful one: top slowest MongoDB queries by their 99th percentile.
</p><p><img src="https://api.contentstack.io/v2/assets/575e4c8f43e9adc538713b6c/download?uid=blt4613a8b34c7b13d3?uid=blt4613a8b34c7b13d3" data-sys-asset-uid="blt93d87fc0dee5c910" style="max-width:100%;">
</p><h3>Getting started with Packetbeat for MongoDB</h3><p>If you didn&#x2019;t try Packetbeat before, we recommend reading and following the <a href="https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-getting-started.html">getting started</a> guide. By default, Packetbeat doesn&#x2019;t index payload information to avoid capturing sensitive data by mistake. But you can easily enable payload indexing from the configuration file. Here is an example configuration section showing the main options:
</p><pre class="prettyprint">protocols:  
  mongodb:
    ports: [27017, 27019]
    send_request: true     # index the request payload
    send_response: true    # index the response payload
    max_docs: 10           # maximum number of documents to index per request/response
    max_doc_length: 1024   # maximum document size to index
</pre><p>Image credit: <a href="https://www.flickr.com/photos/kumaravel/8282786757">green leaf</a>
</p></body></html>

AUTHOR

Articles by Tudor Golubenco

Beats 6.5.0 released

Beats 6.4.0 Released

Beats 6.3.0 Released

Beats 6.2.0 released

Beats 6.1.0 released

Beats 6.0.0 GA released

Beats 5.6.0 released

Beats 6.0.0-beta1 released

Beats 6.0.0-alpha2 released

Sign up for Elastic Cloud free trial

Products & Solutions

Company

Resources

Solutions

Elastic (ELK) Stack

New

Featured topics

News

AUTHOR

Articles by Tudor Golubenco

Beats 6.5.0 released

Beats 6.4.0 Released

Beats 6.3.0 Released

Beats 6.2.0 released

Beats 6.1.0 released

Beats 6.0.0 GA released

Beats 5.6.0 released

Beats 6.0.0-beta1 released

Beats 6.0.0-alpha2 released

Sign up for Elastic Cloud free trial