Logstash 7.9.0 has arrived with faster pipeline startups & restarts, a refreshed Elastic App Search output, continued API key support, and new support for ARM/ARRCH64.

<p>We are happy to announce the general availability of the Logstash 7.9.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/logstash">download</a>! Please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.9/releasenotes.html">release notes</a> for the complete list of bug fixes and features.
</p> <strong><h2>Faster pipeline startups and restarts&nbsp;</h2></strong><p>For some time now, we have heard clear feedback from our users that the compilation of complex Logstash pipelines may take extended periods of time to execute, effectively resulting in longer startup and restart times. In the Logstash 7.9.0 release, we are excited to introduce some substantial optimizations to pipeline compilation that result in over 9x faster pipeline startups and restarts. This should help&nbsp;streamline the pipeline development workflow and enhance overall operational excellence.
</p><p>Through our tests with&nbsp;three different real world pipelines running&nbsp;8 workers each, we have seen pipeline compilation time reduced from&nbsp;up to 28 minutes down to just around a&nbsp;minute. For deeper technical details regarding these optimizations, please see <a href="https://github.com/elastic/logstash/pull/12060">this PR</a>.&nbsp;Additionally, to assist with&nbsp;pipeline development and better understanding of the performance impact of compilation, Logstash now logs the time taken to compile each pipeline:
</p><pre class="prettyprint">[2020-08-12T14:10:29,388][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=&gt;0.7}
</pre> <strong><h2>Unlock more application search use cases<br>
</h2></strong><p>With this Logstash release, we have taken the opportunity to completely revamp the <a href="https://www.elastic.co/guide/en/logstash/7.9/plugins-outputs-elastic_app_search.html">Elastic App Search output plugin</a> with a more robust Ruby client library and greatly expanded integration testing coverage. This enables our users to unlock application search on their datasets that may currently reside in existing databases, raw/blob&nbsp;storage, and messaging queues. It also allows for an easy migration path from a custom Elasticsearch application search use case to <a href="https://www.elastic.co/app-search/">Elastic App Search</a>. This plugin has now graduated to a tier 1 plugin and is compatible with Elastic App Search 7.x
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt770555c62fb6bdb0/5f3a235b9a5d3175f897b3aa/logstash-app-search-blog.png" data-sys-asset-uid="blt770555c62fb6bdb0" alt="logstash-app-search-blog.png" "="">
</p><strong><h2>Continued API key support</h2></strong><p>Following up on the Logstash 7.8.0 release where we added API key support for the Elasticsearch output plugin, Logstash 7.9.0 introduces additional API key support for the <a href="https://www.elastic.co/guide/en/logstash/7.9/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-api_key">Elasticsearch input</a>, <a href="https://www.elastic.co/guide/en/logstash/7.9/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-api_key">Elasticsearch filter</a>, <a href="https://www.elastic.co/guide/en/logstash/7.9/monitoring-internal-collection-legacy.html#monitoring-additional-settings-legacy">Logstash monitoring</a>, and <a href="https://www.elastic.co/guide/en/logstash/7.9/configuring-centralized-pipelines.html#configuration-management-settings">central pipeline management</a>.
</p><p>Please check out <a href="https://www.elastic.co/guide/en/logstash/7.9/ls-security.html#ls-api-keys">the documentation</a> for more information about using API keys with Logstash and Elasticsearch.
</p> <strong><h2>Support for ARM/AARCH64</h2></strong><p>We always try our best to meet our users where they are, and with this release we continue down the path of making Logstash available on more platforms. The Logstash binaries are now&nbsp;tested&nbsp;to work against ARM/AARCH64 systems. This is only the initial step of the journey, and we are looking to offer Docker and other images in the near future as well. ARM/AARCH64 support&nbsp;is currently offered as <code>experimental</code> and therefore not yet recommended for production environments.
</p><p>Please consult our <a href="https://www.elastic.co/support/matrix">support matrix</a> for the full list of operating systems that Logstash supports.
</p> <strong><h2>Try it yourself!</h2></strong><p>Please <a href="https://www.elastic.co/downloads/logstash">download Logstash 7.9.0</a>, try it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a> and <a href="https://elasticstack.slack.com">community Slack</a>. You can report any bugs or feature requests on the Logstash <a href="https://github.com/elastic/logstash/issues">Github issues page</a> or within the <a href="https://github.com/logstash-plugins">respective plugin repositories</a>.
</p>

logstash-blog-banner.png

logstash-blog-thumb.png

What's new in Logstash 7.9.0

Logstash 7.8.0 has arrived with API key support for authenticating to Elasticsearch, proxy support for monitoring and management, and many new supported OS and JVM/JDK.

<p>We are happy to announce the general availability of the Logstash 7.8.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/logstash">download</a>.&nbsp;Please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.8/releasenotes.html">release notes</a> for the complete list of bug fixes and features. </p> <strong><h2>Authenticate to Elasticsearch with API Keys</h2></strong> <p>As an enabler for many use cases across the Elastic Stack, Logstash integrates with Elasticsearch in various ways, and security remains top of mind for both Elastic and the broader community. Alongside other existing authentication options, we have introduced API key authentication for Elasticsearch with Logstash 7.8.0. API keys are secret tokens managed by Elasticsearch and Kibana that can be used as credentials for machine-to-machine authentication and authorization. It offers a flexible way to restrict Elasticsearch permissions for products like Logstash depending on the client’s responsibilities and use cases. For instance, the Logstash Elasticsearch output only needs write access. Users can therefore authenticate with an API key that only allows for writes to Elasticsearch, while reads and other actions would be prohibited. </p> <p>With the 7.8.0 release, users can authenticate directly into Elasticsearch with API keys in the Elasticsearch output plugin. This requires creating an API key in Elasticsearch and specifying it in the new <a href="https://www.elastic.co/guide/en/logstash/7.8/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-api_key"><code>api_key</code> option</a>. Wire encryption (SSL/TLS) must also be enabled. Additionally, the usage of API keys is available in the <a href="https://www.elastic.co/guide/en/logstash/master/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-api_key">Elasticsearch input</a> (v4.7.0 and above) and <a href="https://www.elastic.co/guide/en/logstash/master/plugins-filters-elasticsearch.html#plugins-filters-elasticsearch-api_key">Elasticsearch filter</a> (v3.8.0 and above) plugins with a plugin update:</p> <p><code>bin/logstash-plugin update logstash-input-elasticsearch</code> </p> <p><code>bin/logstash-plugin update logstash-filter-elasticsearch</code></p><p>For information around creating and managing API keys, please consult the <a href="https://www.elastic.co/guide/en/logstash/7.8/ls-security.html#ls-api-keys">Logstash security documentation</a>. </p> <strong><h2>Proxy support for management and monitoring</h2></strong> <p>There are a myriad of ways Logstash ingests data into the Elastic Stack, and sometimes our users may deploy Logstash in a way where it may not be able to communicate directly with Elasticsearch or even reach out to the internet as a whole. Although there has been <a href="https://www.elastic.co/guide/en/logstash/7.8/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-proxy">proxy support</a> available in plugins like the Elasticsearch output, we have now added proxy support for Logstash <a href="https://www.elastic.co/guide/en/logstash/7.8/monitoring-internal-collection-legacy.html#monitoring-general-settings-legacy">monitoring</a> and <a href="https://www.elastic.co/guide/en/logstash/7.8/configuring-centralized-pipelines.html#configuration-management-settings">central management</a> in the 7.8.0 release. The proxy URLs can be configured in the <code>logstash.yml</code> with the <code>xpack.monitoring.elasticsearch.proxy</code> and <code>xpack.management.elasticsearch.proxy</code> settings.&nbsp; </p> <strong><h2>Azure module deprecation</h2></strong> <p>The <a href="https://www.elastic.co/guide/en/logstash/7.8/azure-module.html">Logstash Azure module</a> has been deprecated in Logstash 7.8.0&nbsp;and we plan to remove it completely in a future major release. For Azure users interested in monitoring their deployments with the Elastic Stack, we recommend checking out the <a href="https://www.elastic.co/guide/en/beats/filebeat/7.8/filebeat-module-azure.html">Filebeat Azure module</a> and <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.8/metricbeat-module-azure.html">Metricbeat Azure module</a>. These newer modules provide comprehensive visibility into your Azure infrastructure and services, while complying with the <a href="https://www.elastic.co/guide/en/ecs/current/index.html">Elastic Common Schema (ECS)</a> for more seamless correlation with the many other data sources that may reside together in your Elastic Stack deployment. </p> <strong><h2>Platform and Java Virtual Machine (JVM) support continues to grow</h2></strong> <p>Logstash runs on many different operating platforms alongside various flavors of the Java Development Kit (JDK). We’ve recently been expanding this coverage, and Logstash 7.8.0 continues to trek down this important path. In this release, we’re proud to announce that Logstash has introduced support for running on CentOS/RHEL 8.x and Ubuntu 20.04. Additionally, we’ve added new JDK support for Zulu 11, AdoptOpenJDK 11, and Oracle/OpenJDK/AdoptOpenJDK 14. </p> <strong><h2>Try it yourself!</h2></strong> <p>Please <a href="https://www.elastic.co/downloads/logstash">download Logstash 7.8.0</a>, try it, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a> and <a href="https://elasticstack.slack.com">community Slack</a>. You can report any bugs or feature requests on the Logstash <a href="https://github.com/elastic/logstash/issues">Github issues page</a> or within the <a href="https://github.com/logstash-plugins">respective plugin repositories</a>. </p>

Logstash 7.8.0 released

Logstash 7.7.0 has arrived with new event ordering, easier troubleshooting through better logging, enhanced S3 integrations, and more Windows platform support.

<p>We are happy to announce the general availability of the Logstash 7.7.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/logstash">download</a>! Please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.7/releasenotes.html">release notes</a> for the complete list of bug fixes and features.
</p><p> <strong></strong>
</p><strong><h2>Event ordering with the Java execution engine</h2></strong><p>Within the world of event streaming, preserving the ordering of events can be crucial in a myriad of processing workflows. Historically, many users have achieved this behavior in Logstash with single worker pipelines running in the legacy Ruby execution engine. Although event ordering has not previously been possible with the next-gen Java execution engine (on by default since Logstash 7.0.0), we are excited to reintroduce these capabilities on top of this speedy engine.
</p><p>With Logstash 7.7.0, proper event ordering is the default behavior for pipelines running with a single worker. The new <code>pipeline.ordered</code> setting offers more granular control of this feature. For more information, please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.7/logstash-settings-file.html">logstash.yaml documentation</a>.
</p><p> <strong></strong>
</p><strong><h2>Easier troubleshooting through better logging</h2></strong><p>Building on the logging improvements made over the last few releases, we continue to focus on making Logstash easier to self-service and debug during those times of need. In this release, we have expanded log messages to more accurately display information about plugins and pipelines. This includes adding <code>plugin.name</code> to all log entries across all deployments, and <code>plugin.id</code> and <code>pipeline.id</code> when running in Docker containers. With this valuable information at your fingertips, you can better trace and troubleshoot problems with specific plugin instances especially when used alongside the Logstash <a href="https://www.elastic.co/guide/en/logstash/7.7/logstash-pipeline-viewer.html">pipeline viewer UI</a>.
</p><p> <strong></strong>
</p><strong><h2>Enhanced S3 integrations</h2></strong><p>As the AWS S3 object store continues to be a very popular integration in Logstash, both as a target and destination, we have introduced a few notable features in this release. In the S3 input plugin, we have added support to ingest files restored from AWS Glacier and Glacier Deep Archive. Additionally, there is now more flexible determination of whether a file is gzipped with the inclusion of a new <a href="https://www.elastic.co/guide/en/logstash/7.7/plugins-inputs-s3.html#plugins-inputs-s3-gzip_pattern">gzip_pattern option</a>. For the S3 output plugin, we have added new retry configuration settings to help avoid infinite upload retry loops during miscellaneous error conditions.
</p><p> <strong></strong>
</p><strong><h2>Expanded Windows platform support</h2></strong><p>As data processing pipelines, Logstash often runs on a wide array of operating platforms. Although we have supported Windows Server 2012/R2 and 2016 for some time now, we are excited to announce new platform support for Windows Server 2019 with this release. Check out the <a href="https://www.elastic.co/support/matrix">support matrix</a> for the full list of supported platforms!
</p><p> <strong></strong>
</p><strong><h2>Try it out yourself!</h2></strong><p>Please <a href="https://www.elastic.co/downloads/logstash">download Logstash 7.7.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a>&nbsp;and <a href="https://elasticstack.slack.com">community Slack</a>. You can report any bugs or feature requests on the Logstash <a href="https://github.com/elastic/logstash/issues">Github issues page</a> or within the <a href="https://github.com/logstash-plugins">respective plugin repositories</a>.
</p>

Logstash 7.7.0 released

Beats 7.4.0 is here. It includes new Filebeat inputs for S3 and Kafka, various Functionbeat improvements, new platform support, and many new modules.

<p>We’re pleased to announce the general availability of the Beats 7.4.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/beats">download</a>! Please refer to the <a href="https://www.elastic.co/guide/en/beats/libbeat/7.4/release-notes-7.4.0.html">release notes</a> for the complete list of bug fixes and features. </p> <strong><h2>Scalable consumption from S3 and Kafka</h2></strong> <p>With Filebeat 7.4.0, we’ve introduced new ways to consume data from AWS S3 and Kafka into the Elastic Stack. This ingestion can be done scalably with at-least-once delivery guarantees. </p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd0f2056c7cf5a821/5d89405da3629438cf51e630/beats-7-4-0-blog-s3-kafka.png" data-sys-asset-uid="bltd0f2056c7cf5a821" alt="Ingest data from S3 and Kafka into the Elastic Stack with Filebeat" style="display: block; margin: auto;"></p>    <p>The new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-input-s3.html">S3 input</a> in Filebeat has been added as a beta feature under the Basic license, meaning it’s free to use. It ingests raw log lines from S3 buckets by leveraging SQS queues for scalable consumption. We recommend using this Filebeat input instead of the Logstash S3 input if you are looking for a horizontally scalable solution for ingesting logs from S3. </p>  <p>The new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-input-kafka.html">Kafka input</a> has also been added in Filebeat 7.4.0 and enables data consumption from Kafka topics. Multiple Filebeats can subscribe to the same Kafka consumer group for parallel processing from topics. Additionally, the Kafka input can be used to consume data from Azure Event Hubs given the service supports Kafka interface compatibility. </p> <strong><h2>Functionbeat levels up</h2></strong> <p>If you’ve been following along, Functionbeat has been going through substantial enterprise hardening for the AWS integrations over the last few releases and the 7.4.0 release continues down that path. We’ve heard clear feedback from our users and have enabled <a href="https://www.elastic.co/guide/en/beats/functionbeat/7.4/logstash-output.html">Logstash as an output</a> for data processing with Functionbeat. Additionally, support for configurable function tags has also been added which can be taken advantage of for grouping, filtering, and cost allocation with AWS Lambda. </p>  <p>Most notably, Functionbeat 7.4.0 enables better support for the popular AWS cloud monitoring architecture where CloudWatch Logs are centralized through Kinesis. We’ve added a new CloudWatch Logs Kinesis function type to Functionbeat that ingests CloudWatch Logs through Kinesis in a turnkey fashion, automatically decoding and decompressing the events properly prior to shipment downstream. </p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt22fcb021761f0796/5c304739682c4c4866a7980e/functionbeat-1.png" data-sys-asset-uid="blt22fcb021761f0796" alt="Functionbeat is the serverless data shipper for the Elastic Stack" style="display: block; margin: auto;"><br></p>    <p>This new functionality also enables multi-subscription CloudWatch monitoring where multiple CloudWatch Logs subscriptions can log to a centralized Kinesis for ingestion with Functionbeat. This is a key architecture for many monitoring AWS deployments and we encourage you to try it out! </p> <strong><h2>More platforms, more data</h2></strong> <p>As ubiquitous data shippers, Beats often run on host servers which can encompass a myriad of different types of operating system platforms. The more platforms Beats support means more data that they can collect, spanning from logs and metrics to wire and audit data. With Beats 7.4.0, we are pleased to announce that we have added support for RHEL 8, Amazon Linux 2, Ubuntu 18.04, and Windows Server 2019 across all of the different Beats. If you’re looking to collect data from any of these operating systems, we encourage you to take Beats for a spin to further broader the aperture of observability across your deployments. </p> <strong><h2>Modules galore</h2></strong> <p>The Beats platform has been a foundational driver for the broad ecosystem of modules and data sources within the Elastic Stack. In the 7.4.0 release, we’re proud to announce an abundant addition of new modules across the Observability and SIEM domains. </p>  <p>In the Observability realm, we’ve added a bit of everything like new Filebeat modules for <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-ibmmq.html">IBM MQ</a> and <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-aws.html">S3 server access logs</a>, along with a diverse assortment of new metricsets and dashboards for various different Metricbeat modules. Check out the <a href="https://www.elastic.co/blog/elastic-infrastructure-7-4-0-released">Elastic Infrastructure</a> and <a href="https://www.elastic.co/blog/elastic-logs-7-4-0-released">Elastic Logs</a> 7.4.0 blog posts for additional details! </p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2e86ff845c62ac50/5d89425c34b0e32e8c2532bc/beats-7-4-0-blog-dashboard.jpg" data-sys-asset-uid="blt2e86ff845c62ac50" alt="Explore all the new modules in Beats 7.4.0" style="display: block; margin: auto;"></p>    <p>On the SIEM front, we’ve rewritten the <a href="https://www.elastic.co/guide/en/beats/auditbeat/7.4/auditbeat-dataset-system-socket.html">Auditbeat Socket dataset</a>, added dashboards to the <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-netflow.html">NetFlow module</a>, and introduced net new modules for the <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-cef.html">Common Event Format (CEF)</a> and <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-cisco.html">Cisco Firepower Threat Defense (FTD)</a> in Filebeat. Check out the <a href="https://www.elastic.co/blog/elastic-siem-7-4-0-released">Elastic SIEM 7.4.0 blog post</a> for more details! </p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt244029c1ff67e215/5d8942905624367917a5cfcd/beats-7-4-0-blog-netflow-dash.jpg" data-sys-asset-uid="blt244029c1ff67e215" alt="New in 7.4.0, the Netflow flow exporter dashboard" style="display: block; margin: auto;"></p>    <strong><h2>Try it out yourself!</h2></strong> <p>Please <a href="https://www.elastic.co/downloads/beats">download Beats 7.4.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a>. You can report any bugs or feature requests on the Beats <a href="https://github.com/elastic/beats/issues">Github issues page</a>. </p>

beats-blog-banner.png

beats-blog-thumb.png

Beats 7.4.0 released

Logstash 7.4.0 is now available. Pipeline-to-pipeline communication has gone GA and the NetFlow module has been deprecated.

<p>We’re happy to announce the general availability of the Logstash 7.4.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/logstash">download</a>! Please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.4/logstash-7-4-0.html">release notes</a> for the complete list of bug fixes and features.
</p> <strong><h2>Pipeline-to-pipeline communication GA</h2></strong><p>The <a href="https://www.elastic.co/guide/en/logstash/7.4/pipeline-to-pipeline.html">pipeline-to-pipeline communication</a> feature is now officially GA with the Logstash 7.4.0 release, enabling a production ready approach for connecting <a href="https://www.elastic.co/guide/en/logstash/7.4/multiple-pipelines.html">multiple pipelines</a> within the same Logstash instance. This provides our users the ability to leverage more pipeline logic isolation and modularity, ultimately leading to better processing performance while retaining at-least-once delivery semantics across the entire chain of delivery.
</p><p>The power of chaining pipelines unlocks a plethora of different use cases in practice. Whether you want to separate processing logic for multiple data types coming through a single input or guarantee stronger delivery when using multiple outputs, pipeline-to-pipeline communication allows sophisticated processing to be done more efficiently and in an easier to manage fashion. Check out a few different <a href="https://www.elastic.co/guide/en/logstash/7.4/pipeline-to-pipeline.html#architectural-patterns">architectural patterns</a> we’ve outlined as inspiration; we’re excited to see the myriad of ways you may take advantage of this feature in your own deployments!
</p> <strong><h2>NetFlow module deprecation</h2></strong><p>The <a href="https://www.elastic.co/guide/en/logstash/7.4/netflow-module.html">Logstash NetFlow module</a> has been deprecated in Logstash 7.4.0 and we plan to remove it completely in a future major release. For users interested in NetFlow data analysis, we recommend checking out the <a href="https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-netflow.html">Filebeat NetFlow module</a>. It has functional parity with the Logstash module equivalent while complying with the <a href="https://www.elastic.co/guide/en/ecs/current/index.html">Elastic Common Schema (ECS)</a> for easy usage with the <a href="https://www.elastic.co/siem">Elastic SIEM</a> application and more seamless correlation with other data sources that reside together in your Elastic Stack deployment.
</p> <strong><h2>Try it out yourself!</h2></strong><p>Please <a href="https://www.elastic.co/downloads/logstash">download Logstash 7.4.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a>. You can report any bugs or feature requests on the Logstash <a href="https://github.com/elastic/logstash/issues">Github issues page</a> or within the <a href="https://github.com/logstash-plugins">respective plugin repositories</a>.
</p>

Logstash 7.4.0 released

Beats 7.3.0 is here, and it has new Google Cloud integrations, database metricsets (Oracle, Microsoft, Cockroach, AWS RDS), improved k8s monitoring, and more.

<p>We’re pleased to announce the general availability of the Beats 7.3.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/beats">download</a>!
</p><p>This release is all about further growing the data source ecosystem!<span style="font-size: 12px;"></span></p><p><span style="font-size: 10px;"></span><br></p><h2>Automate Functionbeat with CloudFormation</h2><p><a href="https://www.elastic.co/products/beats/functionbeat">Functionbeat</a> deploys on your serverless environment to collect events generated by cloud services and ship them over to Elasticsearch. The Functionbeat 7.3.0 release introduces the ability to export a CloudFormation template for integration with automation systems. Previously, Functionbeat was only available for manual command line deployment, but with this new functionality, users can now more easily automate Functionbeat deployments through their own CloudFormation stacks. We invite you to try out this new export feature and let us know what you think!
</p><p>CloudFormation templates can be exported with the <code>function</code> subcommand under the <code>export</code>&nbsp;command:</p><pre class="prettyprint">$ ./functionbeat export function &lt;FUNCTION_NAME&gt;
</pre><p><br></p><h2>Google Cloud integrations</h2><p><a href="https://www.elastic.co/products/beats/filebeat">Filebeat 7.3.0</a> introduces a new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-module-googlecloud.html">Google Cloud module</a> that monitors Virtual Private Cloud (VPC) flow logs from Google Cloud through Stackdriver. It ships data to Elasticsearch in ECS format so it’s immediately available for analysis in the <a href="https://www.elastic.co/products/siem">SIEM app</a> (or with <a href="https://www.elastic.co/products/maps">Maps</a>, on a <a href="https://www.elastic.co/what-is/kibana-canvas">Canvas</a> workpad … you name it).</p><p>Along with the module, Filebeat also introduces a new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-input-google-pubsub.html">Google Pub/Sub input</a> for consuming from Google Pub/Sub topics. This input can be leveraged for centralized ingestion of all your events from Google Cloud for real-time analytics with the Elastic Stack.
</p><p><br></p><h2>Databases for days</h2><p>Databases are often the most critical part of a user-facing application. If they’re down or struggling, users are likely to notice the impact very quickly, so it’s important to understand how your database is performing. With the 7.3.0 release, we’re introducing support for three new databases in Metricbeat along with a new database source in Filebeat.
</p><h3>Oracle Database</h3><p>The new Metricbeat <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-module-oracle.html">Oracle module</a> provides the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-metricset-oracle-tablespace.html">tablespace</a> metricset, which includes information about data files and temp files, grouped by tablespace. This module includes information on used and free space, the status of the data files, as well as the status of the tablespace itself.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt145a107bbeaa2510/5d3f56c43871f12662b33b10/beats-7-3-0-metricbeat-oracle-dashboard-blog.jpg" data-sys-asset-uid="blt145a107bbeaa2510" alt="Metricbeat Oracle dashboard" "=""></p><h3>Amazon RDS</h3><p>If you’re using Amazon <a href="https://aws.amazon.com/rds/">Relational Database Service (RDS)</a>, you can now collect a rich set of metrics about your deployment, from CPU and memory usage, to disk and network throughput and latency. See all the details in the new <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-metricset-aws-rds.html">RDS</a> metricset, now available in the Metricbeat <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-module-aws.html">AWS module</a>.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0d6f1e7402e20a23/5d3f56dc2550a26a5bd0aa4c/beats-7-3-0-amazon-rds-dashboard-blog.jpg" data-sys-asset-uid="blt0d6f1e7402e20a23" alt="Metricbeat Amazon RDS dashboard" "=""></p><h3>CockroachDB</h3><p>The Metricbeat <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-module-cockroachdb.html">CockroachDB module</a> exposes the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-metricset-cockroachdb-status.html">status</a> metricset, which is compatible with any CockroachDB version exposing metrics in Prometheus format.</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt07584fd101cd47be/5d3f5705e6078e234ba9df97/beats-7-3-0-metricbeat-cockroach-dashboard-blog.jpg" data-sys-asset-uid="blt07584fd101cd47be" alt="Metricbeat CockroachDB dashboard" "=""></p><h3>Microsoft SQL Server
</h3><p>The new Filebeat <a href="https://www.elastic.co/guide/en/beats/filebeat/7.3/filebeat-module-mssql.html">MS SQL module</a> monitors the Microsoft SQL Server error logs with the Elastic Stack.
</p><p><br></p><h2>Improved Kubernetes monitoring</h2><p>Metricbeat 7.3.0 strengthens Kubernetes observability by introducing metricsets for three additional Kubernetes components:
</p><ul>
	<li>The <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-metricset-kubernetes-controllermanager.html">controllermanager</a> metricset captures metrics from <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/cloud-controller-manager/">kube-controller-manager</a>.</li>
	<li>The <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-metricset-kubernetes-proxy.html">proxy</a> metricset covers <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/">kube-proxy</a>, the network proxy component in Kubernetes.</li>
	<li>The <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-metricset-kubernetes-scheduler.html">scheduler</a> metricset provides visibility into the <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/">kube-scheduler</a>.</li>
</ul><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt564aa21095c29a24/5d3f57222ec4932346e0b246/beats-7-3-0-metricbeat-kubernetes-dashboard-blog.gif" data-sys-asset-uid="blt564aa21095c29a24" alt="Metricbeat Kubernetes (k8s) data" "=""></p><p><br></p><h2>Configuration-only Metricbeat modules</h2><p>We’re always looking for ways to streamline the process of adding new supported data sources. That’s why 7.3.0 introduces a new way of creating Metricbeat modules that doesn’t require a single line of Go code. We’re calling them light modules and they’re effectively a pre-defined configuration on top of an existing, more generic module, such as <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-module-prometheus.html">Prometheus</a> or <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-module-jolokia.html">Jolokia</a>. In fact, the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.3/metricbeat-module-cockroachdb.html">CockroachDB</a> module introduced in this release is our first light module, and it was much easier to create as a result.
</p><p>Stay tuned — we’ll be sharing more about these light modules moving forward.
</p><p><br></p><h2>There’s a lot more to see
</h2><p>If you’ve made it this far, you might want to check out some of our related release blogs:
</p><ul>
	<li>The Elastic APM .NET Agent is now generally available. Plus, Elastic APM adds a new way to view services with aggregate service breakdown charts, agent sample rate configuration right from Kibana, and a <a href="https://www.elastic.co/blog/elastic-apm-7-3-0-released">whole lot more</a>.</li>
	<li>Elastic Uptime has improved the look and feel of multi-location monitors with <a href="https://www.elastic.co/blog/elastic-uptime-monitoring-7-3-0-released">monitor summaries and details</a>.</li>
	<li>Elastic Logs adds <a href="https://www.elastic.co/blog/elastic-logs-7-3-0-released">keyword highlighting</a>, along with the ability to jump to APM traces based on <code>trace.id</code>.</li>
	<li>Metrics Explorer in the Elastic Infrastructure app, which allows you to quickly perform aggregation visualizations against time-series metrics, is now ready for use in production systems. Elastic Infrastructure also adds improved monitoring of additional core Kubernetes services and a new RDS metricset in the AWS module. Learn more in the <a href="https://www.elastic.co/blog/elastic-infrastructure-7-3-0-released">Elastic Infrastructure release blog</a>.</li>
</ul><p>Please refer to the <a href="https://www.elastic.co/guide/en/beats/libbeat/7.3/release-notes-7.3.0.html">release notes</a> for the complete list of bug fixes and features.
</p>

Beats 7.3.0 released

Logstash 7.3.0 is now available with pipeline-to-pipeline communication improvements and the JMS input plugin bundled by default for ingesting data from JMS.

<p>We’re happy to announce the general availability of the Logstash 7.3.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/logstash">download</a>!&nbsp; Please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.3/logstash-7-3-0.html">release notes</a>&nbsp;for the complete list of bug fixes and features.
</p><p><br></p><h2 style="background-color: rgb(255, 255, 255);">Pipeline-to-pipeline communication improvements</h2><p style="background-color: rgb(255, 255, 255);">The <a href="https://www.elastic.co/guide/en/logstash/7.3/pipeline-to-pipeline.html">pipeline-to-pipeline communication</a> feature was introduced back in Logstash 6.3.0 as a new way to connect <a href="https://www.elastic.co/guide/en/logstash/7.3/multiple-pipelines.html">multiple pipelines</a> within the same Logstash instance. Historically,&nbsp;monolithic pipeline configurations were commonly used to construct sophisticated processing logic for handling the data flow of multiple different data sources. Today, configurations can be effectively split out into separate, isolated pipelines that are stitched together with pipeline-to-pipeline communication. This helps both streamline performance and enhance processing modularity.</p><p style="background-color: rgb(255, 255, 255);">With Logstash 7.3.0, we've made a notable effort to spruce up this feature and have <a href="https://github.com/elastic/logstash/issues/10502">resolved all known bugs</a>. Although still in beta, right now would be the perfect time to give it a spin. We look forward to any feedback you may have!</p><p style="background-color: rgb(255, 255, 255);"><br></p><h2>Consuming from JMS queues</h2><p>Java Message Service (JMS) technologies are an important part of some of our users’ production environments. In Logstash 7.2.0, we overhauled the Logstash <a href="https://www.elastic.co/guide/en/logstash/7.3/plugins-inputs-jms.html">JMS input plugin</a>, improving performance, security, and reliability. With the 7.3.0 release, we’re now bundling the JMS input plugin by default so its readily available as soon as you upgrade.
</p><p>The JMS input plugin supports consuming data from any JMS technology into the Elastic Stack by embracing the bring-your-own-driver model, similar to the JDBC plugins. We encourage you to try it out with your own JMS deployments and let us know what you think! Curious what this looks like in action? Glad you asked — <a href="https://www.elastic.co/blog/integrating-jms-with-elasticsearch-service-using-logstash">we have just the blog for you</a>.</p><p><br></p><h2>Feedback</h2><p>Please <a href="https://www.elastic.co/downloads/logstash">download Logstash 7.3.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a>. You can report any bugs or feature requests on the Logstash <a href="https://github.com/elastic/logstash/issues">Github issues page</a> or within the <a href="https://github.com/logstash-plugins">respective plugin repositories</a></p>

Logstash 7.3.0 released

Beats 7.2.0 is here! Learn about scripted processing and various new integrations across the security analytics, cloud native, and Windows ecosystems.

<p>We’re proud to announce the general availability of the Beats 7.2.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/beats">download</a>! This Beats release introduces more processing capabilities along with a myriad of new integrations. Please refer to the <a href="https://www.elastic.co/guide/en/beats/libbeat/7.2/release-notes-7.2.0.html">release notes</a> for the complete list of bug fixes and features.
</p><p>Before upgrading from Beats 6.x, we recommend reviewing the <a href="https://www.elastic.co/guide/en/beats/libbeat/7.2/breaking-changes.html">breaking changes</a> and the <a href="https://www.elastic.co/guide/en/beats/libbeat/7.2/upgrading-6-to-7.html">Beats upgrade guide</a>. If you’re planning to upgrade the entire Elastic Stack, don’t forget to read the <a href="https://www.elastic.co/guide/en/elastic-stack/7.2/upgrading-elastic-stack.html">Elastic Stack upgrade guide</a>.
</p><h2>Scripted Processing at the Edge</h2><p>Event processing is a key component of the data ingestion journey that can be conducted in different layers of the software architecture stack. Although often a server-side responsibility, Elastic Stack users are shifting some lightweight processing and enrichment workloads over to the edge where Beats are collecting the data. Over time, we’ve been grooming a suite of <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filtering-and-enhancing-data.html">processors</a> to equip users with more processing functionality in Beats.
</p><p>With this release, Beats now offer a new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/processor-script.html">script processor</a> for processing events with Javascript code, introducing a whole new level of processing flexibility that was not feasible with existing processors. It also includes an event API that eases the overall event manipulation experience. As Beats often run on host servers, the script processor has been properly sandboxed to only execute <a href="https://www.ecma-international.org/ecma-262/5.1/">ECMAScript 5.1 code</a>. It&nbsp;can therefore only manipulate the event that it’s given and cannot interact with the host or any external services.
</p><h2>More Integrations, More Data, More Insights</h2><p>As&nbsp;lightweight data shippers, we’re continuing to grow the breadth of integrations available in Beats. In this release, Beats has introduced eight new modules and a new Filebeat input across the security analytics, cloud native, and Windows ecosystems.
</p><h3>Security Analytics</h3><p>The Elastic Stack has become a vastly popular platform for security analytics use cases, so enabling wide data ingestion coverage of security data sources and devices is critical for fostering a strong security posture across all the different assets throughout your infrastructure.
</p><p>We’re excited to announce new logging modules for two popular firewall technologies. The <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-panw.html">Palo Alto Networks module in Filebeat</a> monitors PAN-OS firewall logs and the <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-cisco.html">Cisco ASA module in Filebeat</a> monitors Cisco ASA firewall logs. These logs can be received via syslog or extracted directly from file. Additionally, we’ve added a new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-netflow.html">NetFlow module in Filebeat</a> that monitors NetFlow and IPFIX flow records.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5cdd63fe4efe8321/5d07d2fcb0b16b1c530d1129/beats-7-2-0-security-analytics.jpg" data-sys-asset-uid="blt5cdd63fe4efe8321" alt="Security analytics dashboard powered by Beats" "=""></p><p>Beyond these integrations, there’s even more good news for our security analytics users. With the 7.2.0 release, we’ve officially introduced the Elastic SIEM application in Kibana! For more details, check out the <a href="https://www.elastic.co/blog/introducing-elastic-siem">Introducing Elastic SIEM blog post</a>.
</p><h3>Cloud Native</h3><p>With the continued rise of containerization and the cloud, we’re continuing to augment our integrations ecosystem within the cloud native realm. Full details around observability feature updates across the Elastic Stack can be found in the <a href="https://www.elastic.co/blog/elastic-observability-update-7-2-0">Observability 7.2.0 blog post</a>.
</p><p>The <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-nats.html">NATS module</a> is now available in Filebeat for monitoring the NATS messaging system logs. This complements the <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-nats.html">NATS module in Metricbeat</a> that was introduced in Beats 7.0.0. We’ve also added <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-module-coredns.html">CoreDNS modules in Filebeat</a> and <a href="https://www.elastic.co/guide/en/beats/metricbeat/7.2/metricbeat-module-coredns.html">Metricbeat</a> to monitor CoreDNS logs and metrics.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt945cde3c6530437c/5d07d325877575d058475fc5/beats-7-2-0-cloud-native.jpg" data-sys-asset-uid="blt945cde3c6530437c" alt="Cloud logs and metrics powered by Beats" "=""></p><p>Adapting to the evolving world of containers, we’ve introduced a new <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-input-container.html">container input in Filebeat</a> as a more dynamic way of collecting container logs. It supports auto-detection of both Docker and CRI-O log formats. <a href="https://cri-o.io/">CRI-O</a> is an increasingly popular container runtime for Kubernetes. The container input should be used in favor of the existing <a href="https://www.elastic.co/guide/en/beats/filebeat/7.2/filebeat-input-docker.html">Docker input</a>, which is now deprecated.
</p><h3>Windows</h3><p>Windows is an important platform for our users and we enjoy showing love through building stronger integrations with it.
</p><p>Winlogbeat has added two new modules in this release. The <a href="https://www.elastic.co/guide/en/beats/winlogbeat/7.2/winlogbeat-module-sysmon.html">Sysmon module</a> monitors event log records from the Sysinternal System Monitor and the <a href="https://www.elastic.co/guide/en/beats/winlogbeat/7.2/winlogbeat-module-security.html">Security module</a> monitors Windows Security event logs. We’ve also added support for the newer Windows XML Event Log (EVTX) format.
</p><h3>Feedback</h3><p>Please <a href="https://www.elastic.co/downloads/beats">download Beats 7.2.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a>. You can report any bugs or feature requests on the <a href="https://github.com/elastic/beats/issues">GitHub issues page</a>.
</p>

Beats 7.2.0 released

Logstash 7.2.0 is here! Learn about our Java plugins framework and various new integrations like JMS, Elastic App Search, and Google Cloud Storage.

<p>We’re proud to announce the general availability of the Logstash 7.2.0 release. This is the latest stable release and is now available for <a href="https://www.elastic.co/downloads/logstash">download</a>! This Logstash release maintains a core focus on the plugins and integrations ecosystem. Please refer to the <a href="https://www.elastic.co/guide/en/logstash/7.2/logstash-7-2-0.html">release notes</a> for the complete list of bug fixes and features.
</p>
<h2>Welcoming All Java Developers</h2>
<p>Java developers rejoice! With the Logstash 7.2.0 release, support for Java plugins is now GA. This brings first class support for Java plugins where developers can write Logstash plugins with pure Java code, free of any Ruby dependencies.
</p>
<p>The introduction of Java plugins opens the door to a whole new world of Java developers that can now easily contribute to the Logstash plugins ecosystem with their native language. Java plugins will typically have better performance especially when executed in conjunction with the Java execution engine that was enabled by default since 7.0.0. Additionally, these plugins run seamlessly alongside other Ruby plugins in your Logstash pipeline.
</p>
<p>For those interested in writing Java plugins, please take a look at the <a href="https://www.elastic.co/guide/en/logstash/7.2/contributing-java-plugin.html">documentation</a>. We’ve also added some <a href="https://github.com/logstash-plugins/logstash-input-java_input_example">Java plugin examples</a> in Github to help you get started quickly.
</p>
<h2>Consuming from JMS Queues</h2>
<p>Java Message Service (JMS) technologies have been around for a long time and are a key piece of infrastructure in some of our users’ production environments. We’re excited to announce that the Logstash <a href="https://www.elastic.co/guide/en/logstash/master/plugins-inputs-jms.html">JMS input plugin</a> has went through a revamp and is now the recommended option for integrating with and consuming from JMS queues.
</p>
<p>The JMS input plugin supports consuming data from any JMS technology into the Elastic Stack and has been enhanced with more stability, durability, and added support for TLS security over the wire. The plugin embraces the bring-your-own-driver model, similar to the <a href="https://www.elastic.co/guide/en/logstash/7.2/plugins-inputs-jdbc.html">JDBC plugins</a>, which enables a broad coverage of technologies as long as they comply to JMS standards. Although we’ve only tested against Solace, ActiveMQ, and ActiveMQ Artemis, we encourage you to try it out with your own JMS deployments and let us know what you think!
</p>
<p>Please note that this plugin is not bundled by default with Logstash 7.2.0. To use the plugin, you will need to first install it with the Logstash plugin manager using the command below. We plan to bundle this plugin by default moving forward starting with the Logstash 7.3.0 release.
</p>
<pre class="prettyprint">bin/logstash-plugin install logstash-input-jms
</pre>
<h2>Integration with Self-Managed Elastic App Search</h2>
<p>In more good news, <a href="https://www.elastic.co/solutions/app-search">Elastic App Search</a> self-managed has officially gone GA in the 7.2.0 release with more details in this <a href="https://www.elastic.co/blog/elastic-app-search-7-2-0-released">blog post</a> for those that are curious. Alongside this news, Logstash now also supports outputting data to the self-managed version of Elastic App Search with the <a href="https://www.elastic.co/guide/en/logstash-versioned-plugins/current/v1.0.0-plugins-outputs-elastic_app_search.html">Elastic App Search output plugin</a> GA.
</p>
<p>For those with existing custom Elasticsearch application search use cases, Logstash can be used to easily migrate your data over to Elastic App Search. If you have existing data in your data stores or messaging queues that you’d like to unlock for application search, Logstash can also be leveraged to move those datasets into Elastic App Search, regardless of whether it's a fully managed or self-hosted deployment.
</p>
<p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltf3f71561a25d8bd2/5d07cca6561b9b0b537f6bba/logstash-to-elastic-app-search.png" data-sys-asset-uid="bltf3f71561a25d8bd2" alt="logstash-to-elastic-app-search.png">
</p>
<p>Please note that this new plugin version is not bundled by default with Logstash 7.2.0. To use the plugin with self-managed Elastic App Search, you will need to first update it with the Logstash plugin manager using the command below. We plan to bundle this new plugin version moving forward starting with the Logstash 7.3.0 release.
</p>
<pre class="prettyprint">bin/logstash-plugin update logstash-output-elastic_app_search
</pre>
<h2>More Integrations with Google Cloud Platform</h2>
<p>As part of any cloud platform, there are tablestake integrations that are necessary to nurture a tight knit fabric with the Elastic ecosystem. We continue down this road with this release. Logstash now has new and enhanced integrations to read and write from the Google Cloud Storage (GCS) blob store.
</p>
<p>The <a href="https://www.elastic.co/guide/en/logstash/7.2/plugins-inputs-google_cloud_storage.html">GCS input plugin</a> allows users to ingest data from GCS buckets into the Elastic Stack for real-time analytics. Additionally, the <a href="https://www.elastic.co/guide/en/logstash/7.2/plugins-outputs-google_cloud_storage.html">GCS output plugin</a> has gone through a rewrite, leveraging a new Java client for improved performance and stability. We’d like to thank the Google Graphite team for their great efforts with these contributions.
</p>
<h2>Feedback</h2>
<p>Please <a href="https://www.elastic.co/downloads/logstash">download Logstash 7.2.0</a>, try it out, and let us know what you think on Twitter (<a href="https://twitter.com/elastic">@elastic</a>) or in our <a href="https://discuss.elastic.co/">forums</a>. You can report any bugs or feature requests on the Logstash <a href="https://github.com/elastic/logstash/issues">Github issues page</a> or within the <a href="https://github.com/logstash-plugins">respective plugin repositories</a>.
</p>

Logstash 7.2.0 released

With Functionbeat, it's never been easier to ingest serverless cloud data - like Cloudwatch Logs - into Elasticsearch. Learn about our new FaaS data shipper.

<html><head></head><body><p>Serverless computing, or Function-as-a-Service (FaaS), has been a trending architecture pattern in many cloud deployments, often as a complement to existing container, VM, and cloud infrastructures. It enables applications to be run as functions on demand, further simplifying deployments when long-running infrastructure isn&#x2019;t required, and potentially reducing operational cost with users only being charged when functions are being executed.
</p><p>With more workloads moving into the cloud, it has become increasingly important to harness the ability to easily ingest cloud data in a scalable, reliable, and cost efficient fashion. Serverless compute services like AWS Lambda, Azure Functions, and GCP Functions enable users to take the &#x201C;Ops&#x201D; out of DevOps, and the introduction of <a href="https://www.elastic.co/products/beats/functionbeat">Functionbeat</a> now further takes the &#x201C;Dev&#x201D; part out as well, leaving you with a notably simple experience for ingesting cloud data into Elasticsearch or <a href="https://www.elastic.co/cloud">Elastic Cloud</a>. The <a href="https://www.elastic.co/cloud/elasticsearch-service">Elasticsearch Service</a> in Elastic Cloud is the only hosted Elasticsearch offering that is built and run by Elastic, the company behind Elasticsearch.
</p><h2>Introducing Functionbeat</h2><p>Functionbeat is a new addition to the <a href="https://www.elastic.co/products/beats">Beats product suite</a> that can be easily deployed as a function in serverless compute platforms, providing everything you need to configure, deploy, and transform events like logs and metrics coming from your cloud infrastructure. Available as of 6.5 as a beta, it currently supports AWS Lambda and a subset of data sources, namely AWS Cloudwatch Logs and Simple Queue Service (SQS). We have plans to expand support across more cloud providers and data sources as we move forward. Functionbeat is offered under the <a href="https://www.elastic.co/subscriptions">Elastic Basic license</a> which means its freely available for all end users.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt22fcb021761f0796/5c304739682c4c4866a7980e/functionbeat-1.png" data-sys-asset-uid="blt22fcb021761f0796" alt="" style="display: block; margin: auto;">
</p><h3>Ingest from Cloud Data Sources</h3><p>With flexibility in mind, Functionbeat opens up endless opportunities to effectively ingest, transform, enrich, and ship your cloud data to Elasticsearch. Here are two use cases in which you may leverage Functionbeat to accomplish:
</p><ul>
	<li><strong>Monitor AWS with the Elastic Stack</strong> - Functionbeat can receive events from a Cloudwatch Log group, extract relevant fields with the <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/dissect.html">dissect processor</a> to structure the event, and <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/drop-event.html">apply filtering</a> prior to shipping to an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.5/ingest.html">Ingest Node</a> pipeline or directly to Elasticsearch.</li>
	<li><strong>Analyze Application Data from SQS with the Elastic Stack</strong> - for applications sending JSON encoded events to an SQS queue, Functionbeat can be used to listen, ingest, and <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/decode-json-fields.html">decode the JSON event</a> prior to shipping to Elasticsearch.</li>
</ul><h3>Define Processing Logic without Code</h3><p>At the core, Functionbeat operates like any other Beat, using the same underlying framework as <a href="https://www.elastic.co/products/beats/filebeat">Filebeat</a> and <a href="https://www.elastic.co/products/beats/metricbeat">Metricbeat</a>, and exposing similar functionality that you&#x2019;ve already come to know and love. This means that you can add fields to your events, use <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/defining-processors.html#conditions">conditionals</a> to drop or route events, or use any <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/defining-processors.html">processors</a> to manipulate or enrich your data before sending it to Elasticsearch. What&#x2019;s great is that all of this can be achieved through a configuration style experience without the need to actually write and compile code!
</p><h2>Ingesting Cloudwatch Logs in the Real World</h2><p>Let&#x2019;s say we want to send a subset of log messages from Cloudwatch Logs to our Elasticsearch cluster, and to keep the event count reasonable, we are only interested in messages containing the <code>ERROR</code> keyword. We edit the <code>functionbeat.yml</code> and add the following function configuration for a <code>mylogs</code> Lambda function that will be triggered when new events are generated from the Cloudwatch Logs group <code>/aws/lambda/test</code> matching the <code>ERROR</code> pattern:
</p><pre class="prettyprint">functionbeat.provider.aws:
  deploy_bucket: myappdeploybucket
  functions:
    - name: mylogs
      enabled: true
      type: cloudwatch_logs
      concurrency: 5
      description: &quot;A really special lambda&quot;
      fields:
        level: &quot;error&quot;
      triggers:
        - log_group_name: /aws/lambda/test
          filter_pattern: &quot;ERROR&quot;
output.elasticsearch:
  enabled: true
  hosts: [&quot;localhost:9200&quot;]
  username: &quot;elastic&quot;
  password: &quot;changeme&quot;
</pre><p>We can now deploy that function to AWS Lambda, but before proceeding we should make sure the AWS environment variables are correctly configured. The function can be pushed into Lambda using the <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/command-line-options.html#deploy-command">deploy</a> subcommand:
</p><pre class="prettyprint">./functionbeat deploy mylogs
</pre><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt7dcd99036ad35eea/5c3047377c04885167e04da1/functionbeat-2.png" data-sys-asset-uid="blt7dcd99036ad35eea" alt="" style="display: block; margin: auto;">
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blta3ef73adf7f267e5/5c3047355a7621286e3e904b/functionbeat-3.png" data-sys-asset-uid="blta3ef73adf7f267e5" alt="" "="">
</p><p>After the Lambda function is deployed, new Cloudwatch Logs events added to the log group that match the <code>ERROR</code> pattern will be automatically ingested and available in our Elasticsearch cluster. If this is not the case, we can look up the Functionbeat logs in our Cloudwatch Logs panel to debug any issues. Functions can be updated or completely removed from Lambda at anytime using the Functionbeat CLI <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/command-line-options.html#update-command">update</a> and <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/command-line-options.html#remove-command">remove</a> subcommands.
</p><h2>Scaling Concurrency</h2><p>The unit of scale in AWS Lambda is the concurrency and a hard limit of 1000 is used for the concurrency by default. This limit is account wide and represents the maximum number of functions of any kind that can be executed concurrently.
</p><p>The Lambda service will autoscale your functions based on the load and concurrency levels configured for a function and is a good way to control cost and regulate the time it takes to process and execute this work. By default, Functionbeat will use a conservative setting of 5 as the concurrency level for any function it creates, but this can be tuned to more strongly fit the requirements of your use case and scale.
</p><h2>Let&#x2019;s Get the Party Started!</h2><p>Now that we understand what Functionbeat is and the problems it solves, we encourage you to go try it out for your own serverless ingest use cases! Although it&#x2019;s initially been released in beta, we&#x2019;re actively working towards making it GA production ready. In the meantime, we&#x2019;re curious to hear your feedback, so please <a href="https://github.com/elastic/beats/issues/new">submit a Github issue</a> (all our code is open!) or post your thoughts in the <a href="https://discuss.elastic.co/c/beats/functionbeat">Functionbeat forum</a>. Check out the <a href="https://www.elastic.co/guide/en/beats/functionbeat/6.5/functionbeat-overview.html">Functionbeat documentation</a> to get started!
</p></body></html>

functionbeat-blog-banner.png

functionbeat-blog-thumb.png

Functionbeat: Serverless Ingestion for Elasticsearch

New in 6.5, learn how to manage all of your Beats from one location with the new Beats Central Management UI in Kibana.

<html><head></head><body><p>In the world of data analytics, everything starts with data collection. This means that data needs to be ingested quickly, reliably, and it must be structured appropriately so it&#x2019;s available immediately for analytical use. The ability to do exactly that, regardless of the size, shape, or speed of your data set, is why the <a href="/products/beats">Beats</a> product suite is now the core data shipper for many popular use cases. And with many thanks to our community, it continues to expand as we build more off-the-shelf solutions like <a href="/guide/en/beats/filebeat/6.5/filebeat-modules.html">modules</a> and the <a href="/guide/en/infrastructure/guide/6.5/infrastructure-ui-overview.html">Infrastructure UI</a>. With this growth of overall adoption and per deployment footprint, it has become increasingly important for users and enterprises to securely achieve operational excellence, especially when running Beats at scale.
</p><p>By nature, Beats are often deployed on the edge where deployments can easily span into the thousands. With this tier of scale, monitoring and management become two key pillars of operational excellence. In 6.2, we released <a href="/guide/en/kibana/6.5/beats-page.html">Beats monitoring</a> to provide deep visibility into overall deployment health and status. Most recently in 6.5, we are proud to introduce Beats central management which provides a centralized Kibana UI for managing and updating Beats configurations. This feature aims to drastically reduce users&#x2019; reliance on external configuration management tools like Ansible, Puppet, or Chef, enabling Beats configuration changes to be managed directly with the Elastic Stack (namely Kibana and Elasticsearch) and rolled out automatically to entire fleets of Beats. Have you heard that modules can now be managed centrally too?
</p><h2>Beats Central Management</h2><p>Beats centralized management is available in 6.5 as a beta feature. With security posture top-of-mind, this feature is available under the <a href="/subscriptions">Elastic Gold license</a>, or Standard license when using our <a href="/cloud">Elastic Cloud service</a>, to ensure a properly secured deployment. It comes inclusive of a new Beats central management UI in Kibana and leverages Elasticsearch as a centralized configuration store. In the near future, we also plan to expose an API for easier integration with external tools and systems.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte1d5938f1b40261b/5c3047c1ebb7714d6602701c/central-beats-1.png" data-sys-asset-uid="blte1d5938f1b40261b" width="500"><br>
</p><h3>The new interface </h3><p>Core to the central management experience, is the new Beats central management UI. Found in the <em>Management</em> tab of Kibana, this new interface allows users to easily and securely enroll Beats into the central management workflow. After enrollment, these Beats and their configurations can then be managed and automatically orchestrated directly through the interface, regardless of whether it&apos;s one Beats or thousands.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8f20d146d478e135/5c3047bf25a401fc6fe4f489/central-beats-2.jpg" data-sys-asset-uid="blt8f20d146d478e135" alt="" "="">
</p><h3>New enroll command on Filebeat and Metricbeat </h3><p>To enable control of a Beat to central management, <a href="/products/beats/filebeat">Filebeat</a> and <a href="/products/beats/metricbeat">Metricbeat</a> (with others to follow) now have a new enroll command that is used to enroll your Beat with a token generated in our UI, or via a username and password of a restricted use user. This therefore allows easy and simple integration regardless of your workflow!
</p><h3>Beats, now with Configuration Tags </h3><p>Managing the intricacies of a large scale deployment can be challenging. Thus for central management we introduced the concept of <em>tags</em> for Beats configurations. Tags are a collection of separate chunks of configuration data, allowing you to share configuration between Beats in whole or in part. We felt this concept will help to keep things both organized and flexible to multiple use cases.
</p><h3>Security </h3><p>Anytime you introduce a management tool, there is the potential for additional attack vectors. Thus we have taken additional security steps to ensure your infrastructure is safe when using Beats central management, such as:
</p><ul>
	<li>Unique authentication tokens per Beat, ensuring that one compromised edge node will not compromise your stack</li>
	<li>Additional user roles to allow control over who can access central management</li>
	<li>Auto-expiring enrollment tokens</li>
	<li>And of course, Elasticsearch authentication and authorization mechanisms</li>
</ul><h2>Feedback Welcome (and Encouraged)</h2><p>The ability to monitor and manage your Beats deployments through a single pane of Kibana glass is an experience that&#x2019;s worthwhile to explore and we hope you give it a spin. Although the Beats central management feature is currently in beta, we&#x2019;re actively working towards making it GA production ready. In the meantime, we&#x2019;re curious to hear your feedback (please <a href="https://github.com/elastic/beats/issues/new">submit a Github issue</a>, all our code is open!) on the feature as well as what other Beats (including <a href="/guide/en/beats/libbeat/6.5/community-beats.html">Community Beats</a>) you&#x2019;d like to see supported next. Check out the <a href="/guide/en/beats/filebeat/6.5/configuration-central-management.html">Beats central management documentation</a> to get started!
</p></body></html>

central-beats-banner.jpg

central-beats-thumb.jpg

Introducing Beats Central Management in the Elastic Stack

<p>“We’re moving to the cloud!” is now a commonly heard statement that was greeted with skepticism just a couple years ago. The adoption of the cloud has been growing at an astronomical pace. Although this growth is exciting, possessing deep visibility into your cloud infrastructure and application health now naturally becomes an important pillar of operational excellence for many users and companies, especially for mission critical services. The Elastic Stack is already a popular platform choice for monitoring and metrics use cases, and we’ve now made it much easier to monitor your Microsoft Azure deployments.
</p><h2>The Azure Monitoring Module</h2><p>We’re pleased to introduce the Azure monitoring module, now available in Logstash 6.4. The module unlocks instant operational insights into your Azure user activity, infrastructure changes, and Azure SQL Database health across multiple subscriptions. It offers a suite of Kibana dashboards that enable you to detect issues earlier and investigate faster, right out of the box. Significantly expand the overall observability of your Azure Cloud environments by identifying compromised or malicious users, problematic resource groups, or SQL DBs running at unusually high workloads.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt084a1c6d90bd453c/5c57df799cd9dcf90b7451fc/azure_monitoring_arch_600.png" data-sys-asset-uid="blt084a1c6d90bd453c" alt="azure_monitoring_arch_600.png" style="display: block; margin: auto;">
</p><p>Elastic and Microsoft have collaborated to bring you a turnkey solution for monitoring the Azure Cloud. The module integrates with <a href="https://docs.microsoft.com/en-us/azure/monitoring-and-diagnostics/monitoring-overview-azure-monitor">Azure Monitor</a>, centrally collecting and analyzing activity logs and SQL DB diagnostic logs and metrics through Azure Event Hubs. Interested in running Elasticsearch on Azure? Check out this blog post to learn <a href="/blog/deploying-elasticsearch-on-microsoft-azure">how to deploy Elasticsearch on Azure</a> in a couple minutes with <a href="https://azuremarketplace.microsoft.com/marketplace/apps/elastic.elasticsearch">official templates in the Azure Marketplace</a>.
</p><h3>Azure Activity Log Monitoring</h3><p>The core integration of this module is with the activity log, which tracks the overall activity of an Azure subscription. The module bundles Kibana dashboards for administrators to quickly understand the activity landscape across an entire deployment, with dedicated monitoring dashboards for user and alert activity. Quickly uncover suspicious activities like brute force attacks on a user login, unexpected infrastructure creations / deletions, or high alert density from a particular resource group. These are indeed Kibana dashboards, so you not only have the power of Kibana visualizations at your fingertips, but also the flexibility to edit, augment, and even add new dashboards to tailor tightly with your business requirements.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9c716e1d9f1e1a47/5c57df7485820dce0bcc2c27/azure_activity_log_monitor.jpg" data-sys-asset-uid="blt9c716e1d9f1e1a47" alt="azure_activity_log_monitor.jpg" style="display: block; margin: auto;">
</p><h3>Azure SQL Database Monitoring</h3><p>Additionally, we also offer a dedicated experience for monitoring your Azure SQL Database deployments. Keep a pulse on your database health and performance with the module through key real-time metrics like DTU utilization, errors, disk usage, and wait times. Deep dive into query execution and performance to better understand and resolve run-time bottlenecks.
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt8c1b0a345684a351/5c57df6e09024fdb0b3c81ac/azure_sql_database_monitor.jpg" data-sys-asset-uid="blt8c1b0a345684a351" alt="azure_sql_database_monitor.jpg" style="display: block; margin: auto;">
</p><h2>New Azure Event Hubs Input Plugin</h2><p>The Azure monitoring module leverages the new Logstash <a href="/guide/en/logstash/current/plugins-inputs-azure_event_hubs.html">Azure Event Hubs input plugin</a>. This plugin enables Logstash to scalably consume and process data from any Event Hub for real-time analytics with the Elastic Stack. Event Hubs and other messaging queues are often key architectural hubs for centralized data ingestion and resilient streaming ETL. For product feedback, please feel free to file an issue in the <a href="https://github.com/logstash-plugins/logstash-input-azure_event_hubs">plugin Github repo</a>.
</p><h2>Free to Use and Easy to Upgrade</h2><p>The Azure monitoring module has initially been introduced as an experimental feature and is free to use under the <a href="/subscriptions">Elastic Basic License</a>. On top of that, users have the option to upgrade their Elastic deployment with a subscription that includes additional powerful features like security controls, alerting, monitoring, reporting, Graph analytics, and machine learning. Users can quickly achieve an automated monitoring platform by creating <a href="/products/stack/alerting">alerts</a> for automated notifications and <a href="/products/stack/machine-learning">machine learning</a> jobs for anomaly detection and predictive forecasting.
</p><p>Monitoring your Azure deployments with the Elastic Stack has never been easier. Check out the Logstash <a href="/guide/en/logstash/current/azure-module.html">Azure monitoring module documentation</a> to get started!
</p><p>For more information:
</p><ul>
	<li>More details on the <a href="/logstash">Logstash</a> product</li>
	<li>Other available module integrations for <a href="/guide/en/beats/filebeat/current/filebeat-modules.html">Filebeat</a>, <a href="/guide/en/beats/metricbeat/current/metricbeat-modules.html">Metricbeat</a>, and <a href="/guide/en/logstash/current/logstash-modules.html">Logstash</a></li>
	<li><a href="/elasticon/conf/2018/sf/monitor-your-cloud-environment-with-the-logstash-azure-monitoring-module">Azure monitoring module demo</a> from <a href="/elasticon/conf/2018/sf/">Elastic{ON} 2018</a></li>
	<li>Learn more about Microsoft on Elastic’s <a href="/about/partners/technology">technology partner</a> page</li>
	<li><a href="https://azuremarketplace.microsoft.com/marketplace/apps/elastic.elasticsearch">Azure Marketplace Elastic template</a></li>
	<li>Overview of <a href="https://docs.microsoft.com/azure/monitoring-and-diagnostics/monitoring-overview-azure-monitor">Azure Monitor</a> and <a href="https://docs.microsoft.com/azure/event-hubs/event-hubs-about">Event Hubs</a></li>
</ul><p>If you prefer email, feel free to drop us any questions or feedback at <a href="mailto:mailto:monitor-azure@elastic.co">monitor-azure@elastic.co</a>.
</p>

azure_mon_banner2.jpg

azure_mon_thumb.png

Azure Cloud Monitoring with the Elastic Stack

The upcoming Logstash Azure Monitoring module will be the easiest way to monitor your Azure infrastructure and services with the Elastic Stack.

<html><head></head><body><p>At Elasticon 2018, we introduced the upcoming Logstash Azure Monitoring module, which will be the easiest way to monitor Azure infrastructure and services with the Elastic Stack. It comes with the turnkey experience you have come to expect with <a href="/guide/en/logstash/current/logstash-modules.html">Logstash</a> and <a href="/guide/en/beats/filebeat/current/filebeat-modules-overview.html">Beats</a> modules - centralized log and metric ingestion, automated processing and enrichment, and a suite of packaged Kibana dashboards for immediate monitoring exploration.
</p><figure><center><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt52a3406ef8fed3f6/5c57edccbfa899f60ba15b8b/azure-monitor-1.png" data-sys-asset-uid="blt52a3406ef8fed3f6" alt="azure-monitor-1.png" width="650"></center><figcaption>Architecture</figcaption>
</figure><p>With this module, you will have the ability to monitor your Azure subscriptions in various ways:
</p><ul>
	<li>Analyze infrastructure changes and access request activity</li>
	<li>Identify suspicious behaviors and potential malicious actors</li>
	<li>Perform root-cause analysis by investigating user activity</li>
	<li>Monitor and optimize SQL DB deployments</li>
</ul><figure>
&#xA0; <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt79be38ba23ea776c/5c57edc69cd9dcf90b745254/azure-monitor-2.png" data-sys-asset-uid="blt79be38ba23ea776c" alt="azure-monitor-2.png">
<figcaption>Activity Overview</figcaption>
</figure> <figure>
&#xA0; <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt13404c8bcbd2a6f1/5c57edc0dfaabbd40b9c29d0/azure-monitor-3.png" data-sys-asset-uid="blt13404c8bcbd2a6f1" alt="azure-monitor-3.png">
<figcaption>SQL DB Overview</figcaption></figure><p>The Logstash Azure Monitoring module will be coming soon. In the meantime, if there are questions or feedback on the integration, please reach out to <a href="mailto:monitor-azure@elastic.co">monitor-azure@elastic.co</a>. The initial release of the module will include Kibana dashboards for the Azure activity log and SQL DB. As we continue to grow this integration over time, we want to make sure we&#xA0;build&#xA0;what our community needs. If there are additional Azure integrations you would like to see going forward, please do take a minute to fill out our short <a href="http://go.es.io/2F1H6p3">three question survey</a>. Appreciate your time in advance, and looking forward to hearing from you!
</p></body></html>

azure-monitor-4.png

Monitoring Azure with the Elastic Stack - Feedback Welcome

We're incredibly proud to announce the best Logstash ever with Logstash 6.0.0!

<html><head></head><body><p>We&#x2019;re&#xA0;glad to announce that Logstash 6.0.0 has launched! Today marks the first day of 6.0&#x2019;s inter-planetary mission of making life easier for systems administrators and engineers. Can&#x2019;t wait another minute to try it? Head over to our <a href="https://www.elastic.co/downloads/logstash">downloads page</a>&#xA0;and give it a shot! However, you may want to take a minute to read about <a href="https://www.elastic.co/guide/en/logstash/6.0/breaking-changes.html">breaking changes</a>&#xA0;and <a href="https://www.elastic.co/guide/en/logstash/6.0/logstash-6-0-0.html">release notes</a>&#xA0;first. Read on for what&apos;s new in Logstash 6.0!</p><h3><span></span>Streamline Processing with Multiple Pipelines</h3><p>One pipeline no longers rules them all. Logstash 6.0 introduces the ability to run multiple pipelines concurrently for different use cases. The pipelines run together in the same instance, but with independent inputs, filters, and outputs to enable users to isolate processing logic per data source. This keeps your pipeline logic focused and concise. Historically, many Logstash users combine multiple use cases into a single pipeline which requires adding complex conditional logic. With multiple pipelines that&#x2019;s no longer necessary; now, you can organize your config more cleanly, and execute your pipeline more efficiently by using dedicated pipelines per use case.</p><p><span id="docs-internal-guid-6fb5cd00-9bd5-5f01-6af3-fe5ada7ab7d6"><img src="https://lh5.googleusercontent.com/KZ-SFO9bj48u2RAc0Q9unQEHO1ILxNxKHUFxueGa4qD1z-MTkIm3QqPIZk3hwfKTZWHuvgH5gqIAI4gExD4Ps3x5F59zdDzQb_LU6Tj7HqRg_0llDkaGctnAwaHeLT9kCPFgcYKD" width="271" height="237"></span><br></p><p>To add to the fun, each pipeline also has its own independent settings and lifecycles which you can tune to match the respective workload profile desired for that data source. For instance, you may want to allocate more pipeline worker threads for a high volume logging pipeline, and throttle back resources for a lower intensity local metrics pipeline. For more details on multiple pipelines, take a peek at the <a href="https://www.elastic.co/guide/en/logstash/6.0/multiple-pipelines.html">documentation</a>&#xA0;and <a href="https://www.elastic.co/blog/logstash-multiple-pipelines">blog post</a>!</p><h3>Manage Pipelines Centrally with the Elastic Stack</h3><p>In the past, managing pipeline configurations was either a manual task, or you would use config management tools like Puppet or Chef to assist with operational automation. With Logstash 6.0, the centralized pipeline management feature now enables you to manage and automatically orchestrate your Logstash deployments directly with the Elastic Stack through the Kibana single pane of glass.&#xA0;</p><p><span id="docs-internal-guid-6fb5cd00-9bd5-a75d-14b5-bbc290fab71f"><img src="https://lh3.googleusercontent.com/EYcpyojtY6NyN0KYc1O0iaxxjUhN-lIBITnjbqHE-abEerFSbp1LdChRnwcbPV7UHRdADWolhjSS-JdXoXBKqEUd75v5S547rICgwA6jtIOc5VCFireOyCYoAhGYMMlFlesH5deO" width="385" height="310"></span><br></p><p>This feature brings a Pipeline Management UI to Kibana, which you can use to create, edit, and delete pipelines. Underneath this UI we use Elasticsearch to store your pipeline configurations. With a few simple settings, your Logstash nodes can be configured to watch for changes on these pipelines, letting you seamlessly push out pipeline changes without additional operational infrastructure beyond the Elastic Stack.</p><p>Centralized pipeline management is available as an X-Pack feature. To learn more, take a look at the <a href="https://www.elastic.co/guide/en/logstash/6.0/logstash-centralized-pipeline-management.html">documentation</a>&#xA0;and the <a href="https://www.elastic.co/blog/logstash-centralized-pipeline-management">blog post</a>!</p><h3>Visualize&#xA0;Pipeline Logic and Performance</h3><p>We&#x2019;re also proud to announce the long awaited <a href="https://www.elastic.co/guide/en/logstash/6.0/logstash-pipeline-viewer.html">Pipeline Viewer</a> in this 6.0 release, which is a welcome addition to the Logstash Monitoring UI. With this new tool you can both visualize your pipeline configuration and troubleshoot performance bottlenecks at the plugin level. The pipeline viewer displays your Logstash pipelines as a DAG (Directed Acyclic Graph). Overlaid on this DAG are relevant performance metrics for individual inputs, filters, and outputs. Potential performance problems are highlighted to let you quickly determine which portions of your pipeline may be bottlenecks.</p><p>Users should note that this is a beta feature and may be subject to change. One known issue is the lack of ability to cleanly render very large pipelines. This is an area of active work, so expect significant improvements to this feature as we improve it further and move it into general availability.</p><p><span id="docs-internal-guid-6fb5cd00-9bd5-e64a-a40c-8d933b4c0e0d"><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; vertical-align: baseline; white-space: pre-wrap;"><img src="https://lh5.googleusercontent.com/bUi05zoXkD9Izjtkews24YdMhBSSxq8fYihHRbs5TaSW9pnWJdFNho-zJfwaW74oGqGVFUbHE0vJ1Hps0BhlNA-ny5H9nDOdkc2kjjyNrSomx7n_cdLKGb303PwmKzuc7RF-I5Bd" width="624" height="483" style="border-width: initial; border-style: none; transform: rotate(0rad);"></span></span><br></p><h3>A Smooth Path from Ingest Node to Logstash</h3><p>Some users enjoy the convenience of Elasticsearch Ingest Nodes when getting started with the Elastic Stack. However, at a certain level of complexity, Ingest Nodes may&#xA0;not have all the features required to solve your problem&#xA0;and a migration to Logstash may&#xA0;be in order. To ease these transitions we&#x2019;ve created an <a href="https://www.elastic.co/guide/en/logstash/6.0/ingest-converter.html">Ingest pipeline conversion&#xA0;tool</a>&#xA0;that ships with Logstash 6.0. The converter takes an Ingest Node pipeline as an input and spits out the&#xA0;respective Logstash pipeline. See the <a href="https://www.elastic.co/blog/ingest-node-to-logstash-configuration-converter">blog post</a> for more insight!</p><h3>Now With&#xA0;JRuby 9k</h3><p>We&#x2019;ve spent a lot of time moving the Logstash project over to the latest major version of JRuby: JRuby 9000, which has support for modern Ruby syntax and enhanced internals more amenable to optimization. For plugin developers, this means that your old plugins will continue to work in all but the most rare of cases. This upgrade also means that you can use new Ruby features going forward, as well as using Ruby libraries only compatible with JRuby 9k.</p></body></html>

blog-logstash-banner.jpg

blog-logstash-thumb.jpg

Logstash 6.0.0 GA Released

The new Elastic Stack ArcSight integration makes it easier than ever for ArcSight users to explore their security event logs in the Elastic Stack.

<h3>Introduction</h3><p>If you've been following our engineering blogs for the past 9 months or so (see Editor's note at the end for links), you'll recall that we've posted a half dozen examples of how to use the Elastic Stack to augment the ArcSight SIEM by taking a copy of the ArcSight CEF-formatted event data stream, ingesting it into the Elastic Stack, and then searching, analyzing, visualizing, and alerting on these events using various Elastic products and features.
</p><p>We've had such a good response to the blog series, that we decided to make the integration even better and easier to get started. Today, we are very excited to introduce the Logstash ArcSight module that simplifies the Elastic Stack ArcSight integration down to one command. Run one command and start exploring your ArcSight events in Kibana in literally minutes.
</p><p>Before we dig into the details of this new Logstash module, let's take a look at why this integration with ArcSight makes sense.
</p><h3>Security Analytics is a Growing Use Case for the Elastic Stack</h3><p>The security threat landscape has fundamentally changed in the past 5 years. While traditional methods usually involved searching for known bad activity in their log data, today's analysts often need to interactively investigate wherever the data leads them in real time. Additionally, the volumes of security-relevant log data that used to be measured in TB per day are now growing to PB per day. Depending on their specific role, security teams may need tools and platforms optimized for their particular workloads. Factors affecting platform requirements include:
</p><ul>
	<li>Security Mission - Interactive threat hunting vs. continuous threat monitoring</li>
	<li>Retention Timeframe: Short-term vs. long-term</li>
	<li>Incident Response Workflow - Native Integrated vs. downstream integration</li>
	<li>Tactical - Known bad detection vs. anomaly detection</li>
</ul><p>Necessarily, there is not a one-size-fits-all solution for&nbsp;log-based security analytics.
</p><p>Driven by these factors, security analytics has become one of the fastest growing use cases for the Elastic Stack. The Elastic Stack is integrated in popular open-source security projects like <a href="https://isc.sans.edu/forums/diary/Adversary+hunting+with+SOFELK/22592/">SOF-ELK</a>, <a href="http://blog.securityonion.net/2017/07/towards-elastic-on-security-onion.html">Security Onion</a>, and <a href="http://rocknsm.io/">Rock-NSM</a>, and is featured in <a href="https://www.sans.org/course/siem-with-tactical-analytics">SANS</a> security training courses. The recent introduction of machine learning capabilities to the Elastic Stack (via X-Pack) are enabling even more automated detection of attack behaviors for security analysts, as described in this <a href="/blog/using-machine-learning-and-elasticsearch-for-security-analytics-deep-dive">deep dive blog post</a>.&nbsp;
</p><h3>Good Security Analytics Requires "Good Data"</h3><p>Security operations teams have learned over the years that their ability to detect threats and respond to cyber incidents depends upon having "good data" to monitor and analyze. Good data means that the security events generated by systems and devices have been normalized into a common format with common field names, and filtered so that visualization, correlation, and alerting can be implemented efficiently across multiple data sources.
</p><p>Easier said than done! Collecting "good" security data is hard. In fact, <a href="http://blogs.gartner.com/anton-chuvakin/2017/04/11/why-your-security-data-lake-project-will-fail/">in an April 2017 post</a>, security industry analyst Anton Chuvakin cites "dirty data" and "trouble with collecting data" as top reasons for failure in security analytics or security data lake initiatives. He comments on the difficulty of the data collection problem, noting that SIEM vendors have spent 10+ years debugging and innovating their "data collectors" for a reason.
</p><p>Users of the Elastic Stack already address the "good data" challenge with <a href="/products/beats">Beats</a>, our suite of specialized open source data shippers. They install as lightweight agents and send data from hundreds or thousands of machines to Logstash or Elasticsearch.
</p><p>We wanted to make it easy for ArcSight customers who have deployed a security event collection layer using ArcSight Smart Connectors (ArcSight supports connectors for 350-400 devices/systems) to get their security event data into the Elastic Stack to enable new ways to search, analyze, and visualize their data.
</p><p>Last year, ArcSight introduced the ArcSight Data Platform (ADP) which embraces an open architecture and lets ArcSight users send their CEF-formatted security event data to a variety of destinations beyond the ArcSight Logger and ESM SIEM products. The ADP also added a queuing system, called the ArcSight Event Broker, which provides a mechanism for multiple systems to pull security event data from the same queue, therefore enabling a "collect once, use everywhere" paradigm.
</p><p>Our new Logstash ArcSight module makes this integration easy, whether users are using Smart Connectors or the Event Broker, to enjoy powerful downstream capabilities of real-time search, analytics, visualizations, and advanced security analytics with the Elastic Stack.
</p><h3>The Elastic Stack ArcSight Integration</h3><p>OK, now that we understand the motivation behind this integration, let's dig into the details. As we've mentioned, the Elastic Stack ArcSight Integration enables ArcSight users to send a copy of their normalized security event data to the Elastic Stack for data exploration and analysis with Kibana visualizations and dashboards.
</p><p><span id="docs-internal-guid-f295dea6-72d5-d783-a483-3f79e82024cd"><span style="font-size: 11pt; font-family: Arial; color: rgb(0, 0, 0); background-color: transparent; vertical-align: baseline; white-space: pre-wrap;"><img src="https://lh6.googleusercontent.com/f7W9mbAcV67TiSye1gA-VBnbwsJE4VJ2LSXUpU0ghi9cUuOFVupPOERyTDPIjszVSr0hRsB-P1Ixug_E1XfixMRPu4EHbpsWoCS9ODg4PGE4cFI7vxXvO49zDQkmKoR1RX_EnA5W" width="624" height="263" alt="Screen Shot 2017-09-10 at 2.36.02 PM.png" style="border-width: initial; border-style: none; transform: rotate(0rad);"></span></span><br>
</p><p>The new Elastic product feature is called the Logstash ArcSight module. The module includes a Logstash configuration for ingesting and enriching CEF-formatted data from either Smart Connectors or the ADP Event Broker, while bundling a set of Kibana dashboards to view events from common sources. Oh, and it all installs with just one Logstash command.
</p><div class="video embed-container" style="height: 319.725px;">
	<img style="width: 100%; margin: auto; display: block;" class="vidyard-player-embed" src="https://play.vidyard.com/QsqH7mZ1a8ageWgnuj3G9m.jpg" data-uuid="QsqH7mZ1a8ageWgnuj3G9m" data-v="4" data-type="inline">
</div><p>Available starting with the Logstash 5.6 release, the module taps directly into the ArcSight data stream, parses and indexes the security events into Elasticsearch, and installs a set of Kibana dashboards to get you immediately exploring your network, firewall, endpoint, and Windows DNS datasets. ArcSight customers will be able to use the Logstash module to feed their data into the Elastic Stack and start exploring it within minutes.
</p><h3>Free to Use and Easy to Upgrade
</h3><p>The Logstash ArcSight module is free to use. Use of this module requires the installation of X-Pack and a free Basic license from Elastic. Plus, users have the option to upgrade their Elastic deployment with a subscription that includes additional powerful features like security controls, alerting, monitoring, reporting, graph analytics, and machine learning. (Please note that X-Pack features are only available from Elastic).
</p><p>We're excited about this integration and our technology partnership with the ArcSight folks, but it's important to realize what this partnership is not. On the commercial side, neither Micro Focus nor Elastic will sell or distribute each other's products or services. From a technology perspective, customers will be responsible for their integration.
</p><p>Security analytics with Elastic Stack and ArcSight together offers the best of both worlds. See more, act faster, secure it all. Check out the <a href="https://www.elastic.co/guide/en/logstash/5.6/arcsight-module.html">Logstash ArcSight module documentation</a> to get started!
</p><p>For more information:
</p><ul>
	<li>Check out the new <a href="/arcsight">Elastic Stack ArcSight integration page</a></li>
	<li>More details on the <a href="/products/logstash">Logstash</a> product</li>
	<li>Register for our <a href="/webinars/elastic-stack-arcsight-integration">upcoming webinar</a> on the topic</li>
	<li>This integration listed on the <a href="https://marketplace.microfocus.com/arcsight/content/logstash-arcsight-module">ArcSight Marketplace</a></li>
	<li>Learn more about <a href="https://software.microfocus.com/en-us/software/siem-data-collection-log-management-platform">Micro Focus</a> on Elastic's <a href="/about/partners/technology">technology partner</a> page</li>
	<li>Read our <a href="/about/press/elastic-delivers-arcsight-integration-for-flexible-scalable-and-real-time-security-analytics-capabilities">press release</a></li>
</ul><p>If you prefer email, just drop us a line at <a href="mailto:arcsight@elastic.co">arcsight@elastic.co</a>
</p><p>Editor's Note: Be sure to check out this related 6-part blog series. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-1">Part 1</a> kicks off the series with getting started content. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-2">Part 2</a> continues the story with how to proactively monitor security data in Elasticsearch using X-Pack. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-3">Part 3</a> walks you through how to scale the architecture. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-4">Part 4</a> and <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-5">Part 5</a> provide examples of setting up alerts for common security threats using the alerting features in X-Pack. <a href="/blog/integrating-elasticsearch-with-arcsight-siem-part-6">Part 6</a> extends the alerting story with automated anomaly detection using X-Pack machine learning.
</p>

ArcSight_Promo3_Blog 1385x400 .png

ArcSight_Promo3_Blog Thumbnail 720x420.png

Introducing the Elastic Stack ArcSight Integration

Logstash is moving away from the node protocol and multiline filter.  Here we'll outline the reasons along with the recommended alternatives.

<html><head></head><body><p>The Logstash project has been constantly evolving and seeking out novel ways to become more robust and feature rich. &#xA0;At this juncture, it&#x2019;s time to leave certain legacy things behind to make life easier both for Logstash users and developers. &#xA0;The topics to be outlined in detail here are the node protocol for Elasticsearch output and the multiline filter. &#xA0;Below is a brief overview of the Elastic recommended migrations for these respective components.</p><p><br></p><span id="docs-internal-guid-98eea4c5-8f38-7cb2-64a5-7812289f6ca4"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><strong>Discouraging the Node Protocol</strong></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" rel="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><strong><br></strong></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">The node protocol for the Elasticsearch output plugin has been a formidable pain for both Logstash users and developers. &#xA0;It&#x2019;s tough to debug and difficult to maintain. &#xA0;In Logstash 2.0, the <a href="https://www.elastic.co/blog/logstash-2-0-0-beta1-released">HTTP protocol has become the default</a>&#xA0;as it&#x2019;s very fast and possesses a much better operational experience. &#xA0;Additionally, a <a href="https://github.com/elastic/logstash/issues/4446">metrics API</a> is coming soon which will circumvent any use cases where the node protocol was previously used as a way to monitor Logstash node health.</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Therefore, usage of the node protocol is now discouraged, and it&#x2019;s strongly recommended for any current users of this protocol to migrate to the HTTP protocol in the next few months prior to the next major Logstash release. &#xA0;The HTTP protocol with Logstash is also the easiest way to ingest data into your clusters on Found&#xA0;- the best hosted Elasticsearch solution.</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">For any questions or concerns, please feel free to discuss in this <a href="https://github.com/elastic/logstash/issues/4433">Github issue</a>.</p><p><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><strong>Deprecating the Multiline Filter</strong></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><strong><br></strong></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Historically, there&apos;s been&#xA0;two different ways to process multiline events in Logstash. &#xA0;In the last couple months, the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html">multiline codec</a>&#xA0;has been strengthened on various fronts, inclusive of an enhancement which reconciled the stream identity bug. &#xA0;In order to mitigate confusion between the usage of the codec and filter plugins, the single-threaded multiline filter will&#xA0;effectively be deprecated at the Logstash 2.2 GA.</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">As multiline events are best processed earlier in the pipeline, it&#x2019;s strongly recommended to&#xA0;migrate&#xA0;to&#xA0;the more scalable multiline codec in the next few months prior to the next major Logstash release. &#xA0;Additionally, if you&#x2019;re using the awesome, lightweight <a href="https://www.elastic.co/products/beats">Beats</a> platform for shipping your data, this <a href="https://github.com/elastic/beats/issues/461">multiline processing</a> can also be done on the edge with <a href="https://www.elastic.co/products/beats/filebeat">Filebeat</a>.</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">For further details, please review this <a href="https://github.com/elastic/logstash/issues/4386">Github issue</a>.</p><p><br></p><hr><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;">Many exciting things are happening in Logstash land and we&#x2019;ll be talking about them in detail at our <a href="https://www.elastic.co/elasticon">Elastic{ON}16</a> user conference in San Francisco, CA. &#xA0;Please join us to learn more about the Elastic stack and network with many other enthusiastic Elastic users!</p></span></body></html>

road-828733.jpg

Logstash Moving Away from Node Protocol and Multiline Filter

An introduction to the initial Logstash plugins community maintainers and information on how others can join in on the fun!

<html><head></head><body><p>Through the years, the Logstash project has evolved into a kind of swiss army knife for data ingestion, collecting and unifying a variety of data from a myriad of data sources into Elasticsearch and other destinations. Beyond classic logging use cases, you can now unlock deeper insights from <a href="/guide/en/logstash/current/plugins-inputs-jdbc.html">relational and NoSQL data</a>, monitor <a href="/guide/en/logstash/current/plugins-inputs-http_poller.html">custom web applications</a> or any <a href="/guide/en/logstash/current/plugins-inputs-http.html">HTTP endpoints</a>, and easily derive value out of network packet data, system metrics, and more with the&#xA0;<a href="/products/beats">Beats</a> framework. It just keeps on growing...
</p><span id="docs-internal-guid-aae59303-9efb-c266-c5cf-ba081d55dbf3"><p><img src="https://api.contentstack.io/v2/assets/575e4cf5d8edd48f76936f9a/download?uid=bltefb16c0b13a818a4?uid=bltefb16c0b13a818a4" data-sys-asset-uid="bltd80035eddc25d395" alt="Logstash_diag.png" style="max-width:100%;">
</p><h3>The Logstash Plugins Family Continues to Grow
</h3><p>Logstash has seen tremendous activity recently, with around 2 million downloads in the last 6 months. The high extensibility, cardinality (a whopping 200 plugins and counting!), and quality of the Logstash plugins ecosystem is a huge driver behind this success, making Logstash very dynamic and inviting for so many different use cases. Much of this is massive thanks to the many community contributors who have reported issues, provided suggestions, submitted PRs, and even developed new plugins. One of the more notable features of Logstash is how effortlessly it empowers users to create custom plugins for their own use cases, for example, rapidly developing new plugins to ingest data from various in-house cloud services to conduct unified audit logging and security monitoring. New plugin creation is highly encouraged and there&apos;s tons of <a href="/guide/en/logstash/current/contributing-to-logstash.html">documentation</a> on how to get started (being a Ruby expert is not a requirement)!
</p><p>Unsurprisingly, many Logstash plugins were actually written by users like you, and it&apos;s a model that has helped scale the success of the plugins ecosystem. From inception, Jordan Sissel embraced the community and maintained a strong culture of making Logstash users happy. &quot;If a newbie has a bad time, it&apos;s a bug&quot; in Logstash that should be fixed whether in code, or in documentation, and always with some &lt;3. As the Logstash plugins family continues to grow, we&apos;re looking to work even more closely with the community to better foster this success together.
</p><h3>Logstash Plugins Community Maintainers
</h3><p>In a mission to further encourage community participation and sustain the quality of plugins in the ecosystem, we&apos;re introducing Logstash Plugins Community Maintainers. The role of a community maintainer is to own certain plugins in the <a href="https://github.com/logstash-plugins">Logstash Plugins</a> repository, and ensure the projects are healthy and the users are happy. There&apos;s so many enthusiastic, benevolent people in the community, and we&apos;re ecstatic to announce the first seven awesome members who share a similar vision and have stepped up as community maintainers. Each of these wonderful humans will be taking ownership of anywhere from one to seven plugins. In alphabetical order, here&apos;s an introduction to who they are and the plugins they own:
</p><div class="divider"></div><div class="row">
	<div class="col-lg-12 col-md-12 col-sm-12 col-xs-12">
		<div class="row">
			<div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">
				<p><img src="https://api.contentstack.io/v2/assets/575e4cf53dc542cb38c0944d/download?uid=blt2d8bfef877a0232f?uid=blt2d8bfef877a0232f" data-sys-asset-uid="blt358fb178ec9ece89" alt="Avishai Ish-Shalom" style="max-width:100%;">
			</p>
			</div>
			<div class="col-lg-9 col-md-9 col-sm-9 col-xs-12">
				<p><strong>Avishai Ish-Shalom</strong> is a veteran Ops and a survivor of many production skirmishes. In his spare time Avishai is spreading weird ideas and conspiracy theories such as DevOps.
			</p>
			<p><strong>Plugins Maintaining</strong>:
			</p>
			<ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-zeromq.html">logstash-filter-zeromq</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-zeromq.html">logstash-input-zeromq</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-codecs-cloudtrail.html">logstash-codec-cloudtrail</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-pipe.html">logstash-output-pipe</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-riemann.html">logstash-output-riemann</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-zeromq.html">logstash-output-zeromq</a></li>
			</ul>
			
			<p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/nukemberg">nukemberg</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/avishai-ish-shalom">avishai-ish-shalom</a>
			<br>
			<strong>Twitter</strong>: <a href="https://twitter.com/nukemberg">nukemberg</a>
			</p>
			</div>
		</div>
	</div>
</div><div class="divider"></div><div class="row"><div class="col-lg-12 col-md-12 col-sm-12 col-xs-12"><div class="row"><div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">		
				<p><img src="https://api.contentstack.io/v2/assets/575e4cf59e7a83165490d26c/download?uid=blte84417d7cfe1fdf4?uid=blte84417d7cfe1fdf4" data-sys-asset-uid="bltaa26fbbfc5087c0e" alt="Bj&#xF6;rn Puttman" style="max-width:100%;"></p>
			</div><div class="col-lg-9 col-md-9 col-sm-9 col-xs-12"><p><strong>Bj&#xF6;rn Puttman</strong> started programming with a TI-99/4a and previously worked as a web admin for the English board of students, Westf&#xE4;lische Wilhelms Universit&#xE4;t. He&apos;s been working on internet related things as a programmer and admin since 2000 and is currently employed at the dbap GmbH in M&#xFC;nster, Germany where he created the Logstash WebHDFS output plugin.
			</p><p><strong>Plugins Maintaining</strong>:
			</p><ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-webhdfs.html">logstash-output-webhdfs</a></li>
			</ul><p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/dstore_dbap">dstore_dbap</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/dstore-dbap">dstore-dbap</a>
			</p></div></div></div></div><div class="divider"></div><div class="row"><div class="col-lg-12 col-md-12 col-sm-12 col-xs-12"><div class="row"><div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">	
			<p><img src="https://api.contentstack.io/v2/assets/575e4d099985d58976377f06/download?uid=blt291a5a9a27ef9e9a?uid=blt291a5a9a27ef9e9a" data-sys-asset-uid="blt32883350c4764d03" alt="Fabien Baligand" style="max-width:100%;"></p>
			</div><div class="col-lg-9 col-md-9 col-sm-9 col-xs-12"><p><strong>Fabien Baligand</strong> is Tech Lead at ICDC and has a passion for Java, web technologies, and logs! He&apos;s an open source contributor from France and is the creator of the Logstash aggregate filter plugin.
			</p><p><strong>Plugins Maintaining</strong>:
			</p><ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html">logstash-filter-aggregate</a></li>
			</ul><p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/fbaligand">fbaligand</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/fbaligand">fbaligand</a>
			<br>
			<strong>Twitter</strong>: <a href="https://twitter.com/fbaligand">fabligand</a>
			</p></div></div></div></div><div class="divider"></div><div class="row"><div class="col-lg-12 col-md-12 col-sm-12 col-xs-12"><div class="row"><div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">
				<p><img src="https://api.contentstack.io/v2/assets/575e4d099985d58976377f11/download?uid=bltb4ca1dbaf6f3e878?uid=bltb4ca1dbaf6f3e878" data-sys-asset-uid="bltfe5399755a59f65c" alt="Joe Lawson" style="max-width:100%;">
			</p>
			</div><div class="col-lg-9 col-md-9 col-sm-9 col-xs-12"><p>Having practiced computer science near the Blue Ridge Mountains in Charlottesville, Virginia since 2004, <strong>Joe Lawson</strong> (aka joekiller) has successfully contributed to sites that have millions of unique visitors and billions of dollars in revenue. When he isn&apos;t creating self-healing; fault-tolerant systems, he enjoys his family, playing Team Fortress 2, Clemson and University of Virginia sports, and live music.
			</p><p><strong>Plugins Maintaining</strong>:
			</p><ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html">logstash-input-kafka</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html">logstash-output-kafka</a></li>
			</ul><p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/joe_lawson">Joe_Lawson</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/joekiller">joekiller</a>
			<br>
			<strong>Twitter</strong>: <a href="https://twitter.com/joekiller">joekiller</a>
			</p></div></div></div></div><div class="divider"></div><div class="row"><div class="col-lg-12 col-md-12 col-sm-12 col-xs-12"><div class="row"><div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">		
			<p><img src="https://api.contentstack.io/v2/assets/575e4d0ad8edd48f76937321/download?uid=blt9e23bdf5945ecf18?uid=blt9e23bdf5945ecf18" data-sys-asset-uid="blte18bf8fcd44d19a9" alt="Magnus B&#xE4;ck" style="max-width:100%;">
			</p></div><div class="col-lg-9 col-md-9 col-sm-9 col-xs-12"><p><strong>Magnus B&#xE4;ck</strong> is a software engineer residing in south Sweden with a great interest in automating all the things. Has been working with software releases and related tooling for about 11 years; currently working at Axis Communications and previously at Sony Ericsson, Google, and Sony Mobile.
			</p><p><strong>Plugins Maintaining</strong>:
			</p><ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-cidr.html">logstash-filter-cidr</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-prune.html">logstash-filter-prune</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-punct.html">logstash-filter-punct</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-websocket.html">logstash-input-websocket</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-exec.html">logstash-output-exec</a></li>
			</ul><p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/magnusbaeck">magnusbaeck</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/magnusbaeck">magnusbaeck</a>
			</p></div></div></div></div><div class="divider"></div><div class="row"><div class="col-lg-12 col-md-12 col-sm-12 col-xs-12"><div class="row"><div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">
			<p><img src="https://api.contentstack.io/v2/assets/575e4d0943e9adc538714f7a/download?uid=blt21eb5134b1cda77f?uid=blt21eb5134b1cda77f" data-sys-asset-uid="blt11ebf71913b7e227" alt="Philippe Weber" style="max-width:100%;">
			</p></div><div class="col-lg-9 col-md-9 col-sm-9 col-xs-12"><p><strong>Philippe Weber</strong> is a creative web-oriented software engineer based in Luxembourg, happy husband and father of 2 boys. After benefiting so much from existing open-source libraries and tools for years, he was captivated with early Logstash development, started contributing and never stopped following up its success story.
			</p><p><strong>Plugins Maintaining</strong>:
			</p><ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-relp.html">logstash-input-relp</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-stomp.html">logstash-input-stomp</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stomp.html">logstash-output-stomp</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-alter.html">logstash-filter-alter</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html">logstash-filter-clone</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-exec.html">logstash-input-exec</a></li>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-pipe.html">logstash-input-pipe</a></li>
			</ul><p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/wiibaa">wiibaa</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/wiibaa">wiibaa</a>
			<br>
			<strong>Twitter</strong>: <a href="https://twitter.com/wiibaa">wiibaa</a>
			</p></div></div></div></div><div class="divider"></div><div class="row"><div class="col-lg-12 col-md-12 col-sm-12 col-xs-12"><div class="row"><div class="col-lg-3 col-md-3 col-sm-3 col-xs-12">
			<p><img src="https://api.contentstack.io/v2/assets/575e4d0a58208ba076e2959b/download?uid=blt5b5b31d727414594?uid=blt5b5b31d727414594" data-sys-asset-uid="blt07dbb08e296c8f3d" alt="Russell Savage" style="max-width:100%;">
			</p></div><div class="col-lg-9 col-md-9 col-sm-9 col-xs-12"><p><strong>Russell Savage</strong> is a Business Systems Architect at Elastic specializing in pulling data from our marketing and sales systems into Elasticsearch and building tools for further analysis. When he&apos;s not working, you can usually find him on the ski slopes in Lake Tahoe.
			</p><p><strong>Plugins Maintaining</strong>:
			</p><ul>
				<li><a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-salesforce.html">logstash-input-salesforce</a></li>
			</ul><p><strong>Discuss</strong>: <a href="https://discuss.elastic.co/users/russellsavage">russellsavage</a>
			<br>
			<strong>Github</strong>: <a href="https://github.com/russorat">russorat</a>
			<br>
			<strong>Twitter</strong>: <a href="https://twitter.com/russellsavage">russellsavage</a>
			</p></div></div></div></div><div class="divider"></div><p>A big thank you to Russ, Philippe, Magnus, Joe, Fabien, Bj&#xF6;rn, and Avishai for admirably stepping into their community maintainer roles. Feel free to reach out to and @mention them through <a href="https://discuss.elastic.co/c/logstash">Discuss</a> or Github for questions and suggestions on specific plugins or even just to give them a high five!
</p><h3>This Is Awesome, Can I Help Too?
</h3><p>You are awesome and of course you can! We know there&apos;s more out there who would like to get involved, and we&apos;d love to hear from you. Although the overhead of the role is much lower with some existing Ruby and Logstash expertise, we invite anyone interested in being a Logstash Plugins Community Maintainer to shoot a &apos;hello&apos; to logstash-plugins [at] elastic.co along with:
</p><ol>
	<li>What excites you most about Logstash</li>
	<li>What plugin(s) you&apos;re interested in helping with</li>
	<li>Links to Discuss and Github profiles</li>
</ol><h3>What Plugins Need &lt;3?
</h3><p>The focus is around finding some love for our many community plugins which are listed <a href="https://docs.google.com/spreadsheets/d/1eElnJroNKD4jbu7AZhv0Ig648aN6P2OAEzmZwjF5Xok/edit?usp=sharing">here</a>. As shown above with the initial community maintainers, the level of commitment is completely up to you and your allocable bandwidth as we&apos;re all busy with this full-time job we call <em>life</em>.
</p><h3>The Logstash Team is Here to Help!
</h3><p>We sincerely appreciate the efforts of community maintainers! We know maintaining and growing plugins can be non-trivial, and the Logstash team is here to help the walk through this journey. Beyond a direct line to the core team, there&apos;s a new <a href="https://www.elastic.co/guide/en/logstash/current/community-maintainer.html">Community Maintainers Guide</a> that outlines general guidelines along with automated infrastructure for plugins like CI builds, PR testing, and CLA checks.
</p><p>Additionally, this is a great opportunity to build leadership visibility and repertoire in open source with commit access to Logstash plugins. You&apos;ll also be the first to know about upcoming developments and plugin ecosystem changes directly from the Logstash team as your feedback is invaluable! Lastly, did I mention swag?
</p><p>Logstash and the Elastic stack continues to grow quickly. If you&apos;re interested in learning more about the products and network with many other Elastic users, please join us at <a href="https://www.elastic.co/elasticon">Elastic{ON}<sup>16</sup></a>.
</p></span></body></html>

Articles by Alvin Chen