Earlier this week, an <a href="https://issues.apache.org/jira/browse/LOG4J2-1863">advisory</a> was released detailing an object deserialization security flaw in the way Apache Log4j version 2 processes input data (<a href="http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5645">CVE-2017-5645</a>). This flaw would give a remote attacker the ability to execute code of their choosing within the JVM process listening for Log4j events.In a default Logstash install, the <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html">Log4j plugin</a> is installed but not enabled. If you aren't explicitly using this plugin in your configuration, you are not affected by this issue. When used in the Logstash pipeline, the Logstash log4j input plugin accepts Log4j version 1 data from remote applications. Often, any client is able to connect to Logstash because the connection offers no authentication. Given the very purpose of Logstash is to be an endpoint for receiving log data for an organization, it may not be practical to firewall the system as a form of protection. It is our expectation that Logstash will be easily reachable in most environments, and will accept whatever data is passed into it. The currently known exploits for Java object deserialization do not work against default Logstash deployments, but the vulnerability is still present even without a known exploit. We recognize that this doesn’t necessarily mean a Logstash isn’t vulnerable to this flaw, it simply means we’re not aware of a weaponized exploit. The Elastic Security Team does not believe the log4j input can be made 100% invulnerable &nbsp;given the way it receives log data from arbitrary sources. By its very nature, object deserialization is difficult to secure and may be impossible to secure when parsing remote untrusted data. The general consensus in the security community is that if you must do object deserialization, it should only be done between systems that have a high level of trust. We do not have this level of trust with expected Logstash clients. <h3 dir="ltr">Remediation</h3>We have patched the version of Log4j shipped in Logstash against this particular attack using a variant of the patch from the updated version of Apache Log4j. Updates for Logstash will be included in a future release. This will improve the security of the Log4j input, but we continue to have reservations about its security given the prior paragraph. Existing Logstash v5.x and v2.4 users can upgrade the log4j input to receive this fix today by doing the following: <pre>bin/logstash-plugin update logstash-input-log4j </pre><pre>% bin/logstash-plugin update logstash-input-log4j Updating logstash-input-log4j Updated logstash-input-log4j 3.0.3 to 3.0.5 </pre><h3 dir="ltr">Deprecation Notice</h3>Based on the reasons stated above, we are <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html#_deprecation_notice">deprecating</a> the Log4j input. Our recommendation is for current Log4j input users to stop using log4j’s SocketAppender in their applications. For safe transport of log4j logs, users should configure log4j to write logs to disk and use <a href="https://www.elastic.co/beats/filebeat">Filebeat</a> to forward to log information to Logstash. Setting up Filebeat to ship your local logs&nbsp;is easy and we've provided migration steps&nbsp;in <a href="https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html">these&nbsp;docs</a>. This solution removes object deserialization from being used in an insecure manner. Additionally we have marked the Log4j plugin as deprecated and are going to remove Log4j support in Logstash 6.0. Elastic would like to thank Marcio Almeida de Macedo of Red Team at Telstra for alerting us of this issue.

blog-logstash-banner.jpg

The future of Log4j input in Logstash

<html><head></head><body><h2 style="box-sizing: border-box; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">Summary</h2><ul><li>If you&#x2019;re having performance problems read the new&#xA0;<a href="https://www.elastic.co/guide/en/logstash/2.3/performance-troubleshooting.html">Performance Tuning Guide</a>&#xA0;Logstash output workers are confusing.</li><li>Logstash &lt;= 2.1 had slow and safe defaults</li><li>Logstash 2.2 had fast and unsafe defaults, this was mostly fine, but sometimes not</li><li>Logstash 2.3 went back to slow and safe defaults</li><li>Logstash 2.4 will let us start making plugins fast and safe as a default again, but we need to update individual plugins to do this using the new &#x2018;shared&#x2019; output type.</li><li>If your output is the bottleneck today manually set the output workers setting higher. This may cause increased resource usage and may cause some issues on some plugins, but it generally works for common plugins.</li></ul><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">A Brief Overview</h2><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">This blog post is meant to clarify the output worker changes in Logstash 2.2+, and to discuss how we&#x2019;ll be improving output workers in the future. Logstash 2.2 introduced the&#xA0;<a href="https://www.elastic.co/guide/en/logstash/current/pipeline.html">NG Pipeline</a>&#xA0;which provided a sizable performance improvement. It has created some confusion however. Prior to 2.2 there was a notion of Filter Workers (FWs) and Output Workers (OWs). Filter workers executed everything in the&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">filter {}</code>&#xA0;portion of the config, and each plugin in the&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">output {}</code>&#xA0;portion could use a user definable number of OW threads. In 2.2 these were both replaced with one concept of Pipeline Workers (PWs). Confusingly, you can still set OWs, and they still do something very important. Let&#x2019;s compare and contrast.&#xA0;<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The animation below contrasts the threading model between 2.1 and 2.2. <p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);" rel="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);"><img src="https://api.contentstack.io/v2/assets/57927100b252bbf533797ba3/download?uid=blt044416b614ff9178" data-sys-asset-uid="blt044416b614ff9178" alt="thread_changes.gif"> <p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">To put it a bit more formally:<ul><li>Logstash &lt; 2.2<ul><li>Filters / Outputs handle one event at a time</li><li>Queue between filter workers and output workers</li><li>Dedicated Thread per FW</li><li>Dedicated Thread per OW</li><li>Independently controllable filter/output workers</li></ul></li><li>Logstash &gt;= 2.2<ul><li>Filters and outputs deal in batches rather than single events.</li><li>No queue between filter worker and output worker as they now process batches as one unit</li><li>One Pipeline Worker thread pool shared by both filters and outputs</li><li>OWs are no longer threads, they are shared objects that can only be used by a single Pipeline Worker at a time.</li></ul></li></ul><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The upshot of this change is that in 2.2+&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">-w</code>, which now controls the pipeline worker setting, controls how many concurrent threads can exist for the filter and output stage combined (PWs). The output worker setting controls how many of those threads can simultaneously work on that output. The takeaways here are:<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">Output workers should never be set to a number &gt; the number of pipeline workers, since that is the maximum number that can be simultaneously used. You may need more PWs than cores on your system since multiple PWs can be blocked on I/O in a single OW object. Increasing the number of OWs often provides a tangible benefit but be careful. For some outputs adding more can either slow them down or cause them to break in subtle ways.<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">A more in-depth exploration of the history and the evolution of pipeline architecture is available in this video by Logstash creator Jordan Sissel. We encourage you to watch this! <p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);"> <iframe style="width: 500px; height: 281px;" src="//www.youtube.com/embed/FPLHS9Pmgk0" allowfullscreen frameborder="0"></iframe><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">Why Autoscaling Output Workers in Logstash 2.2 was a Mistake</h2><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The NG pipeline was released in 2.2 a few months after 2.0 went out. In Logstash 2.0 we had switched the default protocol for the Elasticsearch output to HTTP. One concern here was that while the plugin was nearly identical in performance / resource utilization to the old default of the Elasticsearch Transport protocol it needed more OWs to reach parity. There was a concern that users upgrading from 1.x to 2.x might feel this slowdown due to the defaults and not tune OW settings. We thought it would also be generally good to, by default, set the number of OWs for every plugin to be equal to the number of PWs. We code reviewed each output we maintain for concurrency issues and set the required options on ones that didn&#x2019;t support &gt; 1 OW, which meant that the pipeline would leave these at a single OW. This turned out to have some...err&#x2026; adverse effects.<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The main problem with this approach is that doing post-hoc code reviews for concurrency is hard. We have a truly massive amount of plugins which made the task even harder. We started getting bug reports in the 2.2.x series related to plugins we&#x2019;d missed that may have been threadsafe in terms of data structures, but not in terms of logic.<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">We made the decision that it was premature to assume that all outputs would behave well in a concurrent environment and to let those bugs shake out in a minor release, so, we reverted it, setting the default back to one OW. So, users who saw speed gains in 2.2.x might have seen speed losses in 2.3.x out of the box. Improving performance is a top priority for us, but correctness of outputs is of even greater importance. Luckily we can achieve both with the new model we&#x2019;re incorporating into the upcoming Logstash 2.4 and 5.0 series.<p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">All this being said, if you&#x2019;re having performance problems please read the new&#xA0;<a href="https://www.elastic.co/guide/en/logstash/2.3/performance-troubleshooting.html%5BPerformance">Performance Tuning Guide</a>. Its an easy to follow set of diagnostic and remedial procedures for managing Logstash performance.<h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; font-size: 1.5em; line-height: 1.25; padding-bottom: 0.3em; border-bottom-width: 1px; border-bottom-style: solid; border-bottom-color: rgb(238, 238, 238); color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; background-color: rgb(255, 255, 255);">The Future</h2><p style="box-sizing: border-box; margin-bottom: 16px; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; background-color: rgb(255, 255, 255);">The future is in just&#xA0;<a href="https://github.com/elastic/logstash/issues/5653">getting rid of OWs</a>. We call these &#x2018;shared&#x2019; outputs. Outputs that can&#x2019;t be parallelized will be the default, and will only support a single instance with synchronized access. The latest version of the ES output (currently only available in 5.x alphas, but most likely coming to ES 2.4.x) is the first shared Logstash output. These outputs manage all their own concurrency internally as a single shared object. The new ES output will run in parallel for as many PWs as you have with no additional configuration. There are some concurrency controls but these must be custom defined by the plugin in a way that makes sense for its own IO. For instance, on the Elasticsearch output plugin the maximum # of connections will be tunable (it defaulted to a very high number by default, at 100), but aside from that there aren&#x2019;t any.<p style="box-sizing: border-box; color: rgb(51, 51, 51); font-family: -apple-system, BlinkMacSystemFont, &quot;Segoe UI&quot;, Roboto, Helvetica, Arial, sans-serif, &quot;Apple Color Emoji&quot;, &quot;Segoe UI Emoji&quot;, &quot;Segoe UI Symbol&quot;; line-height: 21px; margin-bottom: 0px !important; background-color: rgb(255, 255, 255);">In the future we hope to refactor as many Logstash output plugins as possible to use the new&#xA0;<code style="box-sizing: border-box; font-family: Consolas, &quot;Liberation Mono&quot;, Menlo, Courier, monospace; font-size: 11.9px; padding: 0.2em 0px; border-radius: 3px; background-color: rgba(0, 0, 0, 0.0392157);">concurrency :shared</code>&#xA0;option, per&#xA0;<a href="https://github.com/elastic/logstash/issues/5661">Issue #5661</a>&#xA0;and use finer grain locking than what the OW system provided. We&#x2019;ll provide an update when this is done. In the meanwhile, we encourage our users to discuss these issues on our <a href="https://discuss.elastic.co/c/logstash">forum</a>, and open GitHub issues. We are also available on <a href="https://twitter.com/elastic">twitter</a> and <a href="https://webchat.freenode.net/#logstash">IRC</a> to help you out!</body></html>

log-grains-logstash-output-workers-full-bleed.jpg

bg_ss2.jpg

A History of Logstash Output Workers

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Logstash Creator & Tech Lead

Jordan Sissel

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

The content on this page is not available in the selected language.

O conteúdo desta página não está disponível no idioma selecionado. A Elastic está trabalhando para garantir que o conteúdo esteja disponível em vários idiomas. Agradecemos a compreensão.

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/elastic-cloud-account-terms" target="_self">Termos de Serviço</a> e também concorda em receber <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">comunicações especiais</a> por meio dos dados de contato que forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/serverless-preview-terms" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Ao enviar, você concorda com os <a href="/pt/agreements/global/cloud-services-monthly" rel="nofollow">Termos de Serviço do Elastic Cloud</a>. Seus dados pessoais serão tratados de acordo com a <a href="/pt/legal/privacy-statement">Declaração de Privacidade da Elastic</a>.

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Ao enviar a confirmação, você reconhece que leu e concorda com nossos <a href="/pt/legal/terms-of-use" target="_self">Termos de Serviço</a> e que a Elastic pode <a href="/pt/legal/privacy-statement#how-we-use-the-information" target="_self">entrar em contato com você</a> para divulgar nossos produtos e serviços relacionados usando os dados que você forneceu acima. Consulte a <a href="/pt/legal/privacy-statement/" target="_self">Declaração de Privacidade da Elastic</a> para saber mais detalhes ou para cancelar a qualquer momento.

Explore similar demos

Explore outras demonstrações semelhantes

Overview

Load More

Speakers

Close

See more insights

Author

Articles by

Learn more

Watch now (no PT)

Watch now

See all top stories

All (no PT translation)

Contact information

Press Release

Share on Reddit

Share this story

Share by Email

Thank you for your interest!

Contact Info

Follow us on Youtube

Share on LinkedIn

Follow us on LinkedIn

Search Integrations

All Solutions

Video for

Follow us on Twitter

See when this webinar starts in my time zone

Consultar a hora de início do webinar no meu fuso horário

Register to Watch

Register now

See All Posts

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Não vai conseguir participar? Inscreva-se e nós lhe enviaremos a gravação. Você também receberá um email com conteúdo relacionado.

Author

Articles by Jordan Sissel

Logstash Creator & Tech Lead, Elastic

The future of Log4j input in Logstash

A History of Logstash Output Workers