12 April 2018

Logstash Lines: Improved resilience in S3 input

By Monica Sarbu

Welcome to Logstash Lines! With these weekly series, we're keeping you up to date with what's new in Logstash, including the latest commits and releases.

Did you know that Logstash 6.2 is already available? Try it and let us know what you think.

Improved resiliency in the S3 input

The logstash S3 experience continues to receive some much needed attention. For some time the S3 Input plugin would fatally crash the pipeline on errors like "Access Denied" and "No such key". This has now been fixed in #143 and released in version 3.3.1 of the logstash-input-s3.

In progress: Java API for Logstash plugins

We’ve been making tons of progress on the Java API proposal: We decided to start with a simpler Java API that is closer to the Ruby API, which means that it will be easier to convert plugins. We closed down the old PR,  and moved the work over to a feature branch, whose progress you can track on #9432.

We’ve also opened an initial experiment in packaging Java code for pure java plugins in #9339. If you have thoughts on how Java plugins should be packaged, you can follow the meta issue #9227.

Javafication

The conversion of features to Java continues apace. We’ve ported an important internal class, the  output delegator to Java.

Exec input now tracks exit status and execution duration

If you're using the exec input to ingest results of command executions you now have information about the exit code on termination and also the duration of the execution. This information is stored in the "@metadata" field of the event and is useful when executing commands that perform health checks. This improvement was contributed by Jay Greenberg (thank you!).

Documentation

Changes in versioned_plugin_docs:

  • auto generated pr for updated versioned docs #502

All changes

Logstash core:

  • DRY filter wrapper: #9337
  • Return correct java type when invoking build codec: #9336
  • Fix unsafe resource handling, dead code, in PQ: #9332
  • Fix typo in pipeline startup message: #9330
  • PQ can read empty batches: #9328

Logstash integrations:

All input and output plugins (about 150) went through a mass patch release to fix the documentation of the default codec. More context here, example change for a plugin here.

For obvious reasons we won't list the 150 patch releases, only the ones unrelated to this mass change.

logstash-filter-dns - 3.0.9

  • Logging improvement to include DNS resolution failure reason #36

logstash-filter-jdbc_streaming - 1.0.4

  • Load the driver with the system class loader. Fixes issue loading some JDBC drivers in Logstash 6.2+ #263

logstash-filter-translate - 3.1.0

  • Add 'refresh_behaviour' to either 'merge' or 'replace' during a refresh #57

logstash-input-cloudwatch - 2.1.0

  • Add documentation for endpoint, role_arn and role_session_name #29
  • Reduce info level logging verbosity #27

logstash-input-exec - 3.3.0

  • Add metadata data to the event wrt execution duration and exit status #16

logstash-input-jdbc - 4.3.8

  • Clarify use of use_column_value. Make last_run_metadata_path reference in record_last_run entry clickable #273

logstash-input-s3 - 3.3.1

  • Improve error handling when listing/downloading from S3 #144

logstash-input-s3 - 3.3.0

  • Add documentation for endpoint, role_arn and role_session_name #142
  • Add support for additional_settings option #141

logstash-input-sqs - 3.1.0

  • Add documentation for endpoint, role_arn and role_session_name #46
  • Fix sample IAM policy to match to match the documentation #32

logstash-output-s3 - 4.1.0

  • Add documentation for endpoint, role_arn and role_session_name #174
  • Add option for additional settings #173
  • Add more S3 bucket ACLs #158
  • Handle file not found exception on S3 upload #144
  • Document prefix interpolation #154

logstash-output-sqs - 5.1.0

  • Add documentation for endpoint, role_arn and role_session_name #29