06 April 2018

Logstash Lines: Custom endpoints for AWS plugins

By Monica Sarbu

Did you know that Logstash 6.2 is already available? Try it and let us know what you think.

Logstash integrations

AWS Plugins (S3, SQS, Cloudwatch) now support custom endpoints

For a long time we've had many customers and users request the ability to connect Logstash with S3 compatible services like Ceph. We have now added this capability to all plugins that rely on the logstash-mixin-aws, the plugin that provides common configuration and library access to Amazon's AWS Ruby SDK.

We have also added support for the AssumeRole API and looser validation of the "region" parameter.

Other changes

logstash-codec-netflow - 3.11.4

  • Workaround for breaking change in Netflow-Input-UDP 3.2.0

logstash-codec-netflow - 3.11.3

  • Renamed some unknown VMware VDS fields

logstash-mixin-aws - 4.3.0

  • Drop strict value validation for region option #36
  • Add endpoint option to customize the endpoint uri #32
  • Allow user to provide a role to assume #27
  • Update aws-sdk dependency to '~> 2'

logstash-input-jdbc - 4.3.7

  • Load the driver with the system class loader. Fixes issue loading some JDBC drivers in Logstash 6.2+ #263

Logstash core

Fixed string serialization in the Persistent Queue

We finally found the root cause of a class of PQ serialization bugs. When a plugin would perform a substring operation on a Ruby string, the Ruby string class may actually return a slice of the string instead of a new object. Since we reach into Java object that backs the Ruby object, we needed to take special consideration there. We opened a PR, which has been merged into master/6.x, that fixes this issue.

Improved Java Execution Compilation:

We’re glad to say we’ve optimized the compilation of the Java execution engine a bit and also cleaned up some code.

Other changes

  • Fixed a race in the PQ that could lead to corruption in some cases. logstash#93
  • Improve ergonomics of bad `Event#set` error message: logstash#9298
  • Fix nested metadata lookup in java execution: logstash#9297
  • Fix location of config files when using zip/tar as a service: logstash#9304
  • Fix race condition in shutdown of pipelines: logstash#9285

UI

Logstash Pipeline Management

Starting with 6.3.0 users will be able to specify pipeline settings when creating or updating their centrally-managed pipelines.

Screen Shot 2018-04-03 at 6.08.10 AM.png

In addition, Logstash Centralized Configuration Management is getting a public API soon! This API will be hosted in Kibana.

Documentation

Repository: elastic/logstash-docs

Closed PRs

  • #9280 - Clean up trailing slashes in xrefs for 6.x doc
  • #9286 - Clean up trailing slashes in xrefs for 6.2 doc