Logstash Lines: Update for December 12, 2017
Hello Logstashers! We're glad to present you all the past couple week's news in convenient digest form!
Added Support for Resetting Logging Back to Default
We've now added support for resetting logging settings changed via the Logstash web API back to their defaults.
Size caps for log files + gzip on rollover
We've added some much improvements fixes for Logstash's log4j logging. We will cap log files at 100MB per file, and gzip files as they're rolled over automatically. We're targeting 6.2 with this change.
HTTP Input Now Returns 429s Sometimes/ Runs faster/ No Longer Uses Excessive File Descriptors
The Logstash HTTP input previously exhibited poor behavior when the queue was blocked. If a client connected and timed out, LS would not release the connection, but rather, block indefinitely, causing the client to potentially time out. The HTTP protocol doesn’t deal well with long running requests. This plugin will now either return a 429 (busy) error when Logstash is backlogged, or it will time out the request.
If a 429 error is encountered clients should sleep, backing off exponentially with some random jitter, then retry their request.
This plugin will block if the Logstash queue is blocked and there are available HTTP input threads. This will cause most HTTP clients to time out. Sent events will still be processed in this case. This behavior is not optimal and will be changed in a future release. In the future, this plugin will always return a 429 if the queue is busy, and will not time out in the event of a busy queue.
While doing this work, we discovered that the HTTP input had some synchronization bottlenecks which were unnecessary. Users might see a nice perf boost on multicore systems.
LIR Fixes / Experimental Execution Fixes
One of the big changes in 6.0 is our work on a new execution backend as well as a new IR for our compiler. We've seen a number of small discrepancies pop up in the 6.x series that we've been ironing out with both the new IR (used by all users) and the new execution engine (will be optional with 6.1). We've done this work in a variety of commits.
Other Fixes